Token

A token is an entry in a model's fixed vocabulary, identified by an integer ID. Most vocabularies are learned with subword algorithms such as byte-pair encoding, so frequent words, common word fragments, whitespace-prefixed words, punctuation, digit groups and raw bytes each get their own entries. Vocabulary size is a design choice fixed before pretraining: GPT-2 used 50,257 entries, and more recent model families use vocabularies of roughly 100,000 to more than 250,000 entries so that more languages and code compress well. The ID is used to look up a row of the embedding matrix on input, and the output layer produces one logit per ID.

Beyond ordinary text tokens, vocabularies contain special tokens that never appear as literal text: beginning- and end-of-sequence markers, padding, role and turn delimiters used by chat templates, tool-call markers and, in some models, reasoning delimiters. Serving stacks normally refuse to let user text produce these tokens by accident, because a user who could inject a turn delimiter could fake a system or assistant message. Images, audio and video in multimodal models are also converted to tokens (patches or frames mapped to embeddings), which is why a picture has a token cost.

Token counts drive three operational quantities. Pricing is per million input and output tokens, with output typically priced several times higher and cached input cheaper. The context window and maximum output are expressed in tokens. Latency scales with output tokens because each requires a forward pass. A widely quoted rule of thumb is roughly four characters or three quarters of a word per token for English text, but the ratio depends on the tokenizer: Anthropic's documentation notes that 1 million tokens holds fewer words on its newest tokenizer than on earlier models, so identical text can cost different amounts on successive model generations.

The ratio also varies sharply by language and script. Petrov et al. (2023) measured tokenization lengths for parallel translations of the same text and found differences of up to 15 times between languages, persisting even in tokenizers designed to be multilingual. Danish is written in Latin script and fares better than many languages, but long compounds are still split into several pieces, so Danish text typically costs more tokens than equivalent English and fills the context window faster. For budgeting, count tokens with the specific model's tokenizer or token-counting endpoint rather than estimating from words. The AI sense of token is unrelated to authentication tokens, API keys or cryptographic tokens.