Tokenizer

A modern tokenizer is a pipeline: Unicode normalisation (for example NFC or NFKC), pre-tokenisation that splits text into word-like chunks with a regular expression (separating letters, digits, punctuation and whitespace), the subword model itself, and post-processing that adds special tokens. Decoding reverses the mapping. Because the vocabulary and merge rules are learned from a sample corpus before pretraining and the model's embedding matrix is indexed by them, the tokenizer is frozen for the life of the model; changing it means retraining or an expensive vocabulary-adaptation procedure.

Three subword algorithms dominate. Byte-pair encoding, adapted to neural machine translation by Sennrich et al. (2016), starts from characters and repeatedly merges the most frequent adjacent pair, recording the merge order; encoding replays those merges. Byte-level BPE, introduced with GPT-2, starts from the 256 byte values instead of characters, so any input - any script, emoji or binary garbage - can be encoded without an unknown token. WordPiece, used by BERT, chooses merges by likelihood gain rather than raw frequency. The Unigram language-model method (Kudo, 2018) starts from a large candidate vocabulary and prunes it, allowing probabilistic segmentation; it is commonly used through the SentencePiece library (Kudo & Richardson, 2018), which treats text as a raw character stream and marks spaces with a special symbol so no language-specific pre-tokeniser is needed.

Tokenisation explains a number of model weaknesses. The model never sees characters inside a token, so spelling, letter counting, reversing strings and rhyming are harder than they look. Numbers split into irregular chunks hurt arithmetic, which is why some tokenizers split digits individually or in fixed groups. Leading spaces create distinct tokens (" Paris" and "Paris" differ), so trailing whitespace in a prompt can shift outputs. Tokens that were frequent in the tokenizer's training corpus but rare in the model's training data end up with poorly trained embeddings - the "glitch token" phenomenon publicised in 2023 with strings such as "SolidGoldMagikarp", which made models behave erratically.

There are security and operational consequences. Keyword or blocklist filters that operate on words can be bypassed with homoglyphs, zero-width characters, unusual spacing or encodings that tokenise differently but that the model still understands; filters should normalise input and ideally operate on model-level classifiers rather than string matching. Special tokens must be treated as control characters: the encoder used on untrusted text should refuse to emit them, otherwise a user can forge chat-template delimiters. For cost estimation always use the tokenizer of the exact target model, since counts for the same text differ between model families and sometimes between generations of the same family.