Fill-in-the-middle (FIM)

The standard reference is Bavarian et al. (OpenAI, 2022), "Efficient Training of Language Models to Fill in the Middle". A training document is split at two uniformly random character positions into prefix, middle and suffix, then serialised with sentinel tokens. In PSM order the sequence is <PRE> prefix <SUF> suffix <MID> middle <EOT>; in SPM order the suffix comes first, so prefix and middle are contiguous, which plays better with key-value caching when the user keeps typing. The transformed sequence is trained with the ordinary causal next-token loss, so no architectural change is needed; at inference the client supplies everything up to <MID> and the model generates the middle until it emits the end-of-text token.

The paper's central empirical claim is "FIM for free": applying the transformation to a large share of pretraining documents (they tested FIM rates up to 90%) did not measurably hurt left-to-right performance, while it gave strong infilling ability. Several design details mattered. Character-level span selection generalised better than splitting on line or token boundaries, because real cursors sit mid-token. Applying FIM at the context level, after documents are packed into training sequences, beat applying it per document. And adding FIM by fine-tuning an existing model was markedly less compute-efficient than including it during pretraining. The authors recommend a FIM rate between 50% and 90% with joint PSM and SPM training, and alongside InCoder's single-line and multi-line infilling benchmarks they introduced a random-span infilling benchmark built on HumanEval.

Other lines of work converged on the same idea. InCoder (Fried et al., 2022) used causal masking, moving masked spans to the end of the sequence, and code models such as StarCoder expose FIM through dedicated tokens like <fim_prefix>, <fim_suffix> and <fim_middle>. Because sentinel names and ordering differ between model families, a completion client must format prompts exactly as the model was trained, or quality collapses silently.

Known failure modes are practical rather than theoretical. The model may regenerate text that already exists in the suffix, fail to emit the end token and run on, or produce a middle that is syntactically valid but ignores constraints stated far away in the suffix. Tokenisation at the cursor is a subtle source of error: if the prefix ends in the middle of a word, the model sees an unusual token boundary, which is why some clients back up to a token boundary before sending and re-append the characters afterwards. FIM should not be confused with masked language modelling as in BERT, which predicts isolated masked tokens bidirectionally rather than generating a variable-length span autoregressively.