Encoder

A transformer encoder block consists of unmasked multi-head self-attention followed by a position-wise feed-forward network, each with a residual connection and layer normalisation. Because there is no causal mask, every output vector is a function of the entire input sequence, which is what "bidirectional" means in practice. The whole sequence is processed in one parallel forward pass; there is no autoregressive loop and no KV cache, and cost is dominated by the O(n²) attention over the input length. In the original encoder-decoder transformer, the encoder runs once per input and its final hidden states serve as the keys and values that every decoder layer reads through cross-attention.

BERT (Devlin et al., 2019) established the encoder-only pattern. BERT-base has 12 layers, hidden size 768, 12 heads and about 110 million parameters; BERT-large has 24 layers, hidden size 1024 and about 340 million. Pretraining used masked language modelling: 15 % of token positions are selected, of which 80 % are replaced by [MASK], 10 % by a random token and 10 % left unchanged, and the model predicts the originals; a next-sentence-prediction task was added but RoBERTa (2019) showed it could be dropped. Input length was fixed at 512 positions, a limit that shaped chunking practice for years; later encoders such as ModernBERT (2024) extend it to 8,192 tokens. ELECTRA replaced masking with replaced-token detection, which is more sample-efficient.

To use an encoder for a task, a small head is placed on top: a linear classifier on the pooled output for sentence classification, a per-token classifier for named-entity recognition, or start and end pointers for extractive question answering. For embeddings, the per-token vectors must be pooled into one, usually by mean pooling or by taking the special [CLS] vector, and raw BERT outputs pooled this way are poor similarity vectors until the model is further trained contrastively, as Sentence-BERT did. The word "encoder" is also used more broadly for any network that maps an input to a representation, such as the ViT image encoder in a multimodal model or the audio encoder in Whisper.

The practical trade-off against decoders is that an encoder sees full context in both directions and is cheap to run, which suits classification, extraction, retrieval and reranking, but it has no efficient way to generate open-ended text: filling masks one at a time is possible but slow and of poor quality. For narrow classification tasks on a fixed label set, a fine-tuned encoder of a few hundred million parameters is often as accurate as prompting a large generative model, at a fraction of the latency and cost.