Few-shot prompting

Brown et al. (2020) used the GPT-3 paper to define the vocabulary still in use: zero-shot (task description only), one-shot (one demonstration) and few-shot (as many demonstrations as fit in the context window, typically 10 to 100 in their 2,048-token setting), all without gradient updates, and they called the underlying capability in-context learning. Their central result was that few-shot performance improved much more steeply with model size than zero-shot performance, which is why the paper was titled "Language Models are Few-Shot Learners".

What the model learns from the examples is narrower than it looks. Min et al. (2022) found that replacing the gold labels in demonstrations with random labels barely reduced accuracy on many classification tasks; the examples mainly communicate the label space, the input distribution and the output format rather than the input-label mapping itself. Performance is also sensitive to surface choices. Zhao et al. (2021, "Calibrate Before Use") documented majority-label bias (favouring the label most common among the examples), recency bias (favouring the label of the last example) and common-token bias, and Lu et al. (2022) showed that simply reordering the same examples could move accuracy from near state-of-the-art to near chance. Sclar et al. (2023) found differences of up to 76 accuracy points from formatting alone.

Good practice follows from those findings: choose diverse, representative examples that cover edge cases, balance the labels, randomise or test their order, keep formatting identical to the real input, and delimit examples clearly (for instance in tagged blocks) so the model does not confuse them with the live input. Dynamic few-shot selects the examples per request by embedding similarity to the input, a small retrieval step. With instruction-tuned chat models, examples are often most effective for pinning down format, tone and tricky boundary cases, while the instruction carries the task definition. With reasoning models the benefit can be smaller and prescriptive examples may constrain the model's own approach.

The main trade-offs versus alternatives are cost and leakage. Every example is paid for on every call and consumes context window, although a stable example block at the start of the prompt can be served from a prompt cache. Examples copied from production data can put personal data into every request and into provider logs, and the model may reproduce details from them verbatim. When a task needs hundreds of examples, is run at high volume, or requires consistent behaviour that prompting cannot achieve, fine-tuning on the same examples becomes the better option.