Zero-shot prompting

In the GPT-3 paper (Brown et al., 2020) zero-shot meant giving a pretrained model only a natural-language task description and the input, with no demonstrations and no weight updates, and it was the weakest of the three settings: raw pretrained models often continued the text in unexpected ways rather than performing the task, because nothing in pretraining taught them to treat an instruction as something to carry out. Earlier, and separately, "zero-shot learning" in machine learning referred to classifying into categories never seen in training, typically via shared attribute or text embeddings, as in CLIP's zero-shot image classification; the prompting sense borrows the name but not the method.

What made zero-shot prompting practical was instruction tuning. FLAN (Wei et al., 2021) fine-tuned a 137-billion-parameter pretrained model on more than 60 NLP datasets rephrased as natural-language instructions and found that it outperformed zero-shot GPT-3 175B on 20 of 25 held-out datasets; T0 (Sanh et al., 2021) reported similar findings with multitask prompted training, and InstructGPT (Ouyang et al., 2022) added reinforcement learning from human feedback. Held-out task clusters were essential to the evaluation, since a task seen during instruction tuning is not truly zero-shot. Every modern chat model is the product of this lineage, so everyday use of assistants is overwhelmingly zero-shot.

Zero-shot performance depends heavily on how completely the instruction specifies the task. The model must infer the label set, output format, level of detail and edge-case handling from the words alone, so ambiguous instructions produce inconsistent outputs across runs and inputs. Explicit output schemas or structured output, definitions of each category, stated handling of "none of the above", and a role in the system prompt close much of the gap to few-shot. Kojima et al. (2022) showed that zero-shot reasoning also improves markedly with a trigger such as "Let's think step by step" (zero-shot chain-of-thought), and reasoning models make this unnecessary by reasoning by default.

Choosing between zero-shot and few-shot is an empirical question. Zero-shot is cheaper per call, avoids leaking example data into prompts and logs, and avoids anchoring the model on idiosyncrasies of particular examples. Few-shot tends to win when the desired format is unusual, the label boundaries are subtle, or the output must match a house style. A sound workflow starts zero-shot, builds a small labelled evaluation set, measures, and adds examples only where the measurements show a benefit. Benchmark results reported as zero-shot should be read with contamination in mind: if the test set appeared in pretraining data, the model is not solving the task from the instruction alone.