Transfer learning

Pan and Yang's 2010 survey formalises the setting with a source domain and task and a target domain and task, where a domain is a feature space plus a marginal distribution over it. Transfer is useful when the target task has little labelled data but shares structure with the source. Their taxonomy distinguishes inductive transfer (target labels available, different task), transductive transfer (same task, different domain, often called domain adaptation) and unsupervised transfer (no labels in either).

In deep learning the dominant pattern since the early 2010s has been to take a network pretrained on a large dataset (historically ImageNet for vision, later self-supervised text or image-text corpora) and adapt it. The spectrum runs from feature extraction, where the pretrained layers are frozen and only a new output head is trained, through partial fine-tuning of the top layers, to full fine-tuning of all weights, usually with a much smaller learning rate than for training from scratch, so the pretrained weights are only nudged. Early layers learn generic features (edges, textures, subword patterns) and transfer well; later layers are more task-specific. Parameter-efficient fine-tuning methods such as adapters and LoRA train a small number of added parameters while keeping the base frozen, which reduces memory and makes it cheap to keep many task variants of one base model.

Transfer can fail. Negative transfer occurs when source and target are dissimilar enough that the pretrained starting point performs worse than training from scratch, for example natural-photo features applied to spectrograms or radar data with very different statistics. Catastrophic forgetting is the loss of source-task abilities during aggressive fine-tuning; it matters when the adapted model is still expected to handle general inputs. Domain shift between source and target, such as a different camera, language variety or document layout, can remain even after fine-tuning and should be tested explicitly.

Transfer also moves risk. The target model inherits the source model's biases, knowledge gaps and any backdoor planted during pretraining; Gu et al. (2017, BadNets) showed that a backdoor in a traffic-sign detector can persist even after the network is retrained for another task. Using a third-party base model is therefore an AI supply chain decision: verify the source, pin the exact checkpoint by hash, prefer safe serialisation formats such as safetensors over pickle-based files that can execute code on load, and check the base model's licence and documentation (model card, training-data summary).

The terms overlap in current usage. Pretraining produces the source model, fine-tuning is the most common transfer mechanism, and prompting a foundation model with a few examples (in-context learning) achieves some of the same effect without changing any weights.