What is Transfer Learning?

Transfer learning reuses knowledge learned for one problem to improve learning on a related problem. Instead of initializing every parameter randomly, a practitioner starts from a pretrained model or representation and adapts it to a target task.

This approach is especially useful when the target dataset is small, labeling is expensive, or pretraining requires more compute than the target team can justify. Training a model from scratch is the alternative to transfer learning—not a type of transfer learning.

Key takeaways

• Feature extraction keeps a pretrained base frozen and trains a new task-specific head.
• Fine-tuning updates some or all pretrained parameters using target-domain data.
• Parameter-efficient methods such as adapters and LoRA update a small fraction of the model.
• Transfer can fail when source and target domains differ, licenses conflict, or the source model carries unsuitable bias.

Why transfer learning works

Models often learn representations that are useful beyond the exact data on which they were trained. Early layers of an image model may capture reusable local patterns; a language model may learn syntax, semantics, and broad associations from self-supervised prediction. A target task can build on those representations rather than relearn everything from limited examples.

The benefit depends on similarity between the source and target tasks, the scale and quality of pretraining, and the way the model is adapted. Reuse is not guaranteed: transferred features can be irrelevant or actively harmful.

Feature extraction

In feature extraction, the pretrained base is frozen so its parameters do not change. Its output becomes the input to a new classifier, regressor, or other task-specific head. Only the new head is trained.

This is fast and data-efficient, and it reduces the risk of destroying useful pretrained representations. It can also underfit when the target domain differs substantially from pretraining. Layers such as batch normalization require special care because their stored statistics and training behavior can affect adaptation even when most weights are frozen.

Fine-tuning

Fine-tuning updates pretrained parameters on target data. A common workflow is:

1. Load the pretrained model and replace or add the output head.
2. Freeze the base and train the new head.
3. Unfreeze selected layers—or the full model—and continue with a smaller learning rate.
4. Validate for overfitting, forgetting, and target-domain performance.

There is no universal rule that only the final layers should be tuned. The best choice depends on architecture, target-data size, domain similarity, normalization layers, memory, and compute. Full fine-tuning can offer more capacity but requires more resources and may cause catastrophic forgetting.

Parameter-efficient fine-tuning

Large transformers make full fine-tuning expensive. Parameter-efficient fine-tuning (PEFT) modifies or adds a small set of parameters while leaving most of the base model frozen.

• Adapters insert small trainable modules into the network.
• LoRA represents weight updates with low-rank matrices, reducing trainable parameters and optimizer memory.
• Prompt and prefix tuning learn continuous task-specific inputs or internal prefixes.

PEFT can store many task adaptations around one base model, although inference serving, adapter compatibility, and merged-weight management still require careful engineering.

Transfer learning across data types

Image classifiers commonly start from models pretrained on large image datasets. Language systems start from a foundation model and adapt it through supervised fine-tuning, preference optimization, retrieval, or tool use. Speech, audio, protein, and multimodal models follow related patterns.

Transfer can also occur without changing the original model. A frozen model can produce embeddings for a downstream classifier, a vector similarity search system, or a retrieval pipeline.

Domain shift and negative transfer

Domain shift occurs when target inputs differ from source data. A medical image model, for example, may encounter equipment, populations, or acquisition protocols absent from pretraining. Negative transfer means reuse makes target performance worse than an appropriate from-scratch baseline.

Teams should compare adaptation strategies, evaluate meaningful subgroups, and keep a target-domain test set. If the source task is poorly matched, a smaller domain-specific model may outperform a larger general model.

Licensing, provenance, and security

A downloadable model is not automatically safe to deploy. Review the license, permitted uses, training-data disclosures, model-card limitations, and dependency chain. Models can reproduce bias, memorize sensitive data, or contain malicious serialized code. Use trusted formats, scan artifacts, and load untrusted weights in an isolated environment.

When to use transfer learning

Transfer learning is a strong default when a relevant pretrained model exists and target data is limited. Training from scratch may be preferable when the domain is highly specialized, licensing is incompatible, model size exceeds deployment limits, or a simple task does not benefit from a large pretrained representation. The decision should be validated empirically rather than assumed from model scale.

Primary references

• TensorFlow: Transfer learning and fine-tuning
• Hugging Face PEFT documentation
• Hu et al.: LoRA—Low-Rank Adaptation of Large Language Models