Technique Updated 2026-04
Model Distillation
Definition
Distillation transfers knowledge from a large model (teacher) to a smaller model (student), preserving performance at lower cost.
See also in the glossary
L
LLM (Large Language Model)
An LLM is an AI model trained on billions of texts, capable of understanding and generating human language.
S
SLM (Small Language Model)
An SLM is a compact language model optimized to run on local devices with targeted performance on specific tasks.
F
Fine-tuning
Fine-tuning is the process of retraining an existing AI model on a specific dataset to adapt it to a particular domain or task.
Q
Quantization
Quantization reduces the precision of numbers in an AI model to make it smaller and faster, with minimal quality loss.
Tools that use model distillation
Frequently Asked Questions
Why distill instead of fine-tune?
Fine-tuning adapts an existing model. Distillation creates a new smaller model that mimics a larger one. The result is faster and cheaper to run.
Does DeepSeek use distillation?
Yes. DeepSeek used distillation to create compact high-performing models, contributing to their unbeatable value ratio.
