Model Distillation
Distillation trains a small model to mimic a large one, trading some quality for much lower per-token cost at scale.
Definition
Distillation trains a smaller, cheaper model to reproduce a larger model's behavior.
Why it matters for cost
A distilled model can be far cheaper per token and faster, which compounds across high call volumes—if the quality loss is acceptable for the task.
Practical tip
Use distillation for high-volume, lower-stakes steps, and reserve the large model for tasks where quality is critical.
Related reading
Batch APIs process large async request sets at a discount. Ideal for jobs that can tolerate delayed results.
The context window is the maximum tokens a model can see in one request. Bigger windows cost more per call when filled.
Embeddings turn text into vectors for search and retrieval. They're usually far cheaper per token than generation.
Fine-tuning trains a model on your data. It can cut prompt size and calls, but has upfront training cost to weigh.