Context Window
The context window is the maximum tokens a model can see in one request. Bigger windows cost more per call when filled.
Definition
The context window is the total token budget for a single request: system prompt + conversation + retrieved data + the model's response.
Why it matters for cost
Every token in the window is billed as input (except the generated output). A 200k-token window filled with repeated history can turn a cheap call expensive.
Practical tip
Send only what the model needs each turn. Use summaries, retrieval, and chunk selection instead of dumping full history. Smaller contexts mean lower cost and often better answers.
Related reading
Batch APIs process large async request sets at a discount. Ideal for jobs that can tolerate delayed results.
Embeddings turn text into vectors for search and retrieval. They're usually far cheaper per token than generation.
Fine-tuning trains a model on your data. It can cut prompt size and calls, but has upfront training cost to weigh.
Distillation trains a small model to mimic a large one, trading some quality for much lower per-token cost at scale.