Batch API
Batch APIs process large async request sets at a discount. Ideal for jobs that can tolerate delayed results.
Definition
A batch API accepts many requests and processes them asynchronously, usually within a window up to 24 hours, at a discount.
Why it matters for cost
Batch endpoints commonly cost substantially less than real-time calls. Moving delay-tolerant work (backfills, bulk generation) to batch cuts spend without changing output quality.
Practical tip
Route by latency tolerance: batch everything that can wait, keep only live interactions real-time.
Related reading
The context window is the maximum tokens a model can see in one request. Bigger windows cost more per call when filled.
Embeddings turn text into vectors for search and retrieval. They're usually far cheaper per token than generation.
Fine-tuning trains a model on your data. It can cut prompt size and calls, but has upfront training cost to weigh.
Distillation trains a small model to mimic a large one, trading some quality for much lower per-token cost at scale.