Streaming
Streaming returns tokens as they're generated, improving perceived latency. It doesn't change token cost but affects UX and timeouts.
Definition
Streaming delivers model output token-by-token instead of all at once.
Why it matters for cost
Streaming doesn't change the number of billed tokens—you pay the same. But it improves perceived speed and can reduce client-side timeouts that would otherwise trigger a retry (and extra cost).
Practical tip
Use streaming for user-facing generation to avoid timeout-driven retries, while keeping output token caps in place.
Related reading
Batch APIs process large async request sets at a discount. Ideal for jobs that can tolerate delayed results.
The context window is the maximum tokens a model can see in one request. Bigger windows cost more per call when filled.
Embeddings turn text into vectors for search and retrieval. They're usually far cheaper per token than generation.
Fine-tuning trains a model on your data. It can cut prompt size and calls, but has upfront training cost to weigh.