Throughput (Tokens per Second)
Throughput measures how fast a model generates tokens. Higher throughput can lower latency-driven retry and infrastructure cost.
Definition
Throughput is how many tokens a model generates per second.
Why it matters for cost
Slow generation can increase timeout retries and the compute time you pay for on self-hosted setups. Faster models reduce both latency and retry-driven waste.
Practical tip
For high-volume, latency-sensitive paths, weigh throughput alongside per-token price when choosing a model.
Related reading
Batch APIs process large async request sets at a discount. Ideal for jobs that can tolerate delayed results.
The context window is the maximum tokens a model can see in one request. Bigger windows cost more per call when filled.
Embeddings turn text into vectors for search and retrieval. They're usually far cheaper per token than generation.
Fine-tuning trains a model on your data. It can cut prompt size and calls, but has upfront training cost to weigh.