Rate Limit
Rate limits cap how many requests or tokens you can send per window. Hitting them triggers retries that raise cost.
Definition
A rate limit is a cap on requests or tokens per minute/hour set by the provider.
Why it matters for cost
When you hit a limit, your app usually retries. Retries multiply calls and tokens, quietly inflating cost during traffic spikes.
Practical tip
Add backoff and retry caps. Unbounded retries turn a throttling event into a cost spike.
Related reading
Batch APIs process large async request sets at a discount. Ideal for jobs that can tolerate delayed results.
The context window is the maximum tokens a model can see in one request. Bigger windows cost more per call when filled.
Embeddings turn text into vectors for search and retrieval. They're usually far cheaper per token than generation.
Fine-tuning trains a model on your data. It can cut prompt size and calls, but has upfront training cost to weigh.