AI Cost Glossary
The terms behind your AI bill, explained in plain language—so you can reason about cost without decoding vendor docs.
Batch APIs process large async request sets at a discount. Ideal for jobs that can tolerate delayed results.
The context window is the maximum tokens a model can see in one request. Bigger windows cost more per call when filled.
Embeddings turn text into vectors for search and retrieval. They're usually far cheaper per token than generation.
Fine-tuning trains a model on your data. It can cut prompt size and calls, but has upfront training cost to weigh.
Distillation trains a small model to mimic a large one, trading some quality for much lower per-token cost at scale.
Prompt caching lowers the input cost of repeated context by caching a stable prefix across requests.
RAG retrieves relevant context before generation. Done well it cuts token waste; done poorly it bloats the context.
Rate limits cap how many requests or tokens you can send per window. Hitting them triggers retries that raise cost.
Streaming returns tokens as they're generated, improving perceived latency. It doesn't change token cost but affects UX and timeouts.
Temperature controls output randomness. Lower values are cheaper to reason about and more predictable for cost planning.
Throughput measures how fast a model generates tokens. Higher throughput can lower latency-driven retry and infrastructure cost.
A token is the unit LLM providers bill by—roughly a word fragment. Token count drives nearly all AI API cost.