RAG (Retrieval-Augmented Generation)
RAG retrieves relevant context before generation. Done well it cuts token waste; done poorly it bloats the context.
Definition
RAG fetches relevant documents or chunks and injects them into the prompt so the model answers from your data.
Why it matters for cost
Good retrieval sends only the needed chunks, keeping input tokens low. Poor retrieval dumps whole documents into the context, raising cost and hurting relevance.
Practical tip
Retrieve tightly and cap chunk count. Pair RAG with caching for stable retrieved context.
Related reading
Batch APIs process large async request sets at a discount. Ideal for jobs that can tolerate delayed results.
The context window is the maximum tokens a model can see in one request. Bigger windows cost more per call when filled.
Embeddings turn text into vectors for search and retrieval. They're usually far cheaper per token than generation.
Fine-tuning trains a model on your data. It can cut prompt size and calls, but has upfront training cost to weigh.