LLM Inference: Cost and Latency in Production
Where time and money go in an LLM call, and the techniques that cut both: smaller models, caching, streaming, batching and shorter prompts.
LLMsIntermediate2 min readUpdated 11 Oct 2026
Where the time goes
| Phase | Driven by |
|---|---|
| Time to first token | Input length, model size, queueing |
| Generation | Number of output tokens, one at a time |
| Around the model | Retrieval, tool calls, network |
Output tokens usually dominate latency, and they are often priced higher than input tokens.
Techniques
| Technique | Saves |
|---|---|
| Use the smallest model that passes your evals | Cost and latency |
| Route: small model by default, large model for hard cases | Cost |
| Prompt caching for a long, repeated prefix (system prompt, documents) | Input cost and time to first token |
| Limit output length; ask for concise formats | Latency and cost |
| Stream tokens to the user | Perceived latency |
| Batch offline jobs | Cost, where batch pricing is offered |
| Cache whole answers for repeated questions | Everything, for exact repeats |
Measure per request
Log input tokens, output tokens, latency, model and outcome for every call. Cost per successful task is the number that matters, not cost per call.
Finished this note?