# LLM Inference: Cost and Latency in Production

> Where time and money go in an LLM call, and the techniques that cut both: smaller models, caching, streaming, batching and shorter prompts.

- Source: https://akshaythakare.com/notes/llms/inference-cost-and-latency
- Topic: LLMs
- Level: intermediate
- Reading time: 2 min
- Tags: llm, performance, cost, production
- Updated: 2026-10-11

## Where the time goes

| Phase | Driven by |
| --- | --- |
| Time to first token | Input length, model size, queueing |
| Generation | Number of output tokens, one at a time |
| Around the model | Retrieval, tool calls, network |

Output tokens usually dominate latency, and they are often priced higher than input tokens.

## Techniques

| Technique | Saves |
| --- | --- |
| Use the smallest model that passes your evals | Cost and latency |
| Route: small model by default, large model for hard cases | Cost |
| **Prompt caching** for a long, repeated prefix (system prompt, documents) | Input cost and time to first token |
| Limit output length; ask for concise formats | Latency and cost |
| **Stream** tokens to the user | Perceived latency |
| **Batch** offline jobs | Cost, where batch pricing is offered |
| Cache whole answers for repeated questions | Everything, for exact repeats |

## Measure per request

Log input tokens, output tokens, latency, model and outcome for every call. Cost per successful task is the number that matters, not cost per call.

> **Tip: Order of operations**
>
> Shorten the prompt and the output first, then try a smaller model against the evaluation set, then add caching.