Observability for LLM Applications
What to log for every LLM and agent call — prompts, tokens, latency, tool calls, outcomes — and how to turn traces into fixes.
AI ToolsIntermediate2 min readUpdated 11 Oct 2026
Why normal logs are not enough
An LLM app can return HTTP 200 with a wrong, slow or expensive answer. You need to see what the model saw, what it did and what it cost.
What to record per request
| Field | Why |
|---|---|
| Trace id, user / session, feature | Group steps and find a user's problem |
| Model and parameters | Reproduce behaviour |
| Prompt (or template + variables) and response | Debug wrong answers |
| Retrieved document ids | Tell retrieval failures from generation failures |
| Tool calls: name, arguments, result, duration | Debug agents |
| Input / output tokens, latency, cost | Budget and performance |
| Outcome: user feedback, eval score, error | Measure quality |
Traces for agents
An agent run is a tree: plan → tool call → model call → tool call. A trace view shows where time and tokens went and which step went wrong.
Turn traces into improvements
- Sample failed or low-rated traces each week.
- Label the failure type: retrieval miss, wrong tool, bad instruction, hallucination.
- Add the case to the evaluation set.
- Fix, re-run evals, and ship only if nothing else regresses.
Finished this note?