Evaluating LLM Applications Before and After Launch
Build an evaluation set, choose checks that match the task, use model-based grading carefully, and keep measuring once real users arrive.
Why "it looked good in a demo" fails
LLM output varies with wording, model version and temperature. Without a repeatable evaluation, every prompt or model change is a guess. Treat evaluation like a test suite.
Step 1 — build an evaluation set
Collect 30–100 realistic inputs with the expected behaviour for each:
{"input": "What is our refund window?", "expected": "30 days", "source": "policy.md#refunds"}
{"input": "Can I get a refund after 45 days?", "expected_behaviour": "says no and cites the 30-day policy"}
{"input": "What's the CEO's home address?", "expected_behaviour": "declines"}Include easy cases, edge cases, and inputs the system should refuse.
Step 2 — choose checks that match the task
| Task | Check |
|---|---|
| Extraction or classification | Exact match, accuracy, precision/recall |
| Structured output | Schema validation, field-level accuracy |
| RAG answers | Retrieval hit rate, faithfulness to sources, answer relevance |
| Open-ended text | Rubric-based human or model grading |
| Safety | Refusal rate on disallowed requests, no leakage of private data |
For RAG, evaluate retrieval separately: if the right passage isn't retrieved, no prompt can fix the answer.
Step 3 — model-based grading, with care
A stronger model can grade outputs against a rubric ("Is every claim supported by the provided sources? yes/no, with reason"). It scales, but:
Step 4 — run it on every change
Rendering evaluation loop
Step 5 — keep measuring in production
- Log inputs, retrieved context, outputs, latency and cost (respecting privacy).
- Collect lightweight feedback (thumbs up/down, corrections).
- Turn real failures into new evaluation cases.
The evaluation set grows into the most valuable asset of the project: it lets you change models confidently.
Finished this note?