How to evaluate an LLM app: test sets, metrics and LLM judges

LLM apps are non-deterministic and the outputs are free text, so people often judge them by "it seemed fine in the demo". That breaks as soon as you change a prompt, a model or a retrieval setting. Evaluation (evals) is the habit of measuring quality with repeatable tests, like unit tests for AI behaviour.

Step 1: build a test set

Collect real examples of what users ask or send:

• Start with 20 to 50 and grow. A small, good set is far better than none.
• Include typical cases, hard cases, edge cases, and things that must be refused.
• Add every bug or bad answer you find in production, so it never returns.
• For each, record the input and what a good result looks like: an expected answer, key facts that must appear, a required format, or the expected source document.
• Keep a separate held-out set you do not tune against, so you do not overfit your prompt to the tests.

Be careful with private data: remove or anonymise personal information in test examples.

Step 2: choose how to score

Exact or programmatic checks (best when possible)

• Output is valid JSON and matches the schema.
• A required field or phrase is present, a forbidden phrase is absent.
• The classification label equals the expected one.
• Code compiles and passes unit tests.
• A cited source ID is among the retrieved documents.
• Length, language and format rules.

These are cheap, fast and reliable.

Reference comparison

Compare to an expected answer with similarity measures. Useful for short factual answers, but weak for open-ended text because there are many correct wordings.

LLM-as-judge

Use a (usually strong) model to grade answers against a rubric, such as "Is the answer supported by the provided context? Yes or no, with a reason."

Tips for doing it well:

• Write a specific rubric with clear pass/fail criteria, not "rate the quality".
• Judge one aspect at a time (correctness, faithfulness, tone).
• Have the judge give a reason before the score.
• Check the judge against human labels on a sample. Judges have biases, such as favouring longer answers or their own style.
• Use a different or stronger model as judge where possible, and keep its prompt fixed.

Human review

Still the gold standard for subjective quality. Review a regular sample, and use it to calibrate automatic scores.

Step 3: measure the right things

For different apps:

• Classification or extraction: accuracy, precision and recall.
• RAG: did retrieval return the right passage? Is the answer faithful to it? Does it say "I don't know" when it should?
• Agents: did it finish the task, use the right tools, avoid forbidden actions, stay within step and cost limits?
• Safety: does it resist prompt injection and refuse harmful requests in your test cases?
• Operational: latency, token use and cost per request.

Look at failures, not only the average score. Group them by cause to see what to fix.

Step 4: run evals automatically

• Put the eval script in your repository and run it on every change to prompts, models or retrieval, like a CI test.
• Set a threshold: block the change if quality drops.
• Store results over time so you can see trends and compare versions.
• Because outputs vary, run important cases several times or at low temperature, and look at pass rates rather than a single run.

Step 5: monitor in production

Offline tests cannot cover everything real users do.

• Log inputs, outputs, retrieved sources and tool calls (with privacy controls).
• Collect thumbs up/down and written feedback.
• Sample real conversations for review and add the bad ones to your test set.
• Watch for drift: new topics, a provider changing the model version, or a data source changing.
• Alert on error rates, latency and cost spikes.

Common mistakes

• No test set, only demos.
• A test set so small or easy that everything passes.
• Tuning the prompt against the same examples you report on.
• Trusting an LLM judge without checking it against humans.
• Looking only at the average and ignoring serious failures.
• Never updating the set with real failures.

Takeaway: start with a small real test set, prefer programmatic checks, use LLM judges with clear rubrics and human spot checks, run evals on every change, and feed production failures back into the tests.

---

AI & LLM series, Part 2 (quality, agents and cost): 2 of 5. Previous: Fine-tuning vs RAG vs prompting: which one should you use?.