How to eval an LLM app like an engineer
A working map from first principles to production practice — for testing prompts, RAG pipelines, and agents with the same rigor you'd bring to any other system.
Vibes don't scale past week two
Every LLM project starts the same way: you write a prompt, run it a few times, read the output, and think "looks good." That works fine until you change the prompt, swap the model, or ship to more than a handful of users — and then you have no way to know whether you made things better or worse.
An eval is just a test suite for behavior that doesn't have a single correct answer. Instead of assert result == expected, you're asking questions like "did the assistant stay grounded in the retrieved documents?" or "did the agent call the right tool before answering?" The mechanics differ from unit testing, but the goal is identical: catch regressions before your users do, and get a number you can watch move.
The uncomfortable truth most teams learn the hard way is that evals are not a QA afterthought — they're how you find out what's actually broken. Skip them and you're debugging in the dark; do them well and half of your roadmap writes itself from the failures you uncover.
Start with error analysis, not infrastructure
The single biggest mistake teams make is building evaluation tooling before they know what's actually going wrong. The methodology that has emerged from practitioners like Hamel Husain and Shreya Shankar — who've run this process across dozens of production LLM teams — puts error analysis first, every time.
Keep reading traces until roughly 20 in a row surface nothing new — that's your signal you've reached theoretical saturation, though most teams need 100+ traces total before the taxonomy stabilizes. The output of this phase is a short, named list of ways your system actually fails, not a guess at ways it might.
Why this order matters
Build evaluators before doing error analysis and you end up scoring things nobody asked for — a generic "helpfulness" metric, a fluency score, a toxicity filter — while the actual bug (the agent silently drops the second half of multi-part questions) goes unmeasured and unfixed. Error analysis is what tells you which ten metrics out of a hundred possible ones are worth building.
One more staffing note that surprises people: resist distributed consensus-labeling. Appoint one person — a "benevolent dictator" — who owns quality judgments for a given area. Committees produce mushy, averaged labels; a single accountable domain expert produces a consistent taxonomy you can actually build against.
Not every failure needs a judge
Once you know your failure modes, match each one to the cheapest check that can reliably catch it. Most mature eval suites are a mix of all three layers below, not one technique used everywhere.
| Layer | Catches | Cost | Example |
|---|---|---|---|
| Deterministic assertions | Structural / mechanical failures | ~free | Valid JSON, required fields present, regex match, no banned words, latency under budget |
| Model-graded (LLM-as-judge) | Semantic / qualitative failures | low–medium | Is the answer grounded in the retrieved context? Did the tone match brand guidelines? |
| Human review | Anything subtle, novel, or high-stakes | high | Weekly spot-check of production traces; final sign-off before a judge is trusted |
A good rule of thumb: reach for code before you reach for a model. If a regex or a JSON schema check can catch it, use that — it's instant, free, and never drifts between runs. Save the LLM judge for the failures that genuinely require judgment.
Binary beats a 1–5 scale
When you do need a judge, resist the pull toward Likert scales ("rate this 1 to 5"). Binary pass/fail criteria force clearer thinking — both from you, when you write the rubric, and from the judge, when it applies it. A "3 vs. 4" distinction is almost always noise; "did it cite a real source, yes or no" is a decision anyone can make the same way twice.
Treat the judge like a model you're training
An LLM-as-judge evaluator is itself a small ML system, and it needs the same validation discipline as any classifier — otherwise you've just replaced "trust the app" with "trust the judge," which is no improvement.
- Write the rubric from your failure taxonomy, not from a generic template. Each criterion should map to a real failure mode you found in error analysis.
- Label a reference set by hand — a domain expert marks 30–100+ examples pass/fail with a short critique explaining why.
- Run the judge against the same examples and compare its verdicts to the human labels.
- Measure agreement with true positive / true negative rates, not just raw accuracy — a judge that rubber-stamps everything "pass" looks accurate on a mostly-passing dataset while being useless.
- Iterate the prompt (add examples, clarify edge cases, narrow the criterion) until agreement is high enough to trust, then keep spot-checking it over time — judges drift as your app and data change.
Pass Grounded answer
"Per section 4.2 of the uploaded contract, the notice period is 30 days." — claim traces directly to retrieved text.
Fail Ungrounded answer
"The notice period is typically 30 days." — plausible, unsourced, and not actually checkable against the context provided.
Note that the model judging your output doesn't have to be a different, more expensive model — the same model that runs your task can often judge it well, since generating a response and critiquing one against a rubric are different skills.
Split retrieval from generation
A RAG pipeline fails in two independent places — the retriever finds the wrong (or no) documents, or the generator says the wrong thing about the documents it was given. Conflating the two makes debugging much harder, so evaluate them separately before you evaluate the pipeline end to end.
Retrieval metrics
These need a small ground-truth set mapping queries to the documents that should come back.
| Metric | Question it answers |
|---|---|
| Precision@k | Of the top k chunks retrieved, how many were actually relevant? |
| Recall@k | Of all the relevant chunks that exist, how many made it into the top k? |
| Hit rate | Did at least one relevant chunk appear in the top k at all? |
| NDCG@k | Were the most relevant chunks ranked near the top, not just present? |
Generation metrics
These judge what the model did with the context it was handed, usually via an LLM judge or embedding similarity rather than exact-match.
- Faithfulness / groundedness — every claim in the answer traces back to something in the retrieved context; nothing is invented.
- Answer relevancy — the response actually addresses what the user asked, not just something adjacent to it.
- Context precision / relevance — the retrieved chunks that made it into the prompt were worth including, not padding that dilutes the model's attention.
- Completeness — for multi-part questions, did the answer cover every part, or quietly drop one?
Reference-based vs. reference-free
When you have a known-correct answer, compare directly (semantic similarity or an LLM judge scoring "does this match the reference"). In production you usually don't have a reference — that's when faithfulness and relevancy checks against the retrieved context itself (rather than a ground-truth answer) become your main signal.
Keep your test query set stable across experiments — swapping it out between prompt versions makes every comparison meaningless. Build it from real user queries wherever you can, and add adversarial and edge-case queries (ambiguous phrasing, out-of-scope questions, multi-hop questions needing several chunks) once basic retrieval works.
Judge the path, not just the destination
A single-turn LLM eval asks "was this output good?" An agent eval has to ask that plus "did it get there the right way?" — because two agents can reach the same final answer while one silently called a destructive tool it shouldn't have, or looped three times before recovering.
| Dimension | What you're checking |
|---|---|
| Task completion | Did the agent actually accomplish the user's goal, end to end? |
| Tool selection | Did it choose the right tool for each step, and skip tools it didn't need? |
| Tool-call correctness | Were the arguments to each call valid and appropriate given the state at that point? |
| Trajectory / plan quality | Was the sequence of steps sensible, or did it wander, repeat, or take an inefficient path? |
| State tracking | Across multiple turns and tool calls, did it keep an accurate picture of what's already happened? |
This is why agent evaluation leans heavily on trace-level inspection rather than a single pass/fail on the final message: you need to see every intermediate step to know whether "got the right answer" was skill or luck. Most agent-eval tooling exists specifically to make that trace readable — showing the full call graph, not just the transcript.
In practice, build agent evals in layers: unit-test individual tool calls in isolation first (given this state, does it call the right tool with the right arguments?), then evaluate short multi-step trajectories, then run full end-to-end scenarios including adversarial ones — a user who changes their mind mid-task, contradictory instructions, a tool that returns an error.
Two eval sets, two jobs
Don't try to make one eval suite serve both continuous integration and production monitoring — they optimize for different things and should look different.
CI / regression suite
Small, curated, stable (100+ examples). Runs on every PR. Favor cheap, deterministic checks. Every past production bug becomes a permanent regression case here — once fixed, it never breaks again unnoticed.
Production monitoring
Samples live traffic asynchronously. No reference answers available, so lean on LLM judges and reference-free RAG/agent metrics. Its job is discovering new failure modes, not confirming old fixes still hold.
The loop that connects them: production monitoring and periodic error analysis surface a new failure mode → you write a targeted eval for it → a handful of the actual failing examples get added to the CI regression set → that failure mode can never silently regress again. Over months, your CI suite becomes a distilled history of every real bug your app has had.
Where checks run
- Pre-commit / PR check — fast, deterministic assertions and a small judged subset; should finish in the time a normal test suite would.
- Nightly / pre-release — the full regression set plus any expensive judge-based metrics you skip on every PR.
- Continuous production sampling — a rolling percentage of live traffic scored asynchronously, feeding dashboards and alerting on drift.
Mechanically this looks a lot like ordinary software testing: version your prompts and eval datasets together, treat a metric drop as a failing build, and keep the eval run fast enough that people don't route around it.
The building blocks, not a shopping list
You can build a serious eval practice with nothing but a script and a spreadsheet — the discipline in the sections above matters more than any tool. That said, a mature ecosystem now covers most of the plumbing, organized roughly by what problem each layer solves.
| Layer | What it's for | Examples |
|---|---|---|
| Declarative test runners | Config-driven prompt/RAG/agent test suites, CI-friendly | Promptfoo |
| Metric libraries | Ready-made faithfulness, relevancy, and RAG-specific scorers | RAGAS, DeepEval |
| Experiment + trace platforms | Logging traces, comparing runs, human annotation queues | Braintrust, LangSmith, Arize Phoenix |
| ML lifecycle platforms | Tracking experiments alongside broader model/agent ops | MLflow |
Whatever you pick, treat it the same way you'd treat a testing framework: it should run in CI, version alongside your code, and be boring enough that nobody dreads opening it.
Anti-patterns worth naming
Eval-driven development
Building evaluators before doing error analysis. You end up measuring imagined problems and missing real ones.
Generic off-the-shelf metrics
"Helpfulness," BLEU/ROUGE-style scores rarely track what actually breaks in your specific app.
Near-100% pass rates
If your eval suite always passes, it's not testing anything hard enough to matter. Go find the failures.
Outsourcing error analysis
Handing trace review to someone outside the team severs the feedback loop between seeing failures and fixing the product.
Committee labeling
Averaging several people's pass/fail judgments produces mushy, inconsistent ground truth. One accountable owner beats consensus.
Trusting an unvalidated judge
Shipping an LLM-as-judge without measuring its agreement against human labels first — you've just added a second unverified model.
Your first two weeks
A concrete on-ramp if you're starting from zero on a real project.
- Pull 20–30 real transcripts (or generate realistic synthetic ones if you have no users yet).
- Read every one yourself. Write a blunt, specific note on anything that feels off — don't filter yet.
- Group the notes into 4–8 named failure categories. Give each one a short definition and one example.
- For each category, decide: can code check this, or does it need a judge?
- Write the deterministic checks first — they're fast wins.
- For the judge-worthy categories, hand-label 30+ examples pass/fail, then draft and validate one judge prompt per category.
- Wire the resulting suite into your PR pipeline as a required check, even if it only has a dozen cases at first.
- Put 30 minutes on the calendar, every week, to read new traces. This is the habit that keeps the whole system honest.
Evals aren't a phase you finish — they're the mechanism by which you keep learning what your system actually does, for as long as it's running.
What this guide draws on
Every claim above is grounded in current practitioner writing rather than guesswork — this section lists exactly where each part of the guide came from, so you can go deeper on anything that matters most to your project.
Primary sources
Read in full and used directly to shape the structure and claims of sections 01–06.
| Source | Publisher | Used for |
|---|---|---|
| LLM Evals: Everything You Need to Know | Hamel Husain | The error-analysis-first workflow, open/axial coding, binary pass/fail rubrics, judge validation, CI vs. monitoring split (§01–03, §06) |
| A complete guide to RAG evaluation | Evidently AI | Retrieval metrics, generation metrics, reference-based vs. reference-free evaluation (§04) |
| The Complete Guide to LLM & AI Agent Evaluation in 2026 | Adaline | Agent trajectory evaluation, tool-call correctness, trace-level analysis (§05) |
| CI/CD Integration for LLM Eval and Security | Promptfoo docs | Where evals run in a pipeline — PR checks, nightly runs, production sampling (§06) |
Also consulted
Reviewed for context and to sanity-check the tooling landscape in §07.
| Source | Publisher |
|---|---|
| LLM-as-a-Judge in 2026: Top evaluation techniques and best practices | DeepEval |
| DeepEval alternatives (2026) | Braintrust |
| LLM Evaluation Framework Benchmark 2026 | aiml.qa |
| AI Agent Evaluation Frameworks (2026): 7 Compared | Morph |
| AI Evals For Engineers & PMs | Hamel Husain & Shreya Shankar (Maven course) |
Compiled August 2026. This is a fast-moving field — tool names in §07 will age faster than the underlying method in §01–06.