Field guide · v1.1

How to eval an LLM app like an engineer

A working map from first principles to production practice — for testing prompts, RAG pipelines, and agents with the same rigor you'd bring to any other system.

00 — Why evals

Vibes don't scale past week two

Every LLM project starts the same way: you write a prompt, run it a few times, read the output, and think "looks good." That works fine until you change the prompt, swap the model, or ship to more than a handful of users — and then you have no way to know whether you made things better or worse.

An eval is just a test suite for behavior that doesn't have a single correct answer. Instead of assert result == expected, you're asking questions like "did the assistant stay grounded in the retrieved documents?" or "did the agent call the right tool before answering?" The mechanics differ from unit testing, but the goal is identical: catch regressions before your users do, and get a number you can watch move.

The uncomfortable truth most teams learn the hard way is that evals are not a QA afterthought — they're how you find out what's actually broken. Skip them and you're debugging in the dark; do them well and half of your roadmap writes itself from the failures you uncover.

01 — The core loop

Start with error analysis, not infrastructure

The single biggest mistake teams make is building evaluation tooling before they know what's actually going wrong. The methodology that has emerged from practitioners like Hamel Husain and Shreya Shankar — who've run this process across dozens of production LLM teams — puts error analysis first, every time.

Collect
Gather real traces
Pull 20–100 real (or realistic synthetic) transcripts of your app in action.
Open code
Read and annotate
A domain expert reads each one and writes open-ended notes on anything that looks wrong.
Axial code
Group into a taxonomy
Cluster the notes into named failure modes: "cites wrong section," "ignores date filter," "hallucinates tool args."
Automate
Build targeted checks
Only now do you write evaluators — one per failure mode that actually showed up.

Keep reading traces until roughly 20 in a row surface nothing new — that's your signal you've reached theoretical saturation, though most teams need 100+ traces total before the taxonomy stabilizes. The output of this phase is a short, named list of ways your system actually fails, not a guess at ways it might.

Why this order matters

Build evaluators before doing error analysis and you end up scoring things nobody asked for — a generic "helpfulness" metric, a fluency score, a toxicity filter — while the actual bug (the agent silently drops the second half of multi-part questions) goes unmeasured and unfixed. Error analysis is what tells you which ten metrics out of a hundred possible ones are worth building.

One more staffing note that surprises people: resist distributed consensus-labeling. Appoint one person — a "benevolent dictator" — who owns quality judgments for a given area. Committees produce mushy, averaged labels; a single accountable domain expert produces a consistent taxonomy you can actually build against.

02 — Three kinds of check

Not every failure needs a judge

Once you know your failure modes, match each one to the cheapest check that can reliably catch it. Most mature eval suites are a mix of all three layers below, not one technique used everywhere.

LayerCatchesCostExample
Deterministic assertions Structural / mechanical failures ~free Valid JSON, required fields present, regex match, no banned words, latency under budget
Model-graded (LLM-as-judge) Semantic / qualitative failures low–medium Is the answer grounded in the retrieved context? Did the tone match brand guidelines?
Human review Anything subtle, novel, or high-stakes high Weekly spot-check of production traces; final sign-off before a judge is trusted

A good rule of thumb: reach for code before you reach for a model. If a regex or a JSON schema check can catch it, use that — it's instant, free, and never drifts between runs. Save the LLM judge for the failures that genuinely require judgment.

Binary beats a 1–5 scale

When you do need a judge, resist the pull toward Likert scales ("rate this 1 to 5"). Binary pass/fail criteria force clearer thinking — both from you, when you write the rubric, and from the judge, when it applies it. A "3 vs. 4" distinction is almost always noise; "did it cite a real source, yes or no" is a decision anyone can make the same way twice.

03 — LLM-as-judge

Treat the judge like a model you're training

An LLM-as-judge evaluator is itself a small ML system, and it needs the same validation discipline as any classifier — otherwise you've just replaced "trust the app" with "trust the judge," which is no improvement.

  1. Write the rubric from your failure taxonomy, not from a generic template. Each criterion should map to a real failure mode you found in error analysis.
  2. Label a reference set by hand — a domain expert marks 30–100+ examples pass/fail with a short critique explaining why.
  3. Run the judge against the same examples and compare its verdicts to the human labels.
  4. Measure agreement with true positive / true negative rates, not just raw accuracy — a judge that rubber-stamps everything "pass" looks accurate on a mostly-passing dataset while being useless.
  5. Iterate the prompt (add examples, clarify edge cases, narrow the criterion) until agreement is high enough to trust, then keep spot-checking it over time — judges drift as your app and data change.

Pass Grounded answer

"Per section 4.2 of the uploaded contract, the notice period is 30 days." — claim traces directly to retrieved text.

Fail Ungrounded answer

"The notice period is typically 30 days." — plausible, unsourced, and not actually checkable against the context provided.

Note that the model judging your output doesn't have to be a different, more expensive model — the same model that runs your task can often judge it well, since generating a response and critiquing one against a rubric are different skills.

04 — Evaluating RAG

Split retrieval from generation

A RAG pipeline fails in two independent places — the retriever finds the wrong (or no) documents, or the generator says the wrong thing about the documents it was given. Conflating the two makes debugging much harder, so evaluate them separately before you evaluate the pipeline end to end.

Retrieval metrics

These need a small ground-truth set mapping queries to the documents that should come back.

MetricQuestion it answers
Precision@kOf the top k chunks retrieved, how many were actually relevant?
Recall@kOf all the relevant chunks that exist, how many made it into the top k?
Hit rateDid at least one relevant chunk appear in the top k at all?
NDCG@kWere the most relevant chunks ranked near the top, not just present?

Generation metrics

These judge what the model did with the context it was handed, usually via an LLM judge or embedding similarity rather than exact-match.

  • Faithfulness / groundedness — every claim in the answer traces back to something in the retrieved context; nothing is invented.
  • Answer relevancy — the response actually addresses what the user asked, not just something adjacent to it.
  • Context precision / relevance — the retrieved chunks that made it into the prompt were worth including, not padding that dilutes the model's attention.
  • Completeness — for multi-part questions, did the answer cover every part, or quietly drop one?

Reference-based vs. reference-free

When you have a known-correct answer, compare directly (semantic similarity or an LLM judge scoring "does this match the reference"). In production you usually don't have a reference — that's when faithfulness and relevancy checks against the retrieved context itself (rather than a ground-truth answer) become your main signal.

Keep your test query set stable across experiments — swapping it out between prompt versions makes every comparison meaningless. Build it from real user queries wherever you can, and add adversarial and edge-case queries (ambiguous phrasing, out-of-scope questions, multi-hop questions needing several chunks) once basic retrieval works.

05 — Evaluating agents

Judge the path, not just the destination

A single-turn LLM eval asks "was this output good?" An agent eval has to ask that plus "did it get there the right way?" — because two agents can reach the same final answer while one silently called a destructive tool it shouldn't have, or looped three times before recovering.

DimensionWhat you're checking
Task completionDid the agent actually accomplish the user's goal, end to end?
Tool selectionDid it choose the right tool for each step, and skip tools it didn't need?
Tool-call correctnessWere the arguments to each call valid and appropriate given the state at that point?
Trajectory / plan qualityWas the sequence of steps sensible, or did it wander, repeat, or take an inefficient path?
State trackingAcross multiple turns and tool calls, did it keep an accurate picture of what's already happened?

This is why agent evaluation leans heavily on trace-level inspection rather than a single pass/fail on the final message: you need to see every intermediate step to know whether "got the right answer" was skill or luck. Most agent-eval tooling exists specifically to make that trace readable — showing the full call graph, not just the transcript.

In practice, build agent evals in layers: unit-test individual tool calls in isolation first (given this state, does it call the right tool with the right arguments?), then evaluate short multi-step trajectories, then run full end-to-end scenarios including adversarial ones — a user who changes their mind mid-task, contradictory instructions, a tool that returns an error.

06 — In your pipeline

Two eval sets, two jobs

Don't try to make one eval suite serve both continuous integration and production monitoring — they optimize for different things and should look different.

CI / regression suite

Small, curated, stable (100+ examples). Runs on every PR. Favor cheap, deterministic checks. Every past production bug becomes a permanent regression case here — once fixed, it never breaks again unnoticed.

Production monitoring

Samples live traffic asynchronously. No reference answers available, so lean on LLM judges and reference-free RAG/agent metrics. Its job is discovering new failure modes, not confirming old fixes still hold.

The loop that connects them: production monitoring and periodic error analysis surface a new failure mode → you write a targeted eval for it → a handful of the actual failing examples get added to the CI regression set → that failure mode can never silently regress again. Over months, your CI suite becomes a distilled history of every real bug your app has had.

Where checks run

  • Pre-commit / PR check — fast, deterministic assertions and a small judged subset; should finish in the time a normal test suite would.
  • Nightly / pre-release — the full regression set plus any expensive judge-based metrics you skip on every PR.
  • Continuous production sampling — a rolling percentage of live traffic scored asynchronously, feeding dashboards and alerting on drift.

Mechanically this looks a lot like ordinary software testing: version your prompts and eval datasets together, treat a metric drop as a failing build, and keep the eval run fast enough that people don't route around it.

07 — Tooling landscape

The building blocks, not a shopping list

You can build a serious eval practice with nothing but a script and a spreadsheet — the discipline in the sections above matters more than any tool. That said, a mature ecosystem now covers most of the plumbing, organized roughly by what problem each layer solves.

LayerWhat it's forExamples
Declarative test runnersConfig-driven prompt/RAG/agent test suites, CI-friendlyPromptfoo
Metric librariesReady-made faithfulness, relevancy, and RAG-specific scorersRAGAS, DeepEval
Experiment + trace platformsLogging traces, comparing runs, human annotation queuesBraintrust, LangSmith, Arize Phoenix
ML lifecycle platformsTracking experiments alongside broader model/agent opsMLflow

Whatever you pick, treat it the same way you'd treat a testing framework: it should run in CI, version alongside your code, and be boring enough that nobody dreads opening it.

08 — Pitfalls

Anti-patterns worth naming

Eval-driven development

Building evaluators before doing error analysis. You end up measuring imagined problems and missing real ones.

Generic off-the-shelf metrics

"Helpfulness," BLEU/ROUGE-style scores rarely track what actually breaks in your specific app.

Near-100% pass rates

If your eval suite always passes, it's not testing anything hard enough to matter. Go find the failures.

Outsourcing error analysis

Handing trace review to someone outside the team severs the feedback loop between seeing failures and fixing the product.

Committee labeling

Averaging several people's pass/fail judgments produces mushy, inconsistent ground truth. One accountable owner beats consensus.

Trusting an unvalidated judge

Shipping an LLM-as-judge without measuring its agreement against human labels first — you've just added a second unverified model.

09 — Getting started

Your first two weeks

A concrete on-ramp if you're starting from zero on a real project.

  • Pull 20–30 real transcripts (or generate realistic synthetic ones if you have no users yet).
  • Read every one yourself. Write a blunt, specific note on anything that feels off — don't filter yet.
  • Group the notes into 4–8 named failure categories. Give each one a short definition and one example.
  • For each category, decide: can code check this, or does it need a judge?
  • Write the deterministic checks first — they're fast wins.
  • For the judge-worthy categories, hand-label 30+ examples pass/fail, then draft and validate one judge prompt per category.
  • Wire the resulting suite into your PR pipeline as a required check, even if it only has a dozen cases at first.
  • Put 30 minutes on the calendar, every week, to read new traces. This is the habit that keeps the whole system honest.
Evals aren't a phase you finish — they're the mechanism by which you keep learning what your system actually does, for as long as it's running.
10 — Sources

What this guide draws on

Every claim above is grounded in current practitioner writing rather than guesswork — this section lists exactly where each part of the guide came from, so you can go deeper on anything that matters most to your project.

Primary sources

Read in full and used directly to shape the structure and claims of sections 01–06.

SourcePublisherUsed for
LLM Evals: Everything You Need to Know Hamel Husain The error-analysis-first workflow, open/axial coding, binary pass/fail rubrics, judge validation, CI vs. monitoring split (§01–03, §06)
A complete guide to RAG evaluation Evidently AI Retrieval metrics, generation metrics, reference-based vs. reference-free evaluation (§04)
The Complete Guide to LLM & AI Agent Evaluation in 2026 Adaline Agent trajectory evaluation, tool-call correctness, trace-level analysis (§05)
CI/CD Integration for LLM Eval and Security Promptfoo docs Where evals run in a pipeline — PR checks, nightly runs, production sampling (§06)

Also consulted

Reviewed for context and to sanity-check the tooling landscape in §07.

Compiled August 2026. This is a fast-moving field — tool names in §07 will age faster than the underlying method in §01–06.