LLM evals: how to catch regressions before users do (AI engineer interview)

LLM evals are automated tests for AI features: a fixed set of realistic inputs, run through your system on every change, with outputs scored by exact checks, an LLM judge or people. They're how teams change prompts and models every week without quietly making things worse (Hamel Husain on evals).

In interviews, it sounds like this: "Your team changes prompts and models every week. How would you catch quality regressions before users see them?"

Why "we test it manually" isn't enough

LLM outputs change in subtle ways. A prompt tweak that fixes one case can break three others, and a new model version can change formats or tone. Reading a few outputs by eye misses most of this. Interviewers want to hear that you treat quality like tests in CI, not vibes.

Step 1: Build a golden dataset

• Collect real inputs from production (with privacy handled), not just inputs you invented.
• Include past failures and edge cases so known bugs can't come back.
• For each input, write an expected output or grading criteria: the right answer, required facts, or what a good response must and must not do.

Start small. A few dozen strong examples beat a thousand vague ones, and the set grows every time something breaks.

Step 2: Run cheap checks first

Many failures can be caught with plain code, which is fast, free and never flaky:

• Output is valid JSON or matches the schema.
• Required fields or facts are present.
• Banned content is absent, such as competitor names or internal URLs.
• The agent called the right tool with valid arguments.
• Length and latency stay within limits.

Step 3: Add a calibrated LLM judge

For qualities code can't check, such as helpfulness, tone or whether an answer is supported by the source, use a model as a judge with a specific rubric.

Judges have biases: they can prefer longer answers or the first option shown. So:

• Write precise rubrics, ideally pass/fail per criterion rather than a 1–10 score.
• Randomise the order when comparing two outputs.
• Calibrate: label a sample by hand and check the judge agrees before trusting it (Anthropic: develop tests).

Step 4: Gate releases in CI

• Run the suite on every prompt, model or retrieval change, the same way unit tests run on code changes.
• Compare against the current baseline, not an absolute score.
• Block merges that regress on key metrics, and review the specific examples that changed.

This turns "the new prompt feels better" into "the new prompt passes 94% of cases versus 91%, with no regressions on billing questions."

Step 5: Watch production too

Offline sets never cover everything real users do.

• Ship risky changes behind a canary or split test and compare live metrics.
• Sample live traces for human review each week.
• Track signals like thumbs down, retries and escalations to a human.
• Add every new failure to the golden dataset, closing the loop.

The eval stack at a glance

The one-line answer

"I'd treat evals like tests: a golden dataset from real traffic, cheap assertions plus a calibrated LLM judge running in CI on every change, and production monitoring that feeds new failures back into the set."

Follow-up questions to expect

• "How do you evaluate an agent, not just one answer?" Score the whole task: did it succeed, how many steps, which tools, and whether any step was unsafe.
• "What if the judge and humans disagree?" Fix the rubric or the judge prompt, and keep humans in the loop for that criterion.
• "How do you avoid overfitting to the eval set?" Keep a held-out set, refresh with new production data, and watch live metrics too.

Pairs well with our guide on how to debug a RAG system, where the same test set does double duty.

Frequently asked questions