Your AI Agent Passed Evals and Still Failed in Production: Building an Eval Harness That Matches Reality

If your agent passed evals and still failed a customer, your eval set was measuring the wrong world. Offline test cases drift away from production because of distribution shift, tool side effects that mocks hide, and multi-turn conversations that single-turn tests never touch. The fix is an AI agent evaluation harness built from real production traces, scored by calibrated judges, and enforced in CI.

You are not alone in this. VentureBeat’s VB Pulse survey, fielded in June 2026 across 157 organizations with 100 or more employees, found that 50% had shipped an AI feature that cleared internal evaluations and then caused a customer-facing failure. Only 5% said they fully trust automated evaluation as it stands, and the most-cited limitation (29%) was that evaluations align poorly with real-world outcomes. VentureBeat itself calls the sample self-selected and directional, so treat the numbers as a signal, not a census. The signal is loud enough.

Why do offline evals diverge from production?

An eval suite is a model of your users. Like any model, it is wrong in specific, predictable ways. Here are the three that bite agents hardest.

Distribution shift

Your golden set was written by engineers during development. Customers write differently: typos, half-sentences, two requests in one message, Arabic and English mixed in the same turn (a very normal pattern for GCC users). They also ask about things that did not exist when the set was written, like a new product, a new policy, or a new error message. Your pass rate stays green because nobody is testing the inputs that actually arrive.

Tool-call side effects

Most agent evals mock the tools. The mock returns a clean JSON payload in 50ms, every time. In production the CRM API times out, returns a paginated list, or succeeds on the second retry after creating a duplicate record. A text-only eval checks whether the final answer sounds right. It cannot see that the agent issued two refunds to get there. This is why we push teams toward trajectory evaluation: score the sequence of tool calls and their arguments, not just the last message.

Multi-turn drift

Single-turn test cases ask one question and grade one answer. Real sessions run 8 to 20 turns, with the user correcting the agent, changing their mind, or pasting in a document halfway through. Context windows fill up, earlier instructions get crowded out, and the agent forgets a constraint it respected in turn two. If your harness never runs a long conversation, it will never catch this class of failure.

What does an eval harness that matches reality look like?

An AI agent evaluation harness has five parts. Skip one and the gap reopens.

The arrows matter more than the boxes. Traces feed the golden set, the golden set feeds CI, and online monitoring feeds new failures back into traces. That loop is the whole point.

How do you turn production traces into a golden set?

Start with what already went wrong. Here is the process we use:

1. Pull 2 to 4 weeks of traces and cluster them by intent (a quick embedding plus clustering pass works, or just tag by the first tool called).
2. Sample from every cluster, not just the big ones. Rare intents are where agents improvise.
3. Add every known incident as a permanent case: the support escalation, the wrong refund, the hallucinated policy.
4. Write the expected outcome, not the expected wording. For agents that means the expected end state (ticket created, refund not issued) and the acceptable tool sequences.
5. Keep full multi-turn sessions as test cases, and replay them turn by turn.
6. Version the set in git and record which version each release was tested against.

Scrub personal data before traces leave production. For UAE and KSA teams, check the set against your PDPL obligations before you hand it to any SaaS eval tool.

How do you calibrate an LLM-as-judge?

An LLM judge is a measuring instrument. You would not trust a thermometer you never checked against a known temperature.

• Label a sample by hand. Have two people mark a few hundred real outputs pass or fail against the rubric.
• Run the judge on the same sample and measure agreement with the humans. Look at the disagreements, not just the score.
• Split vague rubrics into narrow yes/no questions. “Was the answer helpful?” drifts. “Did the agent confirm before cancelling the order?” does not.
• Pin the judge model and prompt version. Change either one and you have a new instrument with a new baseline.
• Re-check on a schedule, because your traffic changes even if the judge does not.

Use deterministic checks wherever you can: tool name matches, arguments validate against the schema, no write call without a preceding confirmation. Save the judge for what code cannot check.

Which tools fit which part of the harness?

For deeper head-to-heads, see our DeepEval vs RAGAS comparison and the Promptfoo alternatives guide if the OpenAI ownership change is a concern for your procurement team.

The practical default for most teams: one tracing platform (Langfuse or Phoenix if you want open source, Braintrust if you want managed) plus DeepEval for CI gates and Ragas where retrieval quality matters.

How should CI gates work for agents?

Treat the golden set like a unit test suite, with two differences: results are probabilistic, and some failures matter far more than others.

• Run each case several times and gate on pass rate, not a single run. One successful run proves the agent can do the task, not that it will.
• Tier your cases. A safety case (never refund without confirmation) is a hard block at any failure. A tone case can tolerate small dips.
• Gate on deltas, not absolutes. “No more than 2 points below the last release” is easier to live with than a fixed threshold that was set before your traffic changed.
• Trigger on every change that can alter behavior: system prompt, model version, tool schema, retrieval index. Model upgrades are the most common silent regression.
• Track cost and latency in the same run. An agent that passes by making 14 tool calls instead of 3 is a regression too.

What should online monitoring catch?

The VentureBeat survey found that only 23% of organizations monitor whether their agent’s answers are right, while most monitor only whether it is running. Uptime dashboards will not tell you the agent started inventing return policies on Tuesday.

At minimum, sample a slice of live sessions and run the same calibrated judges you use in CI. Alert on drops in task success, spikes in tool errors or retries, and new intent clusters you have no golden cases for. Then close the loop: every flagged session goes to a human for triage, and confirmed failures go into the golden set. Within a few weeks, your eval set starts to look like your production traffic, which was the goal all along.

For agent-specific scoring beyond final answers, our agent trajectory testing guide covers how to score tool sequences step by step.

Where to start this week

Pick your ten worst production conversations from last month. Turn them into test cases with expected end states. Run your current agent against them. If it fails more than one or two, you have just measured your evaluation gap, and you know exactly which cases the next release must pass.

When you want help building the full loop, our Agent Trajectory Testing Sprint is a fixed-scope, 5-day engagement: we curate golden trajectories from your traces, deploy the harness on your stack, calibrate the judges, and wire the gates into CI. Talk to us about your agent.