If your agent works on Tuesday but breaks on Thursday after a prompt tweak, you do not have an agent problem. You have an evaluation problem. Non-deterministic systems demand deterministic test harnesses, yet most teams still ship agents the way they ship landing pages: eyeball the output once and deploy.
In this deep-dive I show you how to build a production eval program from scratch: failure taxonomy, golden datasets, calibrated judges, a DeepEval tutorial with real code, and a CI gate that blocks bad merges.
1. The Problem: Agents Fail Silently
LangChain’s State of Agent Engineering report found that only 27% of teams run evals before every deployment. That means nearly three out of four agent deploys go out with no automated quality check. Traditional software would never accept that: the same input must produce the same output. Agents break that contract because they generate responses from a probability distribution, maintain state across turns, and execute real actions with real consequences.
Worse, most agent failures happen silently, buried in reasoning steps nobody monitors. Your agent can return the correct final answer while calling the wrong tool, leaking data mid-trajectory, or getting lucky on one run and looping forever on the next. Eyeballing the final output catches none of that.
The 4 failure modes that haunt production agents
Tool misuse (37%): the agent calls the right tool with wrong parameters. Hallucinated actions (28%): it claims to have done something it never executed. Infinite loops (19%): it retries without recognizing failure. Scope creep (16%): it acts outside its authorized scope.
Every one of these looks fine if you only read the final message. That is why trajectory evaluation, not answer checking, is the core discipline of this guide.
2. Minimum Concepts: Offline vs Online, Deterministic vs Judge
Before writing code you need four distinctions. Get these wrong and every eval you build measures the wrong thing.
Offline vs online evals
Offline evals run on a fixed dataset before deploy: regressions, prompt comparisons, model swaps. Online evals score live production traces continuously: drift, new failure clusters, quality decay. You need both. Offline gates the merge, online watches what the gate missed.
Deterministic checks first
Tool name correct? Required parameters present? Valid JSON? PII leaked? None of this needs an LLM judge. Deterministic checks are free, instant, and exact. Reserve the expensive judge for what only semantics can score: reasoning quality, task completion, tone.
LLM-as-a-judge, calibrated
A judge without a rubric is an opinion with an API key. Always score with a multi-criterion rubric with anchor descriptions, test for position and length bias, and calibrate against 500+ human-labeled cases before trusting aggregate metrics. Re-calibrate every time the judge model, the prompt, or the system changes.
Trajectory, not just answers
For multi-step agents, score the path: step efficiency, tool correctness, argument correctness, plan adherence, reasoning quality. A correct answer reached through a wrong tool call is a failure wearing a success costume.
The golden rule of eval economics
Run the cascade: deterministic checks on every request, a cheap classifier on what passes, the LLM judge only on what needs semantic scoring. Teams that send everything to the judge pay 10x the cost and wait 10x the latency for worse signal.
3. Tutorial: DeepEval + LangSmith in 5 Steps
The stack I recommend for most teams: DeepEval for pytest-native scoring in CI, LangSmith for dataset management, tracing, and production observability. DeepEval is open source at github.com/confident-ai/deepeval with 15k+ stars and 50+ pre-built metrics. LangSmith is documented at docs.langchain.com/langsmith. One handles the gate, the other handles the memory.
Step 1 — Install and define failure modes from real incidents
Start from past production incidents, not hypotheticals: wrong tool selected, hallucinated price, infinite tool loop, leaked PII. Each incident becomes one eval case. Install the framework first:
pip install deepeval pip install pytest
Step 2 — Write trajectory evals as pytest tests
DeepEval integrates with pytest, including parametrize and parallel flags. Below is the pattern: one test case per incident, a deterministic metric for tool correctness plus a judge-backed metric for task completion. Assert on the trajectory, not just the final string:
import pytest
from deepeval import assert_test
from deepeval.test_case import LLMTestCase
from deepeval.metrics import TaskCompletionMetric
def test_refund_agent_uses_correct_tool():
test_case = LLMTestCase(
input="Refund order #4821, customer was charged twice",
actual_output=agent.run("Refund order #4821"),
expected_output="Refund issued for order #4821 via refund_tool",
tools_called=["refund_tool"],
expected_tools=["refund_tool"],
)
metric = TaskCompletionMetric(threshold=0.7)
assert_test(test_case, [metric])Step 3 — Run the suite locally
One command runs the whole regression set. Green here means the prompt tweak did not silently break the five incidents you already fixed once:
deepeval test run test_agent.py
Step 4 — Gate every merge in CI
An eval program that does not block a merge is advisory forever. Wire the suite into GitHub Actions so a failed eval blocks the PR. Alert on borderline cases within 2 points of the acceptance bar:
jobs:
agent-evals:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- run: pip install deepeval pytest
- run: deepeval test run test_agent.pyStep 5 — Observe production with LangSmith tracing
CI gates the merge, tracing watches what the gate missed. Enable LangSmith tracing with two environment variables, annotate failing production traces, and promote representative failures back into the golden dataset every week. That closed loop is what turns incidents into regression coverage:
LANGSMITH_TRACING=true LANGSMITH_API_KEY=lsv2_your_key_here
When to add Braintrust or Inspect AI
Add Braintrust when dataset management overhead justifies its Pro tier and you want model-agnostic scoring with sandboxed custom scorers. Reach for Inspect AI when you operate in regulated or public-sector environments and must benchmark across five or more model providers in one run. Most teams converge on two tools: one CLI library for CI speed plus one hosted dashboard for audit and review.
Sources: DeepEval on GitHub · LangSmith docs · Braintrust · Mastra LLM Evaluation guide (Jul 2026)
4. Common Mistakes That Kill Eval Programs
I have seen the same five mistakes end more eval programs than any technical limitation. All of them are process failures disguised as tooling problems.
âś… Do this
- • Build the golden dataset from production incidents, stratified across task types
- • Calibrate the judge against 500+ human-labeled cases first
- • Score trajectories: tools, arguments, plan adherence, reasoning
- • Block merges on failed evals in CI
- • Mine production traces weekly and promote failures into the dataset
❌ Stop doing this
- • Trusting public benchmarks as a proxy for your workflows
- • Running an uncalibrated judge and quoting its scores as truth
- • Checking only the final answer of a 12-step agent
- • Keeping evals advisory so regressions ship anyway
- • Treating eval as a one-time project instead of a weekly loop
Golden rule
Benchmarks compare models. Evals protect your product. SWE-bench and WebArena tell you which model is stronger in general; only your incident-derived dataset tells you whether Thursday’s prompt tweak broke refunds. Ship seven things: golden set, calibrated judge, deterministic floor, CI gate, statistical math on deltas, production observability, closed loop. Refine after.
Conclusion
The gap between demo agents and production agents is not framework choice. It is evaluation discipline. Deterministic checks on every request, a calibrated judge for semantics, trajectory scoring instead of answer checking, a CI gate that actually blocks, and a weekly loop from production failures back into the dataset.
Start this week: install DeepEval, convert your last five production incidents into test cases, and wire one CI job. That single afternoon buys you more reliability than any model upgrade. Then add tracing, then calibration, then the rest of the seven.
Your 5-step checklist
Build
- • pip install deepeval + pytest
- • Golden set from real incidents
- • Trajectory tests, not answer checks
Gate
- • deepeval test run in CI
- • Failed eval blocks the merge
- • Alert on borderline scores
Observe
- • LangSmith tracing in prod
- • Calibrate judge (500+ labels)
- • Weekly trace-to-dataset loop
