Automating Agent Evaluation

Anyone who has tuned a production agent knows the feeling: you change one line of a prompt, try three cases locally and they all look fine, ship it, and the next day customer support explodes. The reason is not that you didn't test carefully — it's that agents are non-deterministic, multi-step, and call tools. The few cases you try manually can't represent its behavioral distribution at all. You're testing luck, not quality.

Automating agent evaluation isn't a nice-to-have; it's the ticket of admission for an agent to reach production. This article covers how to build, from scratch, an evaluation pipeline that runs automatically, scores automatically, regresses automatically, and feeds back automatically, so every change to the agent gets an objective score instead of "I think it got better."

1. Why agent evaluation must be automated

Agents have three properties, and each one is personally killing "manual testing":

• Non-determinism: with the same input, this run takes the right path and the next run might wander through three extra steps and still get it wrong. A conclusion from one manual test no longer holds on the next run.
• Multi-step execution: a single agent task may call the model several times and several tools; if any step drifts slightly, everything after it is wrong. Staring at the final answer, you can't tell which step started going bad.
• Tool dependency: agent behavior is affected by tool returns; when the tool's data changes (inventory, exchange rates, API response structure), the agent's output changes. What passes in your test environment may not hold in production.

Three ways manual testing dies:

1. You can't test enough. Manually running 100 tasks takes half a day. Running them every time you change a prompt? Nobody can sustain that.
2. Subjective and unstable. "This answer is okay, I guess" — change the time, change the mood, and the judgment changes. Two people on the team evaluating the same item often disagree.
3. You can't see regressions. You only looked at a few cases that got better and didn't see the other 30 that quietly got worse. Agent degradation is invisible; by the time users report it, it's already too late.

The goal of automation comes down to one sentence: turn "is the agent good" from a subjective feeling into an objective distribution that you can run out on every change. Without this, your iteration is driving with your eyes closed.

2. The evaluation triad: task set + rubric + scoring method

The premise of automation is that the thing being evaluated is itself well defined. The skeleton of agent eval is a triad.

① Task set (representative inputs). Sample from your real traffic and cover three categories: core scenarios (80%, the work the agent does every day), edge cases (ambiguous phrasing, empty input, multi-turn), and adversarial cases (injection, privilege escalation, persuasion). Don't be greedy with scale — 50–200 carefully chosen items beat 1000 noisy ones. The task set should go into Git; it's part of the product logic.

② Rubric (judgment criteria). For each task, annotate "what counts as success" — not a verbatim answer, but decidable conditions. I consistently write three types:

• Success conditions: the markers that this task achieved its goal (e.g., "the answer includes the order status").
• Key facts: facts the answer must include / must get right (e.g., "refund amount = item price - fee").
• Forbidden behaviors: actions that must never occur (e.g., "must not leak other users' data," "must not call the transfer tool").

③ Scoring method. Rules, LLM-judge, and human review, working together (expanded in the next section).

A task item's structure looks roughly like this:

- id: refund_status_001
  input: "I returned that order yesterday, has it been refunded?"
  context:                       # fixture injected into tools/DB
    user_id: u_8821
    orders:
      - { id: o_1, status: refunded, amount: 199, fee: 10 }
  success:
    - The answer states that o_1 has been refunded
    - The refund amount is 189 (199 - 10)
  forbidden:
    - Calling the transfer tool
    - Mentioning other users' orders
  category: core

Once this triad is in place, the "raw materials" for automation are ready.

3. Scoring methods: rules, LLM-as-judge, human spot-checks

Rule scoring (cheap, deterministic, first gate). Hand anything that can be judged mechanically to rules:

• Format compliance: when JSON is expected, it parses and the fields are complete.
• Correct tool calls: which tool should be called, whether the parameters are right, whether any forbidden tool was called.
• Key-fact hits: regex/substring matching for "order refunded" and "189".
• Efficiency metrics: average step count, token consumption, number of tool calls (step count spiking for the same task = degradation).

The upside of rules is zero cost and reproducibility; the downside is that they can't capture semantic quality. So they are the first sieve, not the final verdict.

LLM-as-judge (semantic quality, the main scorer). Semantic-level quality — whether the answer solved the user's problem, whether there are hallucinations, whether the tone is appropriate — can only be handed to a strong model scoring against a rubric. The key is a strict prompt:

JUDGE_PROMPT = """
You are a strict customer-service QA reviewer. Score the agent's answer against the rubric.

User input: {input}
Agent answer: {answer}
Trace executed by the agent (tool calls / intermediate steps): {trace}

Rubric:
{success_criteria}
{forbidden}

For each rubric item, judge pass/fail and give a reason. Finally output JSON:
{{
  "checks": [ {{"name": "...", "pass": true, "reason": "..."}} ],
  "score": 0-1,
  "verdict": "pass|fail"
}}
If any forbidden item is triggered, verdict must be fail and score must not exceed 0.2.
"""

Three key points: feed the trace to the judge along with everything else (you can't judge whether the process was right from the answer alone), judge each rubric item individually rather than giving an overall score, and a single forbidden trigger is a veto. Also, the judge model should be at least one tier stronger than the agent under test — don't let Haiku referee Sonnet.

Human spot-checks (calibrating the gold standard). Automated scores drift — rubrics get biased, judges have preferences, task sets go stale. Every day, randomly sample 20–50 items for human review and compare with the automated scores; the ones with big disagreements are where the rubric or judge needs fixing. Without human spot-checks, your automated scores will sooner or later diverge from the production experience, and then the whole eval system loses trust.

4. Trace-driven: replayable means regressable

The hard part of an agent error isn't finding it; it's reproducing it. Today that case answered wrong, and tomorrow a re-run is right again — non-determinism means you can't catch it in the act. There's only one solution: record a complete trace on every run.

A trace at minimum contains: each step's model input/output, tool name + arguments + return, stop reason, and tokens and latency. Persist it in a structured store, searchable by trace_id.

{
  "trace_id": "t_a1b2",
  "task_id": "refund_status_001",
  "steps": [
    { "type": "llm", "in": "...", "out": "call=search_order", "tokens": 320 },
    { "type": "tool", "name": "search_order", "args": {...}, "result": {...} },
    { "type": "llm", "out": "Refunded 189; arrives in 1–3 business days" }
  ],
  "result": "...", "passed": true, "score": 0.95
}

Traces have two core uses:

1. Failed-case replay and diagnosis. When a case fails, reading the trace tells you whether retrieval didn't surface anything, the model didn't call a tool, or the tool returned abnormally — minute-level localization, no blind guessing.
2. Turning it into a regression test. Freeze the tool returns in that trace as fixtures; on the next replay, you no longer call real tools (you use the recorded returns) and test the orchestration with the same set of model decisions. This nails a flaky bad behavior down into a stable regression case.

Without traces, agent debugging and regression are empty talk. Traces are the foundation of agent debuggability and the ammunition that lets eval keep growing thicker.

5. CI integration: change prompt/model/tool → run eval → block on score drop

Automation truly flexes its power when wired into CI. The rule is hard: if you change anything agent-related — prompts/models/tools/system prompts — the PR must run eval, and if the score drops past a threshold, it can't merge.

# .github/workflows/agent-eval.yml
on:
  pull_request:
    paths: [ "prompts/**", "tools/**", "models.yaml", "src/agent/**" ]
jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pnpm eval --suite core --n 3        # run each task 3 times and average, to control variance
      - run: pnpm eval compare --base main --threshold 0.03
      # score drop > 3%, or any forbidden triggered → non-zero exit, PR red

A few practical points:

• Trigger only on relevant changes; don't make a README edit run eval — slow and burns money.
• Run each task several times and take the mean (agents have variance, a single score is untrustworthy); focus on the distribution, not a single point.
• Set a significant threshold (e.g., ±3%); treat small fluctuations as noise, don't overreact.
• Split into suites: smoke suite (20 items, seconds, run on every PR), core suite (80 items, minutes, run before merge), full suite (run before release). Pick the tier by risk.
• Reports must be readable. In CI, post which cases broke, the top 10 score changes, and any newly triggered forbidden items — don't just dump a pass/fail.

The moment CI is wired to eval, agent iteration switches from "going by feel" to "having a dashboard."

6. Auto-feeding production bad cases back: let the eval set grow itself

Production is the best source for the eval set. Users are better at finding bugs than you — you just need to automatically collect their pain.

Four feedback signals:

• Low-score sessions: those whose online judge score falls below a threshold automatically enter the candidate pool.
• Retry behavior: similar queries within 10 seconds in the same session (similarity > 0.8) basically mean the previous one wasn't answered well.
• Explicit complaints: thumbs-down / handoff to human / bad review — strong signals go straight in.
• Abnormal traces: abnormally many steps, repeatedly calling the same tool, triggering the fallback — these are the fingerprints of an agent running off the rails.

A pipeline settles them out:

def collect_candidates():
    for trace in query_low_score_or_complaint_traces():
        task = sanitize(trace.input)         # desensitize: strip PII, strip business IDs
        rubric = auto_rubric(trace, judge)   # judge reverse-infers success conditions from the trace
        candidate_pool.add(task, rubric, source_trace=trace.id)

# Humans review the candidate pool once a week; after confirming the rubric is sound, it's merged into the official eval set

Three key steps: desensitize (user data must never enter the eval set), judge reverse-infers the rubric (lowering manual annotation cost), and human final review (someone must look before auto-intake, to prevent noise pollution). Once this turns, your eval set grows with the business instead of going stale the moment it's built.

7. Cost control: eval needs to save money too

Running the full eval once can make the token bill tempting enough to want to turn off CI. A few levers:

• Tiered model usage: use cheap models for CI smoke (fast, enough to judge format/tool correctness), and strong models for refined scoring at key nodes and before release. If what you're judging is "is the process right," Haiku is enough; if it's "is the semantics good," then step up to Sonnet/Opus.
• Sampling: not every PR runs the full set. The smoke suite runs on every PR, the core suite runs before merge, and the full suite runs only before release. If you only changed docs, don't run eval.
• Caching: cache the tested agent's output for identical input (saves tokens when re-running within the same commit), cache the mock returns of tools, and cache the judge's verdict for identical (answer, rubric) pairs. Eval involves a lot of repetition, so caching pays off big.
• Concurrency + batching: eval is a naturally parallel task; judge calls go through the batch API, making tokens per unit cheaper.

Remember: eval is infrastructure that runs long-term, and its own cost must be controllable, or sooner or later the budget gets cut.

Summary

Automating agent evaluation is the key leap that pushes agents from "mystical tuning" to "engineering iteration":

• The triad as foundation: task set (representative inputs), rubric (success conditions / key facts / forbidden), and scoring method — none can be missing.
• Three scoring layers working together: rules as the first sieve, LLM-as-judge for semantics, human spot-checks to calibrate the baseline.
• Traces are the foundation: record a complete trace on every run, so failures are replayable and convertible to regression, giving debugging a basis.
• CI blocks regressions: relevant changes trigger eval, and if the score drops past the threshold it's blocked; tiered suites are picked by risk.
• Production auto-feedback + controllable cost: low-score / retry / complaint cases automatically enter the candidate pool, and after desensitization + human final review are taken into the set; eval itself keeps cost in check with cheap models, sampling, and caching.

Remember this line: an agent tuned by hand is a demo; an agent measured by automation is a product. When every prompt change has an objective score and a regression gate, your iteration speed and quality will completely leave behind teams still "posting in the group chat to ask everyone how it feels" — this isn't exaggeration; it's the gap that opens once automated eval is running and you can never go back.