Eval-Driven Development: Treat Evals as CI

Traditional software has TDD (Test-Driven Development): write tests first to define "correct," then write code that satisfies them. AI engineering needs a corresponding paradigm: EDD (Eval-Driven Development). Build an evaluation set first to define "good," then tune the prompt/model/system to satisfy it, and run the evaluation after every change.

The reason is simple: AI components are nondeterministic and cannot be tested precisely with assertions. Without an evaluation set, you have no way to know whether a change made them better or worse.

1. Why AI Engineering Needs EDD

Traditional code: assert(add(1,2) === 3). It is deterministic: right is right, and wrong is wrong.

AI components: given the same prompt + input, the output may differ on every run. "Good" is not a binary right-or-wrong result, but a quality distribution. Therefore:

• You cannot test it with traditional unit-test assertions.
• After changing a prompt, a few outputs may look better while the system has actually regressed overall because you saw the handful that improved but missed the 30 that got worse.
• Without an objective definition of "good," team arguments over whether a prompt works are decided by whoever speaks the loudest.

The core of EDD is to give "good" an objective, quantifiable definition through an evaluation set, so every change can be judged on evidence.

2. The EDD Workflow

1. Define the task + criteria for "good" → build the evaluation set
2. Write the initial prompt/system → run evaluations and get a baseline score
3. Make a change (prompt/model/parameters) → run evaluations → compare with the baseline
4. Score improves → accept and update the baseline
   Score drops → reject and roll back
5. Continue: add newly discovered bad cases to the evaluation set and iterate

Just as TDD follows "red-green-refactor," EDD follows "set the benchmark-run evaluations-iterate." The evaluation set is your "test suite," and its score determines "pass/fail."

3. The Evaluation Set: EDD's Foundation

The quality of the evaluation set determines whether EDD works. Key principles include:

• Cover core scenarios + boundaries. Do not include only "standard questions"; add edge, ambiguous, and adversarial cases.
• Keep expanding it from real traffic. Production bad cases are the best source of evaluation examples. Sanitize them, add them to the set, and prevent regressions.
• Label "expectations," not "reference answers." AI output varies, so annotate key facts and properties such as "the answer should include X," "the format must be JSON," or "it must not include Y," rather than requiring a word-for-word answer.
• Only make it as large as needed. A well-designed set of 50–200 examples is better than 1000 noisy ones.

The evaluation set should be version-controlled in Git alongside the code because it is part of the product logic.

4. Running Evaluations: Automation Is Essential

EDD requires evaluations to run after every change, so they must be automated:

• Provide a script or command that runs the entire evaluation set + scoring.
• Compare the score with the baseline and produce a report showing which metrics changed and which cases failed.
• Integrate it into CI: when a PR changes a prompt or model, CI runs the evaluation and blocks the PR if degradation exceeds the threshold.

Scoring methods, covered in detail in earlier articles on evaluation, include:

• Rule-based scoring for valid-format rates and keyword matches.
• LLM-as-judge for faithfulness and relevancy.
• Human spot checks for critical cases.

Here is what it looks like in CI:

# .github/workflows/eval.yml
on: pull_request (paths: prompts/**, models/**)
jobs:
  eval:
    - run: pnpm eval --baseline main
    - if: metrics dropped > 3%, fail the PR

5. Key Differences Between EDD and TDD

The biggest difference is that "passing" in EDD is probabilistic and has variance. Therefore:

• Evaluate the distribution or mean, not a single run.
• Small metric fluctuations such as ±1% are noise; do not overreact.
• Set a threshold for "significant change," such as ±3%, and treat only changes beyond it as real.

6. Practical Challenges of Adopting EDD

1. Cold-starting the evaluation set. Without an evaluation set, EDD cannot begin, but building one requires annotation and time. The solution is to start small with 20 core scenarios and expand it while using it.

2. The cost of LLM judges. Using a strong model as the judge across an evaluation set consumes tokens. Use a cheaper model for coarse evaluation in CI, then use a strong model for precise evaluation at critical checkpoints.

3. Aligning metrics with product value. "Faithfulness improved by 5%," but retention did not move. Your metric may not capture real value. Keep calibrating the relationship between evaluation metrics and business metrics.

4. Team habits. EDD requires the discipline that every AI change must run through evaluations. Build that culture through enforced processes + persuasive data.

7. Advanced EDD: Layered Evaluations

Mature teams use layered evaluations:

• Smoke evaluations (10–20 examples, run on every PR, measured in seconds): quickly block obvious regressions.
• Core evaluations (50–100 examples, run before merging, measured in minutes): enforce the quality bar.
• Full evaluations (hundreds of examples, run before release, measured in hours): provide comprehensive validation.
• Online evaluations (canary testing with real traffic after release): measure the actual production distribution.

Choose the layer based on the risk of the change to balance speed and coverage.

8. EDD's Real Value

EDD is not just "stricter testing." It changes how AI development works:

• Changes have evidence. Instead of "I think it is better," you have "the evaluation score improved by +X%."
• You can change things confidently. With evaluations as a safeguard, you can make aggressive optimizations such as switching models or refactoring prompts without fearing silent regressions.
• Better team collaboration. Prompt and model reviews have an objective basis instead of being decided by whoever speaks loudest.
• Knowledge accumulation. The evaluation set itself captures product knowledge: what is correct and where the boundaries lie.

For an AI team without EDD, iteration is guesswork; for one with EDD, iteration is engineering. That gap compounds over time.

Summary

Eval-Driven Development (EDD) is TDD for the AI era:

• The evaluation set defines "good" and forms EDD's foundation by covering scenarios + boundaries and continuously incorporating bad cases.
• Automated evaluations belong in CI; run them after every change and block regressions.
• Understand probability: examine distributions instead of individual runs, and set significance thresholds.
• Use layered evaluations: smoke/core/full/online, selected according to the risk of the change.

Remember this: AI development without evaluations is driving blind; with evaluations, it is driving while watching the dashboard. The investment in EDD, including building the set and integrating it with CI, may look substantial, but it is the dividing line between "intuition-led iteration" and "engineering-driven iteration" for AI products. Once it is in place, your iteration speed and quality will pull ahead of teams that are still "tuning prompts by guesswork."