Agent Evaluation Dataset Design: Slice From Real Tasks, Don't Collect Pretty Demos

Many agent projects have a set of "looks fine" test questions before launch: check my order, summarize a document, create a ticket. They are usually written by developers when the system is at its smoothest, so they only prove the model can do ideal tasks; they can't prove the system can handle real users' ambiguous phrasing, missing permissions, dirty data, and tool failures. The value of an evaluation dataset lies not in the number of questions, but in whether it covers real decision costs.

Slice from production workflows

First list the workflows users actually need to complete, not the model's capabilities. Taking a customer-service agent as an example, workflows might be "verify order status and then explain the delay," "identify refund eligibility and generate a draft," and "escalate unsolvable problems to a human." Then break each workflow down into input, required tools, key constraints, expected state changes, and unacceptable outcomes.

When sampling, keep the hard cases: short sentences, typos, synonymous expressions, mutually contradictory context, outdated documents, insufficient permissions, and cross-tenant resource IDs. Real historical requests can provide the distribution, but they must be desensitized first — stripping keys and personal information — and properly authorized for data use. Don't copy raw user conversations straight into the test repository.

Labels should describe outcomes, not just answers

Traditional QA evaluation often has one standard answer; agents are better suited to verifiable outcome labels. For example, a task's annotation might include: whether the correct tool was chosen, whether the correct parameters were supplied, whether an over-privileged request was refused, whether retries stopped after failure, and whether the next step was explained to the user. This way, even if the natural language the model generates differs, you can still judge whether the business behavior is correct.

Record a risk level for each sample. The failure cost of reading a public document and of creating an external payment are completely different, and they must not cancel each other out in one average score. High-risk samples should have stricter pass conditions and be prioritized for regression on version upgrades.

Cover success, boundary, and recovery paths

A mature dataset usually has at least four slices: normal success, input ambiguity, permission or policy denial, and recovery after downstream failure. Testing only success paths makes tool-call accuracy look high while missing the most expensive errors. For example, when a data export fails, the correct behavior might be to prompt that a scope is missing, rather than trying other tools, fabricating results, or retrying continuously.

For multi-step tasks, annotate intermediate checkpoints. Don't just look at whether the final text is fluent; check whether the agent first found the correct entity, whether it got confirmation before a write operation, and whether it used the latest tool result. The intermediate trajectory helps engineers localize whether the problem is in retrieval, planning, tool schemas, or model reasoning.

Make the dataset a versioned asset

Give each sample a stable ID, creation source, the workflow it belongs to, a risk level, and a last-validated date. Every time you add a tool, change a permission policy, or swap a model, record the dataset version and the evaluation version. Deleting a sample should also have a reason, such as the business process being retired, the original data no longer being available, or the sample conflicting with a new policy.

The dataset must not be maintained only by the model team. Support, ops, security, and product staff know best which failures truly affect users, and they should regularly contribute "this week's incidents that should never happen again." Turn these incidents into desensitized regression samples, so the evaluation set grows with the product instead of staying stuck at the demo stage.

Design the mix of evaluation samples

At the start, you don't need to pursue a statistically perfect distribution, but avoid having all samples come from the most common low-risk paths. You can stratify sampling by workflow value and risk: part covers high-frequency success tasks, part specifically covers high-loss failures, and part reserves recently launched features and recent incidents. Each batch records its sampling time window, to prevent the team from evaluating a new system with outdated inputs after a product redesign.

The same task should also keep multiple expression variants. A user might say "help me check my order," or might provide an order number, a screenshot summary, or a stretch of messy context. Grouping these variants into the same task family lets you test whether the system only memorized the test wording. Within a task family, keep at least one minimal-information sample and one conflicting-information sample, to verify whether the agent will proactively clarify rather than force a guess.

Evaluators should also be tiered. Outcomes that can be judged from database state, tool trajectories, or schema validation are automated first; samples involving explanation quality, risk communication, and multiple reasonable paths are then handed to human spot-checks or a calibrated LLM judge. Write the evaluator's own version and confidence into the results, so an unstable judgment isn't mistaken for a model regression.

A bootstrappable minimum set doesn't need to be large, but every workflow should have four kinds of samples: success, clarification, refusal, and dependency failure. First get these samples running stably in CI, then expand from production feedback week by week. Eighty high-value samples that can be stably maintained usually prevent production regressions better than 800 generic questions that nobody owns.

Before release, randomly spot-check the source and annotations of new samples, confirming the questions don't leak answers, don't reuse sensitive user data, and that a reviewer can independently understand the pass conditions. Dataset quality itself needs review; don't abandon governance just because it's "only for testing."

Further reading

• The Core Architecture of AI Agents
• Automating Agent Evaluation
• LLM Observability