Agent Replay Evaluation: Reproducing Problems with Real Trajectories Instead of Just Rerunning the Final Prompt

Saving only the user's final prompt is usually not enough to reproduce an agent incident. Multi-step tasks are jointly affected by retrieval results, tool returns, cache hits, permission state, and model parameters. A production failure may have happened because the second tool returned an empty array, and the model took the wrong branch based on that; when you rerun the first input alone, the system gets different data and the problem disappears.

What is a replayable trajectory

A replay record should save the task input, system config version, model and tool versions, the structured request and desensitized response of each tool call, authorization decisions, the time order, and the final state. The focus is not to save all raw text, but to save the facts that can affect control flow. Sensitive fields are replaced with stable stand-ins or hashes, ensuring that the same trajectory yields the same structure on every replay.

Tool responses must distinguish "recorded values" from "live values." Offline evaluation uses recorded values by default, to avoid third-party APIs, the latest database data, or network fluctuations affecting results. For tests that need to verify contract compatibility, enable a controlled integration environment separately. Mixing the two types of tests makes the cause of failures hard to determine.

Deterministic does not mean ignoring model randomness

Model output is stochastic, so replay needs to fix the model version, temperature, tool list, and prompt version, and record the random seed or request parameters. Even so, do not assume every token is identical. Evaluation should focus on comparing tool selection, key parameters, authorization results, and final business state, rather than comparing explanatory text word by word.

For tasks that allow multiple correct paths, define an acceptable trajectory set. For example, querying the order first then querying logistics, or querying two systems in parallel once, can both pass; but skipping the permission check or writing directly cannot. Such assertions are closer to real agent behavior than a single "golden transcript."

Turn incidents into regression samples

After each production incident, build a desensitized replay package: the trigger conditions, the context before the failure, the expected safeguards, and the post-fix assertions. Do not just write the incident into a postmortem document. If the problem came from a downstream return format change, pin one copy each of the old and new responses; if the problem came from a tool retry, pin the state query result after the timeout.

Replay packages should have a severity level and an expiration date. After business rules change, some old samples will no longer apply, but they cannot be silently deleted. Mark why it became invalid and which new sample replaced it, so the team cannot shrink the test set just to make the scores look better.

Run strategy and cost control

Divide replays into a small pre-commit set, a daily full set, and a pre-release high-risk set. The small set focuses on recently changed tools, the full set covers major workflows, and the high-risk set specifically includes payments, permissions, data export, and cross-tenant scenarios. Record the time, model cost, and flakiness rate of each set; for flaky tests, fix the test environment first, and do not ignore them because of occasional failures.

A good replay system lets engineers see, before release, "which historical tasks changed behavior due to this change." It turns the multi-step agent from an unexplainable black box into an engineering object that, like a distributed system, can be regressed, localized, and drilled.

Manage the fixture lifecycle

The fixtures that replays depend on need to be reviewed like code. Each recorded response should state the source system, recording time, desensitization rules, and the applicable tool version. When a third-party API return structure changes, do not overwrite the old fixture; add a new one and clarify the compatible behavior for the old version. Only then can you tell whether the new tool is an intentional upgrade or an unintentional break of an existing contract.

For responses that contain time, random IDs, or order-unstable fields, use a normalization layer to replace them with stable placeholders. The normalization layer must run after real recording, rather than hand-fabricating "plausible-looking" responses, otherwise the evaluation loses its connection to production failures. Replay packages involving security incidents should restrict access scope and ensure that any keys, personal data, and internal addresses have been removed from the snapshot.

It is recommended to keep both a human-readable summary and a machine-comparable artifact of the replay results. The summary helps reviewers quickly see plan changes, while the structured artifact supports field-level assertions in CI. When a regression occurs, link to the specific trace, tool version, and fixture hash, so engineers can start localizing within minutes, rather than re-guessing what happened at the time.

When the same trajectory fails occasionally, first compare the configuration, fixture hash, and tool call boundaries, rather than immediately marking the sample as flaky. True randomness should be isolated into controllable variables; if it cannot be isolated, move the scenario to integration tests and annotate the environment dependency. Replay evaluation is worth being a release gate only when failures are explainable.

After each release, also retain a short segment of anonymized real-traffic trajectories for comparison. When offline replays pass but the online distribution changes, these samples help the team quickly fill blind spots, without having to wait for user complaints to learn that the evaluation set has fallen behind.

Further Reading

• Core AI Agent Architecture
• LLM Observability
• Automated Agent Evaluation