Agent Online Evaluation Metrics: Connecting Offline Scores to Real User Outcomes

Offline evaluation answers "how does it perform on known samples," while online evaluation answers "do real users complete tasks faster and more safely as a result." The two cannot substitute for each other. Offline sets may miss new user intents, and online data is affected by traffic structure, product UI, and external dependencies, so we need a layered set of metrics that can explain the differences.

First define user task success

Do not treat "the model output a piece of text" as success. For query tasks, success may mean the user obtained the correct data without rephrasing the question; for write operations, success means the target state actually changed and the user confirmed it was correct; for high-risk requests, a correct rejection is also success. Every major workflow should have a business completion signal, not just chat turn count.

Treat whether the user keeps asking follow-ups, clicks human help, undoes an action, or resubmits within a short window as auxiliary signals, but do not rely on these proxy metrics alone. A user not asking follow-ups may be satisfied, or may have given up. Critical flows still need confirmation through backend state or sampled manual review.

Look at quality, safety, and cost together

The online dashboard should have at least three layers. The quality layer includes task success rate, tool parameter correction rate, human takeover rate, and post-confirmation undo rate. The safety layer includes the correct rejection rate of over-privileged requests, the coverage of write operations requiring confirmation, cross-tenant rejections, and sensitive tool call volume. The cost layer includes tokens per successful task, number of tool calls, retries, and P95 latency.

Slice these metrics by tool, tenant type, model version, and content version. Aggregate averages often hide problems: a new model may make low-risk summaries faster while doubling the human takeover rate of some export tool. Only when you can localize to a workflow do you know whether to roll back the model, fix the tool schema, or adjust product guidance.

Gradual rollout is not just about error rates

When releasing a new agent version, start with controlled users or low-risk workflows. Write down the thresholds for expanding traffic and rolling back in advance, such as the false rejection rate for high-risk tasks, repeated write events, P95 latency, and the human takeover ratio. Stop expansion automatically when a threshold is triggered, rather than analyzing after user complaints.

Keep control samples of the old version during the rollout. Differences between versions for the same intent can be supplemented through anonymized splitting or offline replay. Do not skip the control in pursuit of faster expansion, because agent problems usually surface only on long-tail user inputs and dependency anomalies.

Bring online findings back to the offline set

The most valuable product of online monitoring is not pretty charts, but newly added regression samples. An anomalous retry, a wrong clarification, a parameter a user successfully corrected—all should enter the corresponding offline slice after desensitization. This way the offline dataset gradually approaches the real distribution, and online metrics gain better prevention before the next release.

The agent metrics system ultimately serves decisions: which tasks can be automated, which must be confirmed, which version deserves expansion, and where to pause. As long as the metrics let the team make these clear choices, they are closer to production value than any single model score.

Event design determines whether metrics are trustworthy

Online metrics depend on consistent event semantics. A task must at least have start, plan generation, each tool call, authorization decision, user confirmation, final business state, and human takeover events, all sharing a traceId. Do not infer task completion from web clicks or model logs alone, because a task may span pages, workers, and async queues.

Distinguish denominators when computing statistics. The denominator for tool success rate is the actual tool call count, for task success rate it is the started and adjudicable user tasks, and for confirmation coverage it is all write operations requiring confirmation. List the non-adjudicable tasks separately; do not silently remove them from the denominator. Otherwise, seemingly improved metrics may just be excluding difficult tasks.

Set minimum sample sizes for model versions and policy versions. When samples are insufficient, show confidence intervals or "observing," and do not expand traffic based on a small number of successes. Every alert must be traceable to specific trace samples; metrics are only the entry point for discovering problems, and in the end you still need to look at the actual tool trajectory and user outcomes.

The metric definitions themselves must be versioned. When you change the criteria for judging a "successful task," show both the old and new definitions for a period to avoid the appearance of a sudden improvement or deterioration. The dashboard should mark the release timestamps of models, prompts, tool schemas, and policies, so that trend changes can be correlated with specific releases rather than explained by gut feeling in weekly reports.

For tasks that affect user rights, periodically compare automated metrics against sampled manual conclusions. If the two consistently diverge, prioritize checking instrumentation and adjudication logic, rather than directly trusting the better-looking numbers. Reliable online evaluation must first ensure that the thing being measured has not been incorrectly simplified.

All thresholds and exceptions should be written into release records for later postmortem.

In the first week after launch, assign a fixed owner to review daily anomalous samples and sync conclusions to the model, tool, and product owners. Metrics that no one consumes will not improve the system; only metrics that can trigger specific fixes and regression tests are worth collecting long term.

Further Reading

• LLM Observability
• Automated Agent Evaluation
• MCP Audit Logging Design