Agent Failure Recovery Design: The Execution Order of Retry, Degradation, and Human Handoff

When an Agent fails, the most dangerous default action is "try again." Brief network jitter or rate limiting does warrant a retry, but for parameter validation failures, denied permissions, uncertain tool semantics, and timeouts that may have already written data, repeating the execution only amplifies the impact. Reliable recovery must first classify, then choose a recovery path.

Start the assessment with four questions: Is the error transient? Are the external side effects idempotent? Is the input still valid? Is the risk of continuing acceptable? The four answers determine not a single retry count, but a prioritized recovery table. Write this table into the runtime policy rather than hiding it in the model prompt.

Build an Actionable Error Classification

A practical classification includes: TRANSIENT, such as connection resets and recoverable 429s; DEPENDENCY_UNAVAILABLE, such as a long-term downstream outage; VALIDATION, such as parameters not matching the tool schema; AUTHORIZATION, such as insufficient scope; AMBIGUOUS_SIDE_EFFECT, such as a request timeout where you cannot confirm whether the downstream has already written; and POLICY, such as budget overruns or compliance policy violations.

The one requiring the most caution is AMBIGUOUS_SIDE_EFFECT. You must not resend a create-order or send-email request simply because no response was received. The correct order is to first confirm the result using an idempotency key, a query API, or audit events; if success is confirmed, advance the state; if it is confirmed that nothing happened, then consider a controlled retry; if it cannot be confirmed, escalate to a human.

Recovery Order Should Start from the Lowest Risk

For transient errors that can be proven side-effect-free, use bounded exponential backoff with jitter to avoid a swarm of Agents hitting the recovering dependency simultaneously. Before each retry, re-check the cancellation flag, budget, and tenant quota; old tasks must not burst all at once after the system recovers.

If a dependency is unavailable, first degrade to cached results, read-only mode, queued waiting, or partial conclusions. Degradation is not about hiding the error; it is about explicitly telling the caller which parts have not been verified. For retrieval Agents, you can return existing evidence and mark its freshness; for write Agents, it is better to stop rather than guess.

Parameter validation and authorization errors generally should not be auto-retried. They require re-planning, additional input, or confirmation from someone with permission. Letting the model repeatedly generate the same illegal tool call wastes cost and pollutes the context. The executor should return deterministic errors in a structured way to the planning layer—for example, missing fields, allowed value ranges, and required scope—rather than only a natural-language error message.

Define Explicit Human Handoff Conditions

Human handoff should not be the dumping ground for all unknown errors. Give it actionable conditions: side-effect state cannot be confirmed, consecutive failures exceed the threshold, the budget is about to be exhausted, high-risk resources are involved, evidence conflicts, or policy requires human judgment. The handoff ticket should carry the run ID, the last stable checkpoint, the executed side effects, the error classification, suggested options, and required permissions.

Once human handling is complete, the result must also be written back as a state transition, not as a "handled it" line in a chat tool. Only then can the system learn which errors frequently require escalation and progressively codify the automatable parts as policy.

Use Drills to Verify That Recovery Does Not Harm Users

Rehearse at least four failure scenarios in the staging environment: the tool returns 429, the connection drops after the tool executes, the user permission is revoked during the wait, and the model produces invalid parameters. Record the retry count, external call count, final state, and time to human intervention for each case. The success criterion is not that the task must complete, but that there are no unexplained side effects on any failure path.

Adjust backoff, thresholds, and degradation scope once the recovery strategy has data. "Automatic recovery" without observational data is often just automatically repeating errors.

Give the Recovery Strategy Observable Circuit Breakers

Every error classification should have metrics: attempt count, time from first failure to recovery, idempotent query hit rate, degradation ratio, human handoff ratio, and final side-effect confirmation rate. Looking only at "task success rate" hides many dangerous retries—for example, an order API called three times but ultimately returning success only once.

Set hard circuit breakers for high-risk tools: once the maximum number of uncertain side effects, consecutive policy denials, or the per-tenant error budget is reached, new tasks may only enter the human queue. Alert content should carry the error classification, dependency name, run ID, and the most recent safe-check result—not just send an "Agent failed" message. Only when the team can directly decide from the alert whether to retry, degrade, or pause does the recovery strategy genuinely reduce incident response time.

After each failure drill, archive the actual path along with the policy version. Only a recovery process that can answer "why it stopped" and "who confirmed to continue" is worth reusing on higher-risk tools.

Further Reading

• MCP Timeouts, Retries, and Idempotency
• LLM SLO Monitoring
• LLM Disaster Recovery