Durable Workflows for Long-Running Agents: Resumable Checkpoints, Timeouts, and Idempotency Boundaries

"Long-running" is not just turning up the HTTP timeout. An Agent that waits for a user-uploaded file, polls a third-party approval, processes thousands of records, or produces a report in stages will inevitably hit process restarts, duplicate messages, jitter from dependency services, and tenant quota changes. If runtime state lives only in memory, every deploy forces the task to start over, or worse, to re-write from the middle.

A reliable design starts from a persistence boundary: define every place that can safely stop as a checkpoint. A checkpoint is not a full dump of "what the model has thought so far"; it is the minimal set of facts needed to resume the next step, such as the processed pagination cursor, the current plan version, the idempotency keys of successful tool calls, the correlation IDs of awaited events, and the last verifiable output.

Break the task into replayable steps

A batch-import Agent can be split into five steps: prepare, shard, execute, summarize, and deliver. After each step, write an immutable event or a versioned snapshot, and at the start of the next step read it from storage instead of relying on the current worker's memory. For model reasoning results, persist the structured decisions and an input hash; for external writes, persist the request summary, idempotency key, and the resource ID returned downstream.

A sketch data model could look like:

type RunCheckpoint = {
  runId: string;
  step: "PREPARE" | "SHARD" | "EXECUTE" | "SUMMARIZE" | "DELIVER";
  version: number;
  cursor?: string;
  planHash: string;
  completedToolKeys: string[];
  waitForEvent?: string;
  updatedAt: string;
};

On resume, first verify that planHash and the task version are compatible, then read the incomplete steps. When a code change alters step semantics, do not pretend old snapshots are still safe; provide a migration function or send the task to human review. Making the schema version a checkpoint field prevents the silent failure of "a new worker interpreting old state with new assumptions."

Idempotency keys matter more than retry counts

Retries are not the problem; duplicated side effects are. For actions like creating tickets, charging money, sending notifications, or writing files, generate a stable idempotency key before the call, for example composed from runId + step + resourceId + inputHash. After a service recovers, if the same key already has a result, reuse it instead of calling downstream again.

Do not use a random UUID as the idempotency key for each retry; that prevents the downstream from recognizing duplicates. Do not rely only on "check-then-write" either, because two workers may pass the check at the same time. The final deduplication boundary should be provided by the downstream API, a unique constraint, or a transactional outbox. For legacy systems without idempotency support, at minimum route the call requests and results through a controlled queue, and write human compensation into the flow.

Timeouts must distinguish three semantics

The overall task deadline, the per-step timeout, and the external-call timeout should not be collapsed into a single number. A 30-second external-call timeout may permit a retry; a three-day wait for a user may resolve to a cancellation; a task that exceeds its budget should stop reasoning and emit current progress. Every timeout should be recorded as an event, carrying its trigger condition and the next state, so operators can tell whether the cause is a slow dependency, an unresponsive user, or a design error.

For queued systems, also set tenant-level concurrency and rate limits. A long-running task must not grab all workers the instant it resumes, especially since model calls and database scans amplify peak load. A resume policy should check quotas, rate limits, and cancellation flags before claiming the next chunk of work.

Use failure drills to verify whether checkpoints actually work

The most effective test is not waiting for a production outage. Prepare a ten-step test task and deliberately terminate the worker after steps 3, 6, and 9; redeliver the same message; let one external call first time out and then succeed. Verification points include: completed side effects are not duplicated, cursors do not move backward, task state does not skip approvals, and the final summary can explain the reason for each resume.

You can record a recovery metric in a staging environment: the time from worker restart to the task resuming, and the number of external calls avoided by idempotency hits. These turn the slogan of "supports recovery" into an observable capability.

Cleanup strategy decides how long recovery data survives

Persistence is not indefinite retention. Checkpoints, tool summaries, and intermediate artifacts need retention windows set by task type: user-downloadable deliverables may be kept for days, audit records longer per compliance, while raw prompts and temp files should be kept shorter with least-privilege access. Cleanup tasks must also understand run state; you cannot delete the only context for a task that is waiting on approval right before it resumes.

A good approach is to write the final summary and linked resources first when the task reaches a terminal state, then asynchronously clean up the regenerable intermediate data; record the cleanup version and reason before deletion. When testing the recovery mechanism, also test cleanup boundaries: completed tasks should not be re-run due to cleanup, and incomplete tasks must not lose their only checkpoint due to over-cleanup. That way long-running Agents do not convert reliability problems into storage and privacy problems.

Further reading

• AI Agent Architecture
• Agent Concurrency and Scheduling
• LLM Disaster Recovery