Disaster Recovery and Resilience for LLM Applications

When a traditional web application goes down, it can at least return a 500, and users can refresh and retry. An LLM application fails more awkwardly: if the model API times out, you cannot produce an answer; if the vector database goes down, retrieval returns nothing; if a tool call fails, your Agent gets stuck halfway through a step. The user sees a loading spinner for thirty seconds, followed by "Something went wrong. Please try again."

An AI application's ability to stay "online" depends on a long chain: the model API, vector database, retrieval system, tools, your own business services, and the network. Every link can be a single point of failure. Treating them as reliable by default leaves your production environment exposed. This article explains disaster recovery and resilience design for LLM applications, from a complete map of failure sources and the four-part resilience toolkit to multi-provider recovery, data resilience, monitoring, and chaos drills.

1. A Complete Map of AI Application Failure Sources

Inventorying everything that can fail is more important than immediately writing code. Failure sources in LLM applications fall into roughly six categories:

User request
 → Gateway / Edge           Network jitter, rate limiting, regional outages
 → Model API                Timeouts, 429 rate limits, 503s, context limit exceeded, content rejected
 → Vector DB / retrieval    Query timeouts, corrupted indexes, no results, embedding service outage
 → Tool calls (MCP/functions) Upstream API outage, invalid JSON, permission failures, slow responses
 → First-party services     DB/Redis/Queue; conventional failures that can still leave the Agent unable to respond
 → Memory / session storage Loss = user "amnesia"; a worse experience than an error

Each failure category has different symptoms and requires a different response. A model 429 calls for retries with backoff; a vector database outage calls for keyword search as a fallback; a tool failure should prompt the Agent to take another path or degrade gracefully; a session store outage should fall back to a read-only cache. Resilience design starts by listing every dependency in the chain and documenting what happens when each one fails.

Practical tip: Draw a dependency graph and write three things next to every external dependency: What does failure look like? Can the system degrade gracefully? Is there a fallback? That graph is your disaster recovery design document.

2. The Four-Part Resilience Toolkit: Degradation, Circuit Breaking, Retries, and Fallbacks

These four patterns are mature in traditional microservices. LLM applications should adopt them too, but their parameters must be tuned for model behavior.

1. Retries (exponential backoff + jitter)

Model API 429s, 503s, and network jitter are transient, so retries are highly effective. The key is exponential backoff + random jitter; otherwise, every client retries at the same time and creates a thundering herd. Retry only idempotent operations and transient errors. Retry 5xx responses, 429s, and timeouts; do not retry 4xx responses other than 429. A 400 indicates a parameter problem, and retrying it ten thousand times still produces a 400.

async function withRetry<T>(
  fn: () => Promise<T>,
  opts: { retries?: number; baseMs?: number } = {},
): Promise<T> {
  const { retries = 3, baseMs = 500 } = opts;
  let lastErr: unknown;
  for (let i = 0; i <= retries; i++) {
    try {
      return await fn();
    } catch (e: any) {
      lastErr = e;
      const retryable = e.status === 429 || e.status >= 500 || e.code === 'ETIMEDOUT';
      if (!retryable || i === retries) throw e;
      const jitter = Math.random() * baseMs;
      const delay = baseMs * 2 ** i + jitter;
      await new Promise((r) => setTimeout(r, delay));
    }
  }
  throw lastErr;
}

2. Circuit breaking (short-circuiting consecutive failures)

Retries handle transient disruption; circuit breakers handle sustained failures. If a model API has already been down for 30 seconds, retrying every request three times only adds latency and cost. When consecutive failures reach a threshold, the circuit breaker opens. Subsequent requests fail fast without reaching the downstream service, giving that service time to recover while protecting your own latency.

3. Degradation (switching to a smaller model or bare-bones mode)

If the primary model is down or times out, degrade to a faster, cheaper model. If that model also fails, switch to "bare-bones mode": turn off RAG and use plain chat, or disable multi-turn tool calls and provide a single-turn answer. Degradation does not mean returning an error. It means providing a rougher but still usable experience.

async function answer(query: string): Promise<string> {
  // Primary path: large model + RAG + tools
  try {
    return await withRetry(() => callLLM('claude-sonnet', query, { rag: true, tools: true }), { retries: 2 });
  } catch (e) {
    metrics.degradation('sonnet-rag-failed');
  }
  // Degradation level 1: small model + RAG, no tools
  try {
    return await withRetry(() => callLLM('claude-haiku', query, { rag: true, tools: false }));
  } catch (e) {
    metrics.degradation('haiku-rag-failed');
  }
  // Degradation level 2: plain chat, no retrieval
  return await withRetry(() => callLLM('claude-haiku', query, { rag: false, tools: false }));
}

4. Fallbacks (cache / static responses)

Even if every path is down, do not show users a stack trace. A semantic cache stores answers to frequent questions in a KV store or vector database so that a similar query can hit the cache directly next time. It can be a lifesaver when the model is unavailable. If even that is unavailable, return an honest fallback message: "The service is temporarily unavailable. This has been recorded. Please try again later," and trigger an alert.

Practical tip: Combine the four components into a "failure ladder": retry → degrade after the circuit breaker opens → use a fallback if degradation also fails. Retries without degradation only amplify failure latency; degradation without a fallback still ends in a 500 at the last mile.

3. Multi-Model, Multi-Provider Disaster Recovery

Betting everything on one model provider ties your production SLO to someone else's status page. Active and standby paths are central to AI application disaster recovery:

Primary: OpenAI (gpt-4o)
  ↓ Consecutive failures / circuit opens
Backup 1: Anthropic (claude-sonnet)
  ↓ Consecutive failures
Backup 2: Self-hosted (vLLM + Qwen, private network, self-managed SLA)

The key to switching between primary and backup paths is not simply calling three SDKs. It is abstracting a unified invocation layer so that upper layers do not depend on a specific provider. On failure, it should automatically fail over according to priority and record that a provider is currently unhealthy. Combined with a circuit breaker, it can use half-open probes a few minutes later to detect recovery.

const providers = [
  { name: 'openai', call: (q: string) => openai.chat(q), circuit: new CircuitBreaker() },
  { name: 'anthropic', call: (q: string) => anthropic.messages(q), circuit: new CircuitBreaker() },
  { name: 'self-hosted', call: (q: string) => vllm.generate(q), circuit: new CircuitBreaker() },
];

async function resilientComplete(query: string): Promise<string> {
  for (const p of providers) {
    if (p.circuit.isOpen()) continue; // Skip while the circuit is open
    try {
      const out = await withRetry(() => p.call(query), { retries: 2 });
      p.circuit.recordSuccess();
      return out;
    } catch (e) {
      p.circuit.recordFailure();
      metrics.providerFailover(p.name);
    }
  }
  throw new Error('all providers unavailable');
}

Several practical considerations:

• Prompts must be compatible across providers. Different vendors use slightly different syntax for system / tool / JSON modes. The unified invocation layer should handle those differences so that formatting does not break after failover.
• The degradation chain is also a cost chain. Use an expensive flagship model as the primary and a cheaper model as the backup. Costs then fall during failover, with slightly worse quality but no interruption, which is exactly what you would expect.
• A self-hosted fallback is the ultimate insurance. Even if it can run only a small 7B model, it may be your only way to remain online during a widespread outage.

4. Data Resilience: Vector Databases, Memory, and Sessions

Model outages can be routed around; lost data cannot. LLM applications have three categories of core data, each requiring a different recovery strategy.

Vector database / knowledge base: This is the lifeblood of RAG. Rebuilding an embedding index may take hours and cost thousands, so you cannot recompute it on the spot after losing it. The core approach is snapshots + replicas: regularly snapshot data to object storage (S3/OSS), enable replicas on the production cluster (Pinecone/Qdrant/Milvus all support them), and restore from a snapshot instead of re-embedding the original documents. Version important changes such as document additions and deletions so that a failure can be rolled back to the previous consistent state.

Memory / conversation history: Losing a conversation halfway through makes the AI appear to have "forgotten," which is a worse experience than an error. Treat session storage as a core business database: persist it to a DB instead of keeping it only in memory/Redis, and use Redis only as a hot cache. Use idempotency keys for critical operations such as writing messages and updating memory so retries cannot create duplicate writes:

// Use a stable idempotency key for each message so network retries do not create duplicates
await db.message.upsert({
  where: { idempotencyKey: `${sessionId}:${messageId}` },
  create: { idempotencyKey: `${sessionId}:${messageId}`, role, content, createdAt },
  update: {}, // Skip if it already exists
});

Replayable requests: An Agent task may run for 10 steps and fail at step 5. If the entire task is not reentrant, such as when it has already sent an email, retrying it sends the email twice. Design Agent tasks to be reentrant: persist a checkpoint after every step and resume from the previous step after a failure instead of starting over. This imposes stricter idempotency requirements than traditional microservices because an Agent has a longer chain of side effects.

Practical tip: Ask yourself, "Would replaying this step right now cause a problem?" If you cannot answer "no," the design is not finished. In Agent workflows, replay safety is far more important than speed.

5. Monitoring: Health, Error Rates, and Degradation Triggers

After implementing resilience, you still need to be able to see it working. Monitoring for an LLM application should cover three levels:

Dependency health: Give every external dependency, including model providers, the vector database, and tool backends, an independent health probe. Probe them periodically and expose their status through /health. Circuit-breaker state, whether traffic is currently using the primary or backup path, and the current degradation level should all be observable metrics.

Error rates and error types: Do not look only at the overall error rate. Break it down by model timeouts, 429s, content-safety rejections, vector database timeouts, and tool failures. Track the trend for each error type independently so problems can be located quickly.

Degradation trigger rate: This is a critical metric unique to LLM applications. A high degradation trigger rate means the primary path remains unhealthy. Users may still receive answers through fallbacks, but their experience is quietly deteriorating. Alert on the degradation rate, number of failovers, and duration of open circuits. Do not wait for user complaints to discover that the primary model has been down for half an hour.

Example alert rules in Prometheus style:

# Model provider error rate > 10% over 5 minutes
rate(llm_provider_errors_total[5m]) / rate(llm_calls_total[5m]) > 0.1

# Sudden spike in degradation triggers (primary path failure)
rate(degradation_triggered_total[5m]) > 1

# Circuit breaker open for more than 2 minutes
circuit_breaker_open_seconds > 120

6. Chaos Drills: Proactively Injecting Failures

Resilience designs that are never tested might as well not exist. The core of chaos engineering is to create a failure proactively before it occurs for real. LLM applications should focus on injecting these failure types:

• Model timeout/outage: Use a proxy layer to delay requests to the model provider by 30 seconds or return a 503 directly. Verify that circuit breaking and failover trigger as expected and that users receive a degraded answer.
• Empty or timed-out vector database: In staging, point the vector database connection to an empty instance. Verify that RAG degrades gracefully to plain chat instead of returning an error.
• Tool-call failure: Mock the tool backend to return a 500 or invalid JSON. Verify that the Agent can take another path after the tool fails or give the user appropriate feedback.
• Unavailable session storage: Disconnect the DB and verify that the service can still provide limited functionality through a read-only cache instead of collapsing entirely.

Before each drill, document the expected behavior: the circuit breaker should open within N seconds, the degradation rate should rise to X, and the user should see Y. Then inject the fault and compare the actual metrics. Any mismatch between expected and actual behavior is a bug in the resilience design. Disaster recovery that has never been exercised exists only in a PPT.

7. Incident Runbooks and Postmortems

Production incidents will happen. Whether you can mitigate them quickly depends on having an incident runbook. Create a one-page runbook for every critical failure scenario, such as a total model-provider outage, vector index corruption, or a failed primary session database. Include symptoms, scope of impact, mitigation steps such as switching to a backup, rolling back, or enabling degradation, verification steps, and the owner.

After an incident, you must conduct a blameless postmortem that focuses on three things:

1. Mean time to detect (MTTD): How long did it take from the start of the failure until an alert fired? If it took too long, add monitoring.
2. Mean time to recovery (MTTR): How long did it take from the alert until service recovered? If it took too long, add automation such as automatic failover or one-click degradation.
3. Resilience gaps: Which previously overlooked dependency or boundary did the incident expose? Add it to the design and to the chaos-drill scenarios.

Turn every incident into a new degradation path, a new alert rule, or a new drill scenario. That is how a system becomes more resilient over time.

Summary

• Map failure sources first: Model APIs, vector databases, retrieval, tools, first-party services, and networks. Draw a dependency graph and document what happens when each dependency fails.
• The four-part toolkit is the foundation: Retries (exponential backoff + jitter) → circuit breaking (fast fail after consecutive failures) → degradation (switch to a smaller model/bare-bones mode) → fallbacks (cache/static responses). Combine them into a failure ladder.
• Use multiple providers with primary and backup paths: Abstract a unified invocation layer, fail over automatically, and use a small self-hosted model as the ultimate fallback.
• Data resilience is nonnegotiable: Use snapshots + replicas for vector databases, make sessions and memory idempotent and reentrant, and design Agent tasks to be replayable.
• Monitor + inject chaos + review incidents: Watch degradation rates and open-circuit duration, not only error rates; regularly inject failures in drills; and conduct blameless postmortems to address gaps.

Resilience is not a one-time architecture diagram. It is an ongoing habit: ask "What happens if it fails?" for every new dependency, and feed every incident's answer back into the system. What keeps an AI application reliably online during failures is not a smarter model, but a more careful engineer.