Gracefully Draining Kubernetes Inference Pods: Traffic Removal, In-Flight Requests, and Termination Budgets

Online inference requests may run for tens of seconds and may return SSE or streamed tokens over long-lived connections. If a process exits as soon as it receives TERM during a Deployment update, clients receive truncated responses. Simply extending terminationGracePeriodSeconds may allow old connections to finish, but new connections will continue arriving. Graceful termination must coordinate both "stop being selected as an endpoint for new traffic" and "stop accepting new work at the application layer," then wait for in-flight requests to reach zero. Traffic-splitting ratios belong in Canary Rollouts for Model Serving; this article covers only the termination contract for an individual Pod.

Model Termination as a State Machine

A service needs at least four states: STARTING, SERVING, DRAINING, and STOPPED. Readiness succeeds only in SERVING. The operation that enters DRAINING must be idempotent and atomically cause readiness to fail, reject new inference requests, and stop taking jobs from internal queues while allowing registered requests to continue. New requests must receive an explicit, retryable status rather than being accepted and canceled halfway through processing.

When Kubernetes terminates a Pod, the grace-period timer starts first, then preStop runs. After the hook completes, the container runtime normally sends TERM to the main process. At the same time, the control plane updates EndpointSlice: a terminating endpoint has ready set to false and exposes the terminating/serving conditions. However, kube-proxy, Ingress, service meshes, and external load balancers all have propagation delays. Observing an EndpointSlice update does not mean the last client has stopped reusing its connection. The application's own admission gate is therefore essential.

The following Pod fragment provides a practical implementation. /app/server drain is a tested subcommand inside the image. It tells the main process to change state through a local Unix socket, then waits ten seconds for traffic removal to propagate:

spec:
  terminationGracePeriodSeconds: 120
  containers:
  - name: inference
    image: registry.example.com/inference@sha256:<digest>
    ports:
    - name: admin
      containerPort: 8081
    readinessProbe:
      httpGet:
        path: /ready
        port: admin
      periodSeconds: 2
      failureThreshold: 1
    lifecycle:
      preStop:
        exec:
          command: ["/app/server", "drain", "--propagation-wait=10s"]

The 120 seconds includes both hook execution and the time available to exit after receiving TERM; these periods are not added together. The process uses the same state machine for the drain subcommand and TERM: stop listen/accept or quickly reject new requests, wait for the in-flight counter to reach zero, flush required audit data, and exit voluntarily before the deadline. Reserve 10 to 20 seconds for sidecars, logs, and runtime cleanup. Do not make the request timeout equal to the entire grace period, or a small delay will trigger KILL.

Separate Responsibilities for Readiness, Liveness, and Startup

Readiness indicates whether the Pod can receive traffic and may fail during model overload or draining. Liveness determines only whether the process is unable to recover by itself; a full queue or slow model loading must not cause a restart. The startup probe covers weight loading and delays the other probes until it succeeds. Probe endpoints must not depend on a downstream database or public model API, because an external disruption could otherwise remove every replica simultaneously. Load-Balancing Strategies determine how requests reach ready endpoints, but connection-pool keep-alive, HTTP/2 streams, and gateway retries still require dedicated testing.

Increment the in-flight counter after authentication succeeds but before the service commits to accepting the job. Decrement it only after the response has been written completely or explicitly canceled. For streaming responses, do not decrement the counter when generation finishes; wait until the result of writing the final frame is known. For asynchronous jobs, the work is complete only when the message lease has been transferred or the result has been persisted. During termination, stop fetching new messages and ensure that another worker can safely replay any unfinished lease.

PDBs Cover Only Some Disruptions

A PodDisruptionBudget can constrain voluntary disruptions that use the Eviction API, such as node drains, but it cannot prevent node failures, OOM events, or direct deletion of a Pod or Deployment. A Deployment or StatefulSet's rolling pace is also controlled by its own strategy. A PDB does not replace maxUnavailable and maxSurge. Keep ready replicas distributed across nodes and availability zones, and treat a single-replica service as having an unavoidable interruption window.

For long-running requests, clients also need a request ID and an idempotency key. A gateway may retry only when no response body has been sent and the operation can be replayed safely. Blindly switching Pods after a stream has partially returned can generate duplicate tokens or duplicate charges. For broader state recovery, see LLM Disaster Recovery.

Prove Lossless Termination Under Continuous Traffic

In a staging environment, continuously send short, long, and streaming requests while performing a rolling update and actively draining nodes. Record the receiving Pod, start and end times, termination reason, and response byte count for every request ID. Success means there are no unexplained losses, duplicate executions, or truncated responses, and the old Pod reaches zero in-flight requests and exits within its budget.

kubectl get endpointslice -l kubernetes.io/service-name=inference -o yaml
kubectl get pods -n ai -w
kubectl logs -n ai <terminating-pod> --timestamps
kubectl get events -n ai --sort-by=.lastTimestamp

Fault injection should cover a preStop timeout, an application ignoring TERM, a 30-second external LB delay, a client disconnecting midway, a stuck sidecar, and complete node loss. Monitor draining_pods, rejected new requests, in-flight requests, forced exits, truncated streams, and the distribution of termination durations, and incorporate them into LLM SLO Monitoring. Seeing zero 5xx responses is not enough; client timeouts and incomplete streams must also be counted.

A Rollback Must Preserve Old Replicas Too

If draining hangs in the new version, pause the rollout first, retain capacity in the old ReplicaSet, and stop deleting additional Pods. After fixing the state machine, test it again with a single canary. You can roll the Deployment back to the previous image and lifecycle configuration, but do not routinely set the grace period to 0 as a temporary recovery measure. Doing so explicitly accepts the loss of in-flight requests.

If an incorrect readiness probe removes every replica at once, restore the previous probe and scale back to the verified replica count first. Confirm that EndpointSlice contains enough ready endpoints before handling stuck Pods. A genuinely unreachable node cannot execute preStop or receive TERM, so critical requests must support idempotent retries at the application layer. Graceful termination is never a substitute for persistence and recovery mechanisms.

Further Reading

• Canary Rollouts for Model Serving
• Load-Balancing Strategies
• LLM SLO Monitoring
• LLM Disaster Recovery