GPU Inference Capacity Planning: Token Throughput, Memory Models, Peaks, and Failure Redundancy

"How many GPUs do we need for one million requests per day?" has no reliable answer. Two requests may each count as one, but one could be a classification task with a 200-token input and a 50-token output, while the other could include 20K tokens of context and generate 2K tokens. The former can use dynamic batching efficiently; the latter occupies KV Cache for an extended period and slows other requests in the same batch. GPU inference capacity planning must start with workload distributions and service objectives, not work backward from average daily request counts or vendor-reported peak throughput.

For each model and task, first collect input tokens, output tokens, arrival times, whether streaming is enabled, priority, time-to-first-token SLO, completion-latency SLO, cancellation rate, and cache hit rate. Retain at least P50, P95, P99, and minute-level peaks because daily averages conceal product launches and overlapping time-zone demand. Sensitive prompts do not need to enter the capacity dataset; length, task category, tenant, and sanitized routing labels are usually sufficient.

Model Prefill and Decode Separately

LLM inference consists of prefill and decode. Long inputs mainly consume prefill compute and KV Cache, while long outputs occupy decode slots for longer. The capacity table should separately record prefill tokens/s, decode tokens/s, time-to-first-token latency curves, maximum concurrent sequences, and memory usage for every GPU, precision, tensor-parallelism setting, model version, and inference runtime. Do not treat peak tokens/s from a single short-input laboratory test as production capacity.

The memory budget can be divided into model weights, runtime workspace, KV Cache, and safety margin. Weight memory is relatively fixed, while KV Cache grows with the number of layers, sequence length, concurrency, and data type. The runtime may also consume additional memory because of CUDA Graph, quantized kernels, or fragmentation. Use load testing to find the maximum concurrency that operates reliably, and confirm that it still avoids OOM after mixing long and short requests, cancellations, bursts, and sustained operation. Theoretical formulas alone can easily miss implementation overhead.

Batching improves throughput but may increase queueing time. Set a maximum batch wait and a separate queue for interactive requests; use throughput-oriented queues and time windows for LLM Batch Inference. If both traffic classes share a pool, offline jobs must be preemptible or strictly capped. They must not occupy all decode slots during the next morning's peak simply because the overnight batch has not finished.

Estimate with a Queueing Model, Then Calibrate with Replay

A practical starting point is to calculate the required token service rate over the peak window and multiply it by the inverse of the target utilization. Suppose a task receives 20 requests per second at peak, with a P95 input of 1,500 tokens and output of 300 tokens. You cannot simply calculate 20 * 1800 and divide by the peak throughput of one GPU because prefill and decode use different resources, requests overlap, and the tail is longer. Replay the real arrival-time and length sequences against candidate configurations, then observe queueing, time to first token, and completion latency.

Higher target utilization is not always better. Keeping a GPU 95% busy may appear economical, but it leaves no headroom for bursts, failure rescheduling, or long requests. Interactive pools often need lower steady-state utilization than batch pools. The platform should report GPU SM utilization, memory usage, queue depth, active sequences, waiting time, prefill/decode throughput, rejected requests, and cost per million tokens. Basic telemetry can integrate with Prometheus and Grafana Observability.

Capacity conclusions should be recorded as a reproducible configuration matrix, for example:

modelDigest: sha256:...
gpu: H100-80GB
runtime: serving-v2.4
precision: fp8
tensorParallel: 2
workload: interactive-code-2026q3
maxStableSequences: 72
testedPeakRps: 18
firstTokenP95: 780ms
decodeP95: 31ms/token
headroom: 30%

Any change to model weights, runtime, drivers, precision, or scheduling parameters should invalidate the previous conclusions. Otherwise, continuing to allocate capacity based on months-old per-GPU figures after a platform upgrade can leave a capacity shortfall that becomes visible only at a real traffic peak.

Reserve Capacity for Scaling Delays and Failures

Scaling GPU nodes usually takes longer than scaling CPUs. Cloud capacity may be unavailable, and after a node starts, it still needs drivers installed, large weights downloaded and verified, and the model warmed up. Autoscaling cannot rely solely on GPU utilization because queues have already formed by the time utilization rises. Better leading indicators include queue wait by priority, estimated time to drain, active token count, and available KV Cache. After scaling out, count a model toward available capacity only once it is genuinely ready.

Minimum redundancy is determined by failure domains, not a fixed percentage. If the design objective requires preserving the SLO after losing one node, validate N+1 capacity. If critical traffic must remain available after an availability-zone failure, ensure that the remaining zones have enough warm capacity, weight replicas, and network ingress to take over. Do not count instances that have not yet started as failure capacity. Platforms that share GPUs among multiple models must also simulate whether failover for the most popular model crowds out other tenants.

Define the degradation order in advance for capacity shortages: first pause low-priority batch jobs, then limit very long outputs and new sessions, and finally queue or reject requests according to tenant quotas. Only smaller models that have passed quality and compliance evaluations may serve as substitutes. Silently truncating context or switching models at random turns an infrastructure failure into incorrect answers. Cloud Cost Optimization provides a procurement and utilization perspective, while AI Cost Attribution helps determine which products and tenants should bear the cost of reserved capacity.

Recalculate Weekly Instead of Guessing Annually

A capacity plan should include at least the baseline, business growth, known events, model migrations, failure headroom, and an uncertainty range. Replay actual distributions every week and compare forecasts with observed results. If token lengths rise, cache hit rates fall, or a new feature adds multi-turn conversations, update the demand forecast for the next four to twelve weeks. Label the availability of reserved, on-demand, and interruptible resources separately; do not use cheap but reclaimable capacity to satisfy critical online SLOs.

Production acceptance should include three drills: gradually increase traffic to the forecast peak and sustain it long enough; inject a sudden burst and observe queueing and scaling; and remove a node or availability zone to validate redundancy. Load-test figures are meaningful only when request-length distributions, model configurations, and SLOs match production. The final deliverable is not "purchase ten GPUs," but a capacity contract that documents the load, assumptions, performance evidence, failure headroom, scaling latency, and cost ownership.

Further Reading

• LLM Batch Inference
• Prometheus and Grafana Observability
• Cloud Cost Optimization
• AI Cost Attribution