LLM Inference Optimization: Speculative Decoding and KV Cache

LLM inference becomes a core bottleneck at scale. But the slowness is not uniform. Understand two stages and two optimizations, and you can improve inference speed by an order of magnitude. This article covers the most valuable optimization techniques for self-hosting LLMs.

I. First Understand the Two Stages of LLM Inference

An LLM generates a response in two stages:

1. Prefill: Process the input prompt and compute the KV states for every position. This step can be parallelized by processing the entire prompt in one pass, so it is fast. Most time-to-first-token latency is spent here.

2. Decode: Generate output tokens one at a time. Each token depends on the previous one, making the process strictly serial. An output of 1000 tokens requires 1000 serial forward passes. Most total latency is spent here.

The optimization priority is therefore clear: prefill relies on parallelism and is already fast enough, while decode requires fewer serial steps or faster execution at each step. Both optimizations below target decode.

II. KV Cache: Do Not Recompute History

During decode, generating each new token would theoretically require recomputing attention over all preceding tokens. That is clearly wasteful: the earlier tokens have not changed, so repeating the computation serves no purpose.

KV Cache caches the Keys and Values for all previous tokens at every layer. When generating a new token, the model reuses them and computes only the new token's portion.

The effect is substantial: computation per decode step falls from O(sequence length) to O(1), producing a major speedup. This is a foundational LLM inference optimization, and nearly every inference framework enables it by default.

The cost is VRAM. The KV cache consumes VRAM, growing with sequence length × layer count × dimensionality. With long contexts or high concurrency, the KV cache can consume more VRAM than the model weights. This has led to several related optimizations:

• PagedAttention (vLLM): Manages the KV cache in pages to reduce fragmentation and improve VRAM utilization. This is central to vLLM's high throughput.
• Quantized KV cache: fp16 → fp8/int8, cutting VRAM use in half.
• KV cache offloading: Offloads part of the cache to CPU/SSD for long contexts.

KV cache trades VRAM for speed. Understanding it explains why long contexts and high concurrency are especially VRAM-intensive.

III. Speculative Decoding: Use a Small Model to Accelerate a Large Model

KV cache makes each step faster, but decode remains serial, advancing one token per step. Speculative Decoding breaks this serial pattern by having a small model first "guess" the next few tokens, then having the large model validate those guesses in parallel in a single pass.

Small model (draft) quickly generates k candidate tokens: [t1, t2, t3]
Large model (target) computes all k positions in parallel in one pass → validates the guesses
  ├ Correct guesses: Accept directly (no separate large-model pass needed)
  └ Incorrect guesses: Regenerate from the first incorrect position

The key insight is: the cost for the large model to validate k tokens in parallel is roughly equal to the cost of generating one token itself because the validation runs in parallel. If the small model guesses accurately, such as getting 3 out of k=4 tokens right, one large-model pass makes three tokens of effective progress, producing a speedup close to 3x.

The result: when guesses are accurate, throughput improves by 2–3x, while the output remains exactly equivalent because the large model validates it, so there is no quality loss.

Requirements:

• A small model is available (the draft model, usually a smaller model from the same family).
• The small model predicts accurately (the task matches its capabilities; predictions are less accurate for purely creative writing, limiting the speedup).
• The inference framework supports it (vLLM, TensorRT-LLM, and TGI all do).

The cost is wasted compute when predictions are inaccurate because tokens are validated but not accepted. It remains a net benefit overall, but it has limits.

IV. Continuous Batching

Traditional batching waits for a batch of requests to arrive before processing them together, or waits for the entire batch to finish before starting the next one. The problem is that request lengths vary: short requests wait for long ones, leaving the GPU idle.

Continuous batching, also called in-flight batching or dynamic batching, lets requests enter as they arrive and leave as soon as they finish, dynamically forming batches at every step. Completed short requests are removed and new requests are added, keeping the GPU fully utilized.

The result is a major throughput improvement, and it is one of the core capabilities of vLLM and TGI. Combined with PagedAttention, it is standard for high-concurrency inference.

V. Quantization: Make the Model Itself Lighter

Quantizing model weights from fp16 to int8/int4 directly reduces VRAM use and increases speed:

• VRAM use falls to one-half or one-quarter (the same GPU can run a larger model).
• Inference is faster (int8/int4 operations are faster).
• The cost is a slight loss of quality (int4 can be noticeable in some tasks, while int8 is effectively lossless).

Common options include AWQ, GPTQ, and fp8 on newer hardware. Frameworks such as vLLM natively support loading quantized models.

VI. Putting It All Together: An Inference Optimization Checklist

To accelerate a self-hosted LLM, prioritize by ROI:

1. KV Cache (enabled by default; foundational).
2. Continuous batching + PagedAttention (use a framework such as vLLM; essential at high concurrency).
3. Quantization (AWQ/GPTQ; saves VRAM + increases speed with controllable quality loss).
4. Speculative Decoding (when a suitable small model is available, throughput can double again).
5. prefix/prompt caching (reuse KV for identical prefixes; produces major savings when the system prompt is fixed).

The first three are "standard," while the final two are "advanced." Stack them together, and throughput on a single GPU can be an order of magnitude higher than with naive inference.

VII. When to Optimize It Yourself vs. Use an API

These optimizations mainly apply to self-hosting, where you run your own inference service.

• Using a commercial API: optimization is the provider's responsibility. You can control only the prompt by shortening it or using prompt caching, plus concurrency.
• Self-hosting: all the optimizations above apply and are essential for using fewer GPUs and increasing throughput.

Therefore, "should self-hosted inference be optimized?" is not an optional question; optimization is mandatory. Without it, self-hosting is uncompetitive in both cost and latency.

Summary

For LLM inference optimization, understand the two stages and master two core techniques:

• Two stages: prefill (parallel and fast) + decode (serial and slow, making it the optimization priority).
• KV Cache: Avoid recomputing history by "trading VRAM for speed." PagedAttention further improves VRAM efficiency.
• Speculative Decoding: A small model predicts and a large model validates, breaking the serial pattern, doubling throughput, and preserving quality.
• Add continuous batching + quantization + prompt caching, and self-hosted inference can become an order of magnitude faster.

Remember: the bottleneck in LLM inference is not "insufficient compute," but "serial decoding + wasted VRAM." The tools that address these problems, including KV cache, speculative decoding, and PagedAttention, are already mature. Self-hosting an LLM without these optimizations amounts to wasting the GPU you bought.