Prefill-Decode Disaggregation: When and Why to Split Your Inference Stack

Photo by Kvistholt Photography on Unsplash Every major inference engine now supports it. vLLM shipped disaggregated prefill as a stable feature. SGLang has it. LLM-d built its entire architecture around it. NVIDIA’s Dynamo and TensorRT-LLM support it natively. The infrastructure story has converged. What hasn’t converged is practitioner understanding of when it’s worth doing. Most teams either skip disaggregation entirely because it sounds like a research technique or they adopt it reflexively because the throughput numbers in vendor blog posts look compelling without checking whether their workload actually has the shape that makes those numbers real. Both are mistakes. This post is the decision framework that’s been missing. TL;DR Prefill and decode have fundamentally different resource profiles — compute-bound versus memory-bound — and that mismatch is the entire reason disaggregation exists. Disaggregation pays off when you have variable prompt lengths, high concurrency, and interactivity requirements (low inter-token latency). It does not pay off for small-scale or latency-tolerant batch workloads. Chunked prefill is the cheaper first lever — try it before you build separate node pools. KV cache transfer between prefill and decode nodes is the operational cost nobody puts on the slide—network bandwidth and transfer latency can eat your throughput gains if the connector isn’t tuned. Real production numbers: disaggregated setups have shown throughput gains in the 75–250% range over collocated serving, but the hardware cost and complexity scale with it. Why Prefill and Decode Don’t Belong on the Same GPU Every LLM inference request has two phases with opposite performance characteristics. Prefill processes your entire input prompt in a single forward pass — every token attends to every other token in one dense matrix multiply. This is compute-bound: the GPU’s raw FLOPS throughput is the limiting factor, and a long prompt keeps the GPU’s compute units saturated. Decode generates output tokens one at a time. Each new token requires reading the key-value cache for every previously generated token from GPU memory. This is memory-bound: the GPU spends most of its cycles moving KV cache tensors, not computing, and its compute units sit mostly idle. When both phases run on the same GPU under continuous batching — the default in most serving stacks — a new prefill request has to be interleaved into an active batch of decode requests. That interleaving preempts ongoing decode work, and the batch has to pause token generation while it processes the new prompt. This is the direct cause of the unpredictable latency spikes you see under bursty traffic: a well-behaved decode stream gets stalled every time a large new prompt shows up in the batch. Prefill draws close to peak GPU power (70–100%), while decode typically uses only 20–40%. Running both on identical hardware means you’re either over-provisioning for decode’s compute needs or under-provisioning for prefill's—you can’t tune the same GPU for both profiles simultaneously. Disaggregation resolves this by physically separating the two phases onto different node pools, each independently scaled and independently hardware-matched: compute-heavy GPUs (H100 class) for prefill and memory-bandwidth-heavy or cheaper GPUs for decode. The Decision Framework Disaggregation is not free. It requires standing up two node pools, a KV cache transfer layer between them, and meaningfully more operational surface area than a single collocated deployment. Here’s when the tradeoff is worth it. Reach for disaggregation when: Your prompts are long and variable. If you’re consistently hitting prompt lengths in the thousands of tokens with high concurrency, Prefill's compute demand is great enough and unpredictable enough to genuinely disrupt decode when they share hardware. You need low, stable inter-token latency. Interactive applications—chat interfaces, coding assistants, anything with a human staring at the token stream—are exactly where prefill interference is most visible. Removing it is the primary latency win disaggregation delivers, independent of raw throughput. You’re operating at real production scale. The RTP-LLM production deployment benchmarks at 480B-parameter MoE scale show disaggregation delivering meaningfully lower time-to-first-token and higher cache hit rates than collocated serving at that concurrency level. At a small scale, the fixed overhead of the separate deployment and transfer layer swamps the benefit. You have workload heterogeneity that benefits from independent scaling. If your prefill load and decode load don’t scale together—bursty prompt-heavy traffic at one time of day, sustained generation-heavy traffic at another—separate pools let you scale each independently instead of over-provisioning one to satisfy the other. Skip disaggregation when: Your workload is small or steady-state. Published testing has found that under-scaled or poorly tuned disaggregated setups can actually underperform collocated serving by 20–30%, because the fixed cost of cross-node KV transfer isn’t amortized across enough traffic. Your prompts are short, or your prefix cache hit rate is high. If most of your prompt is already cached—a long shared system prompt, for instance—the actual prefill compute per request is small, and running it locally on the decode worker is often faster and simpler than paying network transfer costs to a separate prefill node. You haven’t tried chunked prefill yet. This is the cheaper, simpler lever most teams should reach for first. Chunked Prefill: The Step Before Disaggregation Before standing up separate node pools, chunked prefill addresses the same interference problem within a single collocated deployment. Instead of processing an entire long prompt in one uninterrupted forward pass, the engine splits it into smaller chunks and interleaves them with ongoing decode steps — piggybacking decode work onto the prefill computation instead of letting one fully block the other. # vLLM: enabling chunked prefill from vllm import LLM, SamplingParams llm = LLM( model="meta-llama/Llama-3.1-70B-Instruct", enable_chunked_prefill=True, max_num_batched_tokens=2048, # chunk size ceiling per batch step max_num_seqs=256, # concurrent sequence cap ) This is a single-node, single-engine optimization — no new infrastructure, no KV transfer layer, no second node pool. For teams under the concurrency threshold where disaggregation pays off, chunked prefill alone often closes most of the latency gap. Only move to full disaggregation once you’ve confirmed chunked prefill isn’t enough for your traffic pattern. The Operational Cost Nobody Puts on the Slide: KV Cache Transfer The throughput numbers in vendor benchmarks assume a well-tuned KV transfer layer between your prefill and decode nodes. This is the part that’s genuinely hard to get right, and it’s where most first attempts at disaggregation lose the gains they were promised. When a prefill node finishes processing a prompt, it has to hand the resulting KV cache—which can be gigabytes for long contexts—to a decode node over the network, fast enough that the decode node isn’t sitting idle waiting for it. The transfer mechanism matters enormously: # vLLM: NIXL-based KV transfer configuration (prefill node) from vllm import LLM prefill_llm = LLM( model="meta-llama/Llama-3.1-70B-Instruct", kv_transfer_config={ "kv_connector": "NixlConnector", "kv_role": "kv_producer", }, ) # Decode node decode_llm = LLM( model="meta-llama/Llama-3.1-70B-Instruct", kv_transfer_config={ "kv_connector": "NixlConnector", "kv_role": "kv_consumer", }, ) NIXL (NVIDIA Inference Xfer Library) and similar connectors handle the low-level transfer over high-bandwidth interconnects like UCX or RDMA-capable networking. On AMD hardware, the MORI-IO connector serves the equivalent role. Recent testing on 8-GPU MI300X nodes showed disaggregated serving achieving roughly 2.5x higher goodput than collocated serving on the same hardware, but that number depends entirely on the interconnect being fast enough that KV transfer doesn’t become the new bottleneck. If your nodes are connected over standard networking rather than a high-bandwidth RDMA fabric, budget real engineering time for this layer. A disaggregated setup with a slow transfer connector can genuinely underperform a well-tuned collocated deployment—the throughput math only works if the KV cache moves fast enough to keep the decode node fed. Real Numbers, With Context Published benchmark comparisons on Llama 3.1 70B have shown disaggregated H100+H200 setups costing roughly 45% more per hour in hardware than a collocated equivalent, while delivering roughly 75% more throughput — a favorable tradeoff if your bottleneck is genuinely throughput and not cost per token at low volume. At a larger scale and with more aggressive architectures—intra-GPU disaggregation approaches like Nexus, which split a single GPU’s resources between prefill and decode rather than using separate physical nodes—research benchmarks report up to 2.2x higher throughput and over an order of magnitude lower time-to-first-token compared to standard vLLM collocated serving, using half the GPU count of traditional node-level disaggregation. The spread between these numbers is the point: “disaggregation gives you 2x throughput” is not a fixed constant. It’s a function of your prompt length distribution, concurrency, hardware, interconnect quality, and which disaggregation architecture you implement. Treat every vendor benchmark as a ceiling, not a guarantee, until you’ve reproduced something close to it on your own traffic shape. Gotchas: What Nobody Tells You 1. Disaggregation changes your failure modes, not just your performance profile. A collocated deployment fails as a unit. A disaggregated deployment can fail asymmetrically — your prefill pool can be healthy while your decode pool is degraded, or the KV transfer layer itself can become a silent bottleneck that doesn’t show up as a node-level failure. Instrument the transfer layer explicitly; don’t rely on per-node health checks alone. 2. Prefill can become memory-bound too, at long enough context lengths. The clean compute-bound/memory-bound split that motivates disaggregation breaks down as context length grows very large—at that point, prefill starts converging toward the same memory-bandwidth constraints as decode, which undermines the hardware specialization argument. If your workload involves very long contexts (100K+ tokens), validate that the compute/memory split still holds for your actual prompt distribution before assuming the standard disaggregation playbook applies. 3. “It works in the vendor’s benchmark” doesn’t mean it works at your concurrency level. Most public benchmarks are run at concurrency and batch sizes tuned to showcase the architecture favorably. Reproduce the comparison at your actual expected traffic volume before committing engineering time to the migration—the crossover point where disaggregation beats collocated serving is workload-specific, not universal. Conclusion Prefill-decode disaggregation solves a real, structural problem: two phases of LLM inference with opposite resource profiles being forced to share hardware. The infrastructure to do it well now exists across every major serving engine. But the decision to adopt it should be driven by your workload’s actual shape—prompt length distribution, concurrency, and latency requirements—not by the throughput number in the most recent vendor blog post. Start with chunked prefill. If that’s not enough, and your traffic genuinely has the long-prompt, high-concurrency, interactivity-sensitive profile that disaggregation is built for, move to full disaggregation—and budget real engineering time for the KV transfer layer, because that’s where the promised gains actually get won or lost. Prefill-Decode Disaggregation: When and Why to Split Your Inference Stack was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.