The KV-cache bill: what disaggregated inference has to move
Splitting prefill and decode onto separate GPUs removes their interference, and creates a transfer. How many bytes is that, how long does it take, and when can you hide it?
Why disaggregate at all?
LLM inference has two phases with opposite profiles. Prefill processes the whole prompt in one pass, is compute-bound, and arrives in bursts. Decode generates one token per step, is bound by memory bandwidth, and is sensitive to latency. On the same GPUs, a large prefill stalls every in-flight decode (TPOT spikes), and a full decode batch delays new prefills (TTFT spikes). Chunked prefill reduces the interference but doesn’t remove it.
Disaggregated serving (DistServe, Splitwise and Mooncake are well-known designs) runs the phases on separate GPU pools, so each pool can be batched and scaled for its own bottleneck. The cost is that the KV cache prefill produces must reach a decode GPU before the first token can be generated.
How big is the KV cache?
Each transformer layer stores one key and one value vector per KV head per token:
Here are the numbers for public model configs at 16-bit precision. Notice what grouped-query attention (GQA) did: Llama 2 7B uses one KV head per query head, while Llama 3.1 8B shares 8 KV heads across 32 query heads.
| Model | Layers × KV heads × dim | KV / token | 8K-token prompt |
|---|
Three things the calculator shows
1. The ratio barely moves with prompt length
Drag the prompt length from 128 to 128K tokens. Both the bytes and the dense FLOPs (≈ 2 · params · tokens) scale linearly, so the transfer-to-compute ratio stays roughly flat until attention’s quadratic term takes over at long contexts, and then compute grows faster than bytes. The ratio is set mostly by KV bytes per parameter, the GPU’s achieved FLOPs and the link.
2. Tensor parallelism helps the network as well as compute
With TP=8, the KV cache is already sharded across 8 GPUs, and each GPU can send its shard over its own NIC. Aggregate bandwidth scales with TP, which is why rail-optimized fabrics (GPU i of every node on the same leaf) matter for KV transfer and not only for training collectives.
3. Pipelining hides the transfer, until the link falls behind
Layer i’s KV is final as soon as layer i finishes, so a transport can send it while layer i+1 computes. When per-layer transfer time is below per-layer compute time, only the last layer’s transfer is exposed. When it isn’t, the backlog grows layer by layer and adds straight to TTFT. Try Llama 2 7B (MHA) on a 100 Gb/s NIC: it becomes link-bound. Then switch to Llama 3.1 8B (same size, GQA) or to FP8 KV, and the exposure shrinks to almost nothing. Large GQA models like the 70B keep up comfortably, because they do far more compute per KV byte.
What the calculator leaves out
The arithmetic is the easy part. These are the parts I’m building kvwire to measure:
- Layout mismatch. Paged KV caches scatter a sequence across many small blocks. Moving thousands of small blocks means many small RDMA writes that never reach link efficiency, unless the transport gathers them or a kernel re-lays them out first.
- Different parallelism on each side. Prefill at TP=8 handing off to decode at TP=2 means re-sharding in flight.
- Failure mid-transfer. A link flap halfway through a 2 GiB handoff must not leave decode generating from a half-written cache. The transport needs completion semantics and the coordinator needs a retry or recompute policy.
- Congestion. KV bursts share a fabric with everything else, and on lossless RoCE that brings PFC behaviour into the picture.
Every number kvwire publishes will sit next to the measured hardware ceiling and link to its run. Until then, this page is the model it will be checked against.