Your prompt cache is a side channel
Automatic prefix caching is one of the best efficiency wins in LLM serving: if a new request starts with tokens the server has already prefilled, it reuses their KV blocks. Across tenants, that speed-up is also a signal.
…support assistant for Contoso. Escalate to……support assistant for {candidate}. Escalate to… and times TTFTWhy block size matters, and why it isn’t a defence
Prefix caches work in fixed-size blocks: a block is reused only if every token in it, and every token before it, matches. With a block of 16 tokens, an attacker can’t learn a prompt one token at a time; each test is all-or-nothing for 16 tokens. That makes reading free text impractical. It does nothing against confirmation: when the attacker knows the template and only one field is secret, each candidate fills the block, and the hit-or-miss answer is exactly what they need. Larger blocks even make each hit save more time, as the calculator shows.
The fix is an isolation boundary, not a smaller signal
Noise and rate limits only raise the number of probes, and averaging beats noise at √n. The structural fix is to make cross-tenant hits impossible: include a per-tenant secret in the cache key. vLLM’s design supports exactly this: a cache_salt on the request is injected into the hash of the first block, so only requests with the same salt can reuse each other’s KV blocks. Turn it on above: the two distributions become one.
What it costs is the efficiency of cross-tenant sharing, which in a multi-tenant service you mostly shouldn’t have been relying on. Tenants still share the system prompts they themselves repeat. It’s the same trade as everywhere else on this site: a performance optimisation that crosses a trust boundary is a channel across that boundary.
- Salt per tenant (or per trust domain), derived server-side; never accept the salt from the client as the only isolation.
- Treat shared-cache features (prefix caching, KV offload to shared tiers, disaggregated KV transfer) as multi-tenant data stores in the threat model.
- Don’t rely on TTFT jitter as protection. Measure the separation on your own stack with probes like these before deciding it’s too small to matter.