The kernel can’t see your tensors: RDMA’s trust model
RDMA is fast because it takes the remote CPU and kernel out of the data path. That is also why every host-based security control you own goes blind the moment the gradients start moving.
What actually authorises a remote write
The receiver registers a memory region (for GPUDirect, a slice of GPU memory) under a protection domain, and gets back an rkey. It hands the peer an (address, rkey) pair. From then on, any RDMA WRITE that arrives on a queue pair in that protection domain, with a matching rkey and an address inside the region, is placed into memory by the NIC. No interrupt, no syscall, no policy check in software.
Three things follow. The rkey is a bearer capability: whoever holds it can write. It travels in plaintext in each WRITE packet’s RDMA extended transport header, so anyone who can observe the fabric learns it. And it is 32 bits, long-lived for as long as the region stays registered. For a KV-cache pool registered once at startup, as most disaggregated-inference transports do for speed, that is the life of the server.
Reading the calculator
At full 32-bit entropy, a modest guess rate and a handful of long-lived regions, an off-path attacker gets there in days, and “registered at startup” means the key lives longer than that. Drop the effective entropy (key generation that is less random than it looks was one of ReDMArk’s findings) and it’s minutes. The lever that changes the shape of the curve is lifetime. A type-2 memory window, bound per transfer and invalidated when the transfer ends, turns a server-lifetime capability into a milliseconds-lifetime one.
That isn’t only a security property. It is the same mechanism that stops a delayed write from a timed-out transfer from landing in a KV block that has since been reused: the NIC rejects it with a remote access error. In kvwire this is ADR-0005: per-transfer memory windows are the first option for stale-write fencing, to be decided on measured bind cost. Correctness and security want the same thing here, which is the best kind of security control: it pays for itself.
The controls, at the layer that can enforce them
| Control | What it stops | Cost on the hot path |
|---|---|---|
| Protection domain + QP per tenant | a tenant’s QP using another tenant’s rkey | none (setup only) |
| Partitions (IB partition keys) / VRF or VLAN per tenant (RoCE) | packets from outside the tenant’s partition reaching its QPs | none (switch / NIC) |
| Type-2 memory windows, per transfer, invalidated on completion or abort | long-lived keys; late writes into reused memory | one bind per transfer (measure it) |
| Inline encryption (IPsec / MACsec offload on NICs that support it) | reading rkeys and tensors off the wire | NIC offload; verify line rate |
| Telemetry at the NIC / DPU (counters, flow records, access errors) | blind spots: the host can’t see this traffic | none on the data path |
| No pickle or untyped RPC on the control path | RCE through the KV-transfer side channel (CVE-2025-32444) | none |
Notice the last column. None of these controls sits in software on the per-message path, which is what makes them deployable on a fabric whose whole reason to exist is microseconds.