Field guide: how AI infrastructure actually gets breached

Six patterns

1. Control planes are remote code execution by design

Ray’s job API, a Slurm controller and a Kubernetes API server all exist to run code on GPUs. ShadowRay is what happens when one is reachable without authentication: the vendor considers it intended behaviour, scanners skip “disputed” CVEs, and attackers run miners on your accelerators. Treat every orchestration endpoint as an RCE endpoint, because it is one.

2. Artifacts are code

A PyTorch checkpoint is a pickle, and unpickling is a stack machine that can import and call any function. Wiz ran code inside Hugging Face’s inference infrastructure by uploading a model. PyTorch’s weights_only=True safe mode itself had an RCE bypass. And the same mistake reappears on the network: vLLM’s Mooncake KV-transfer path unpickled data from ZeroMQ sockets bound to every interface. Disassemble one yourself →

3. The GPU stack is kernel attack surface

The NVIDIA Container Toolkit runs privileged hooks on the host for every GPU container. Two container escapes in a year (CVE-2024-0132, NVIDIAScape) mean the container is not the tenant boundary on a shared GPU host. And GPU memory is shared state: LeftoverLocals reconstructed another process’s LLM output from uncleared local memory.

4. A token’s scope is its blast radius

A link meant for downloading open models was an Azure SAS token with full control of 38 TB, so it could also have overwritten the models. The GitHub MCP toxic flow needed no exploit at all, just a token that could read every private repository. Scope is a security control you can set today, for free.

5. Agents take instructions from data

EchoLeak steered Copilot with an email. A malicious commit put a wiper prompt into a shipped coding agent. The Nx malware didn’t bring its own scanner; it drove the AI CLI already on the machine. And GTG-1002 shows attackers now operate at agent speed. You can’t filter every injection, so limit what a fooled agent can do. Measure the blast radius →

6. Performance shortcuts become side channels

Prefix caching shares work across requests. If requests come from different tenants, a cache hit is observable as a faster time-to-first-token (Early Bird, vLLM advisory). The fix, a per-tenant salt in the cache key, already exists in vLLM, but it is opt-in. Run the attack → Underneath all of it, the fabric’s own access control is a guessable capability, not cryptography (ReDMArk). RDMA’s trust model →

What I would put in a new AI cluster first

Ordered by blast radius removed per unit of engineering effort, cheapest first. None of these touches the training step’s hot path.

1. No public control planes. Ray, Slurm, Kubernetes, Triton, vLLM and dashboards on private networks only, with authentication even there. Closes ShadowRay-class exposure.
2. Short-lived, least-scope credentials everywhere. One task, one scope. No account-wide storage tokens, no cluster-admin kubeconfigs on laptops or in agents.
3. safetensors only for anything you didn’t produce, and hashes plus signatures on everything you did. No pickle on any network path, including internal RPC and KV transfer.
4. Agents get read-only tools by default. Mutations go through an allowlisted, audited executor; destructive actions need a human. Egress allowlists on agent sandboxes.
5. Patch the GPU container stack like the kernel, and keep untrusted images off shared GPU hosts. Use VMs, MIG or confidential computing where tenants don’t trust each other.
6. Tenant isolation in the fabric: partition keys or VRFs per tenant, scoped and revocable memory windows, inline encryption where the NIC offloads it, telemetry at the NIC or DPU.
7. Per-tenant cache isolation in serving (salted prefix-cache keys), egress allowlists on inference pods, and memory scrubbing on GPU reassignment.
8. Detection at agent speed: alert on new job types, new egress destinations and new token scopes in minutes, not on next week’s audit.