Naresh Kumar · Secure AI/ML infrastructure: GPU networking · distributed systems · zero trust

Frontier models deserve infrastructure that can’t be stalled, slowed, or stolen.

A career built below the framework layer: kernel networking, RDMA/RoCE, DPDK, SR-IOV, KVM and NCCL. Now applied to the three things a training or inference cluster must get right at scale: reliability, efficiency and, now that AI agents hold the keys, security.

GPU cluster networking Distributed training NCCL · RDMA / RoCE v2 AI-infra security Zero trust for agents Disaggregated inference DPDK · SR-IOV · KVM Linux kernel · eBPF / XDP
See the work → Explore the attack surface →
Fig. 0 · Ring all-reduce under failure & attack8 GPUs
Ring all-reduce losing a link, recovering, then refusing an intruder Eight GPUs pass gradient chunks around a ring. The link between GPU 2 and GPU 3 fails, the stall spreads, every rank times out, and the job restores onto a spare. Then an unattested host tries to join the communicator and is denied while training continues. A goodput timeline shows productive time in blue, stalled time hatched and restart time in red.

Illustrative timeline. Every second the bar isn’t blue is lost goodput. Break the ring yourself →

Repo · reliability↗
goodput: fault attribution & recovery for multi-node training
Design, interfaces, test harness public. M1 of 6.
Repo · runnable↗
nicprof: which NIC counter explains which millisecond of step time
nicprof demo · 69 tests
Repo · efficiency↗
kvwire: RDMA KV-cache transport for disaggregated inference
vLLM KV-connector plug-in. M1 in progress.
Field guide · security→
18 AI-infra incidents, each mapped to the layer it hit and the control that contains it
Every entry links its advisory or CVE
Model · Hugging Face↗
qwen3-nccl-rca-lora: grounded root-cause analysis for NCCL incidents
Model card public · weights after goodput’s labeled data

Work

Three problems I work on

AI lab infrastructure splits along three lines: keep the job running, make each GPU-hour count, and make sure nobody else gets to use either. Kernel and fabric depth is the foundation for all three.

Reliability at scale

goodput fault-tolerant training orchestrator

Problem
One bad GPU, NIC, cable or switch port stalls a collective, and every rank times out the same way.
Why it’s hard
Symptoms are global and causes are local. Stock tooling detects in minutes, attributes nothing, and restarts the whole job, sometimes onto the same broken hardware.
Approach
Inject faults with a ground-truth ledger, capture NCCL flight-recorder signatures, and correlate them with DCGM and NIC telemetry to name the culprit rank and component. Correct attribution makes small recoveries safe: re-init communicators in-process, replace one node, or restore from a peer’s in-memory checkpoint.
Status
M1 of 6 · in progress  No results measured yet; every future number will link to a run.
Models
The injector’s ledger labels every incident by construction. That data trains two Hugging Face models that plug back in as goodput’s classifier: a calibrated, evidence-citing triage tier (ModernBERT, milliseconds on CPU) and a grounded root-cause tier (Qwen3 + LoRA, RL on rewards a script can check). To take the slot, they must beat goodput’s own rules. design
Repo ↗ nicprof ↗ RCA model card ↗ Post: ring under failure Post: NCCL triage Post: checkpoint interval

Efficiency at scale

kvwire disaggregated inference engine

Problem
Prefill is compute-bound, decode is memory-bandwidth-bound. Colocated, they interfere: prefills spike TPOT, decodes spike TTFT.
Why it’s hard
Disaggregation trades interference for data movement. The KV cache has to cross the network before decode can start, and has to stay correct when the network fails mid-transfer.
Approach
A GPUDirect RDMA transport, a Triton KV re-layout kernel (kept only if the data justifies it), and a failure-aware coordinator. It plugs into vLLM through the KV connector interface and replaces the parts that move bytes and decide what to do when moving bytes goes wrong.
Status
M1 · in progress  Baselines and hardware ceilings first; results cells stay “—” until measured.
Repo ↗ CUDA/Triton kernels ↗ Post: the KV-cache bill Post: PFC deadlock

Trust at scale

Secure by construction zero trust for GPU fabrics, model artifacts and AI agents

“Assume the model will be fooled. Design so that being fooled is cheap.”

Problem
The fastest paths in an AI cluster are the least watched. RDMA bypasses the host kernel, so firewalls and eBPF never see it. Checkpoints are programs: pickle runs code on load. And AI agents now hold the credentials, taking instructions from whatever they read.
Why it’s hard
Every control costs latency or bandwidth on hardware that is billed by the second, so security that slows the step gets turned off. The work is finding the controls that sit off the hot path: at the NIC, in the artifact format, at the tool boundary.
Approach
Threat-model each layer against real incidents, then build the control into the system rather than bolting it on. Already in my repos: goodput’s fault injector is default-deny. It acts only on allowlisted hosts, every fault carries an auto-revert watchdog, destructive mechanisms need an explicit opt-in, and GPU resets, firmware and switch changes are never automated (SECURITY.md). kvwire’s ADR-0005 evaluates per-transfer memory windows, invalidated before a KV block is reused, so the NIC itself rejects a late or stray write.
Status
safety invariants in code fencing ADR proposed goodput RCA tier: recommends, never acts · planned  Plus five interactive security posts below.
Field guide: AI-infra breaches Post: agent blast radius Post: pickle is a program Post: RDMA’s trust model Post: prefix-cache side channel

Security

The AI-infra attack surface, layer by layer

Pick a layer. Each one shows what goes wrong, the controls I design for, and the public incidents that already happened there, each linked to its advisory or CVE.

Each layer is a trust boundary. An attacker who crosses one gets everything that layer was trusted with. Measure an agent’s blast radius →

Writing

Simulators, capacity models & threat models

Each post runs in the browser: break a ring, size a KV transfer, disassemble a malicious checkpoint, measure an agent’s blast radius. Every model states its assumptions.

Upstream & evidence

Things you can run or read

Each entry says what problem it addresses and links to the source.

NIC counter → step-time attribution nicprof

The gap: a 512-GPU job slows down; the network team has millions of counter series, the ML team has a step-time graph, and nobody can say which counter explains which millisecond.

What it does: aligns RDMA/ethtool/SR-IOV counters to each rank’s step windows, fits a non-negative model of excess step time, and traces PFC pause back to its origin so a victim port isn’t blamed as the cause. NIC-side telemetry is also where kernel-bypass traffic becomes visible at all.

Runnable demoSource ↗

Kernel performance toolkit kptk

The gap: “CPU 80%” and “THP on” don’t explain why a workload is slow.

What it does: explains slowness from kernel evidence (run-queue delay, allocation vs access NUMA locality, THP fallback and compaction stalls) and calls perf_event_open(2) directly with multiplexing correction. Standard library only, 43 tests.

Runnable demoSource ↗

Pickle opcode disassembler & scanner in-browser

The gap: “don’t load untrusted checkpoints” is advice; seeing GLOBAL os system sitting inside a model file is understanding.

What it does: a dependency-free pickle-protocol disassembler that steps through a checkpoint’s opcodes on a stack machine and flags every import outside a weights-only allowlist, next to a safetensors header parser for contrast.

Runs in browserOpen it →

Upstream: Linux memory manager UAPI headers PR #10

What changed: fixed typos in the UAPI headers of an open-source Linux memory manager project. Small, and merged.

MergedPR ↗

Toolbox

Skill → the artifact that shows it

No self-ratings. Each area links to the repo or write-up behind it, with an honest stage.

AreaStageEvidence
Systems & networking
Collectives & NCCLdesign + harness goodput ↗ distributed-training-framework-nccl ↗ ring all-reduce post rail topology post
RDMA / RoCE / PFCrunnable nicprof ↗ kvwire ↗ PFC deadlock post packet path post
Linux kernel performancerunnable kptk ↗
Inference servingearly kvwire ↗ high-performance-llm-inference-engine ↗ KV-cache post
GPU kernels (CUDA / Triton)early custom-cuda-fused-attention-triton ↗
eBPF / XDP / tcearly kernel-level-ai-traffic-shaper ↗ packet path post
DPDKwrite-up packet path post
Incident triage modelsdesign qwen3-nccl-rca-lora model card ↗ NCCL triage post
AI/ML infrastructure security
Threat modeling AI infrawrite-up breach field guide layered threat map
AI agent & tool-use securitywrite-up + design agent blast-radius simulator goodput safety invariants ↗
Model & checkpoint supply chainruns in browser pickle disassembler & scanner
RDMA / fabric securitywrite-up + design RDMA trust-model post kvwire ADR-0005: memory-window fencing ↗
Multi-tenant inference isolationwrite-up prefix-cache side-channel post

Fabric & collectives

NCCL · RDMA verbs · RoCE v2 · InfiniBand · GPUDirect RDMA · PFC / ECN / DCQCN · rail-optimized topologies · NVLink

Host datapath

Linux networking · eBPF / XDP / tc · DPDK · SR-IOV · KVM / virtio · perf / ftrace

Training & serving

PyTorch · vLLM · CUDA · Triton · Ray · Slurm · Kubernetes · DCGM · Prometheus · Grafana

AI-infra security

Threat modeling (STRIDE, MITRE ATLAS, OWASP LLM Top 10) · prompt-injection containment · MCP tool scoping · workload identity (SPIFFE/SPIRE) · Kubernetes RBAC, Pod Security & admission control · eBPF runtime security (Falco, Tetragon)

Supply chain & isolation

safetensors · Sigstore / cosign · SLSA · SBOMs · secrets management · MIG · confidential computing (NVIDIA H100 CC, AMD SEV-SNP, Intel TDX) · IB partition keys · MACsec / IPsec offload · BlueField DPUs

Languages

C · C++ · Python · Go · Bash

Profiles

Where the work lives

GitHub
@nareshns2004

goodput, kvwire, nicprof, kptk and the rest of the source behind this page.

View repositories ↗
Hugging Face
@nareshns2004

Root-cause models for GPU-cluster incidents. Model cards first; weights and evals once goodput’s labeled data lands.

View profile ↗
LeetCode
@nareshns2004

Algorithms and data-structures practice.

View profile ↗
Open to roles

Building training or inference clusters that have to be fast and safe? Let’s talk.

I’m looking for GPU cluster networking, distributed-training infrastructure, inference-platform and AI-infrastructure security roles at AI labs and infra teams.

Open to roles in USACanadaEurope