NVIDIA Dynamo and LLM-D Assess
Overview
NVIDIA Dynamo and LLM-D focus on disaggregated, high-scale LLM inference across GPU fleets with KV cache optimization (NVIDIA Dynamo). Dynamo is NVIDIA's open-source distributed inference framework that sits above engines such as vLLM, SGLang and TensorRT-LLM (Dynamo GitHub); llm-d is a Kubernetes-native distributed inference stack founded by Red Hat, Google Cloud, IBM Research, CoreWeave and NVIDIA and accepted into the CNCF Sandbox in March 2026 (CNCF). The two are not strict alternatives: NVIDIA's Dynamo 1.0 announcement names llm-d among the open-source frameworks integrating Dynamo components (NVIDIA press release).
Assess for hyperscale self-hosted inference. Requires deep GPU platform engineering; not a substitute for application-level agent design.
Adoption Signals
- NVIDIA announced Dynamo 1.0 as production software on 16 March 2026, stating integration by AWS, Microsoft Azure, Google Cloud and OCI and adoption by companies including Cursor, Perplexity, PayPal and Pinterest (NVIDIA press release).
- NVIDIA's technical blog lists production deployments at AstraZeneca, Baseten, ByteDance, CoreWeave, DigitalOcean, SoftBank Corp. and Together AI, among others (NVIDIA Technical Blog).
- llm-d was accepted as a CNCF Sandbox project on 24 March 2026, supported by AMD, Cisco, Hugging Face, Intel, Lambda and Mistral AI in addition to its founders; the CNCF post reports about 120k tokens/s on Qwen3-32B with 8 vLLM pods on 16 H100 GPUs (CNCF).
- Red Hat made llm-d generally available in Red Hat OpenShift AI 3.0 in October 2025, giving regulated enterprises a supported distribution (Red Hat press release).
- Active release trains as of 30 September 2026: Dynamo v1.5.0 (21 September 2026, about 8,200 GitHub stars) and llm-d v0.10.0 (29 September 2026, about 4,700 stars), which moves llm-d onto vLLM v0.30.0 (Dynamo releases, llm-d releases).
Risks
- Benchmark conditions rarely match yours. The up-to-7x Dynamo result refers to DeepSeek R1 in FP4 on GB200 NVL72 in the SemiAnalysis InferenceX benchmark (NVIDIA Technical Blog); gains on a handful of H100 or L40S nodes serving mid-sized models will be smaller and must be measured.
- Hardware coupling. Dynamo's documentation and optimizations centre on NVIDIA Hopper and Blackwell GPUs (Dynamo GitHub); teams that want AMD or TPU options keep more freedom with llm-d, which explicitly targets multiple accelerator vendors (CNCF).
- Larger blast radius inside the cluster. Disaggregated pods exchange KV cache and requests over the internal network; the vLLM SSRF advisory CVE-2026-24779 explicitly calls out llm-d-style deployments, where a compromised pod can reach other pods (OSV).
- Moving interfaces. llm-d v0.10.0 renamed and merged repositories and deprecated its own container images in favour of upstream vLLM images (llm-d v0.10.0); expect Helm values, image references and routing configuration to need rework across releases.
Pros & Cons
Advantages
- Both projects have production-grade releases: NVIDIA declared Dynamo 1.0 production-ready in March 2026, Red Hat has shipped llm-d as generally available since OpenShift AI 3.0 (October 2025), and llm-d joined the CNCF Sandbox in March 2026.
- They add the pieces a single vLLM deployment lacks at scale: KV-cache-aware and prefix-cache-aware routing, disaggregated prefill and decode pools, KV cache offloading to CPU and storage tiers, and SLA-driven autoscaling.
- Both are Apache-2.0 open source and engine-agnostic to a degree: Dynamo runs vLLM, SGLang and TensorRT-LLM backends, and llm-d builds on the Kubernetes Gateway API Inference Extension and targets NVIDIA, AMD and Google accelerators.
Disadvantages
- The headline gains come from rack-scale hardware: NVIDIA's up-to-7x figure is measured on GB200 NVL72 with wide expert parallelism for DeepSeek R1, a setup most European mid-market GPU estates do not have.
- Operational depth is high: disaggregated serving, KV transfer between pods, gang scheduling and inference-aware gateways add components and failure modes that need dedicated GPU platform engineers.
- Both move fast: Dynamo shipped v1.5.0 in September 2026 and llm-d v0.10.0 deprecated its own CUDA and AWS images and a latency-predictor component, so configurations age quickly.
Recommendation
Assess Dynamo and llm-d only if you already operate vLLM or SGLang in production on a multi-node GPU cluster and have a measured problem they address: poor prefix-cache hit rates under load, long prompts that make prefill and decode compete, or GPU utilization too low to justify the fleet. Run the evaluation on your own models, prompt-length distribution and GPU types, and compare against a tuned single-engine baseline with a simple load balancer rather than against default settings.
For Kubernetes-centred and multi-vendor estates, and for teams on OpenShift, llm-d is the more natural starting point; for NVIDIA-only clusters with Blackwell hardware, Dynamo is worth a parallel test. Neither is a fit for teams serving a few models on one or two nodes, and neither replaces an AI gateway, evaluation or guardrails at the application layer. Keep network policies between serving pods strict and track both projects' security advisories alongside vLLM's.
Sources
Consulting
From radar to implementation
You have assessed the technologies — we help you bring them into production systems safely and measurably.
