Caricamento del Technology Radar
Kubernetes for AI inference uses cloud-native primitives to deploy, schedule, scale, observe, and operate model-serving workloads across cloud, on-premises, and edge environments. It is strongest when an organization already has Kubernetes platform engineering, GPU operations, service mesh, CI/CD, policy, and observability practices.
Model-serving frameworks build on Kubernetes rather than replacing it. KServe uses Kubernetes resources such as InferenceService and supports Horizontal Pod Autoscaler-based scaling in Standard mode, while NVIDIA’s Kubernetes device plugin exposes GPU resources, tracks GPU health, and lets containers request GPUs through nvidia.com/gpu (KServe, NVIDIA GitHub).
Keep this in Adopt for organizations already standardized on Kubernetes. Kubernetes alone is not an inference platform, but it is a proven substrate for GPU scheduling, model serving, rollout control, multi-tenancy, ingress, and observability when paired with the right serving and accelerator stack.
InferenceService deployments and can use Kubernetes HPA for autoscaling in Standard deployment mode (KServe).GPU operations are specialized. NVIDIA’s device plugin notes production caveats including limited comprehensive GPU health checking and GPU cleanup features, and recommends Helm for production deployment rather than the basic static DaemonSet example (NVIDIA GitHub).
Autoscaling is not automatic success. KServe Standard mode does not support scale-to-zero, and raw deployment HPA only supports CPU and memory metrics, so GPU saturation, queue depth, token throughput, and request latency may need custom metrics or separate autoscaling patterns (KServe).
Model loading and cold starts can dominate latency. Large models need image strategy, model cache, warm pools, startup probes, rolling updates, and capacity reservations designed explicitly.
Kubernetes does not provide model governance by itself. Teams still need model registry, eval gates, prompt/version tracking, data access policies, audit logs, cost attribution, and safety controls.
Adopt Kubernetes for AI inference when workloads need repeatable operations across cloud, on-premises, and edge environments, or when existing platform teams already operate production Kubernetes. Build a platform layer that includes GPU scheduling, serving controllers, autoscaling, ingress, model caching, telemetry, release strategies, and cost attribution.
Do not ask every ML team to assemble its own serving stack. Provide paved-road templates for small models, GPU-backed models, autoscaling policies, safe rollouts, and observability so inference workloads behave like production services.