vLLM Trial

Overview

vLLM is a high-throughput LLM inference engine using PagedAttention and continuous batching for GPU clusters (vLLM docs). It started in UC Berkeley's Sky Computing Lab (vLLM GitHub), has been a PyTorch Foundation hosted project since May 2025 (PyTorch Foundation), and exposes an OpenAI-compatible HTTP server (OpenAI-compatible server).

Trial as the serving layer behind internal model endpoints when latency and cost per token matter. Pair with autoscaling, model versioning, and safety filters at the gateway.

Adoption Signals

• Project scale as of 30 September 2026: about 93,000 GitHub stars, more than 2,000 contributors, Apache-2.0 licence, and v0.30.0 released on 22 September 2026 (vLLM GitHub, vLLM releases).
• Neutral governance: the PyTorch Foundation accepted vLLM as a hosted project on 7 May 2025 as one of the first projects under its new umbrella structure (PyTorch Foundation).
• Named large-scale production use: LinkedIn serves more than 50 GenAI use cases on vLLM across thousands of hosts, with sub-600 ms latency targets for AI Job Search query understanding (LinkedIn Engineering).
• Commercial backing: Inferact, founded by vLLM's original creators, raised a USD 150 million seed round at a valuation of USD 800 million in January 2026 to commercialize the engine (TechCrunch).
• Default engine inside distributed serving stacks: llm-d, a CNCF Sandbox project, bridges KServe-style control planes and vLLM (CNCF) and is the distributed inference layer of Red Hat AI 3 (Red Hat), and NVIDIA Dynamo supports vLLM as one of its three backends (Dynamo GitHub).

Risks

• Exposed attack surface. CVE-2026-22778 allowed unauthenticated remote code execution through a crafted video URL in versions 0.8.3 to 0.14.0 (Orca Security), and CVE-2026-24779 allowed SSRF into internal networks, which the advisory flags as particularly critical in containerized environments such as llm-d (OSV).
• Partial authentication. The vLLM security guide states that --api-key covers only some path prefixes and warns not to rely on it alone; it also documents a prefix-cache timing side channel that needs cache salting in multi-tenant setups (vLLM security).
• Upgrade churn. Releases v0.28.0 to v0.30.0 moved bitsandbytes to an out-of-tree plugin, deprecated Model Runner V1 and the legacy api_server entrypoint, and removed deprecated environment variables (vLLM releases); unpinned images can break serving configurations.
• Tuning is workload-specific. LinkedIn's gains came from scheduler parameters, prefix caching and its own upstream patches (LinkedIn Engineering); default settings rarely reproduce published throughput on different models and GPUs.

Recommendation

Trial vLLM as the engine behind self-hosted open-weight model endpoints when a team has steady traffic, its own GPUs or reserved cloud GPUs, and a reason not to use a managed API, such as data residency or cost per token at volume. Put it behind an authenticating gateway or LiteLLM-style proxy, keep it on a private network, disable multimodal media fetching unless it is needed, and set up Prometheus metrics and a load test with your own prompts before the first production route.

Pin an exact version, subscribe to the project's security advisories, and plan a patch path measured in days rather than quarters. For small internal workloads or developer laptops, a simpler runtime or a managed endpoint is usually enough; move to llm-d or NVIDIA Dynamo only once a single vLLM deployment per model no longer meets latency or cost targets.

Sources

• vLLM docs
• vLLM GitHub
• vLLM releases
• vLLM OpenAI-compatible server
• vLLM security guide
• PyTorch Foundation welcomes vLLM
• LinkedIn Engineering: How we leveraged vLLM
• TechCrunch: Inferact raises USD 150M
• Orca Security: CVE-2026-22778
• OSV: CVE-2026-24779
• CNCF: Welcome llm-d
• NVIDIA Dynamo GitHub
• Red Hat AI 3 press release