vLLM Architecture & Security: A Source-Code-Level Deep Dive

The previous report surveyed the inference runtime landscape — vLLM, SGLang, TensorRT-LLM, and nine other engines — and identified seven demonstrated cross-tenant attacks. This is the deep dive. vLLM is the most widely adopted open-source LLM inference engine, with 84,000+ GitHub stars and 2,000+ contributors. Its core innovation, PagedAttention, solved the 60-80% memory waste problem in LLM serving and enabled 2-4x throughput improvements. But the architectural choices that make vLLM fast — shared KV cache memory pools, continuous batching, prefix caching, ZMQ inter-process communication — also create attack surfaces that have been exploited in practice.

This research goes to the source-code level: the block pool implementation, the scheduler loop, the weight loading pipeline, the attention kernel selection. Eight demonstrated cross-tenant attacks are documented with working exploits, confirmed CVEs, and mitigations. Eighteen CVEs are catalogued across supply chain, IPC, memory, and API security. The result is a picture of a system designed for performance in trusted environments, now being asked to serve untrusted tenants — and the gap between what it provides and what that demands.

The V1 multi-process architecture

vLLM V1, merged in late 2024, was a significant re-architecture. The original V0 engine ran as a single process with async scheduling. V1 separated concerns into a multi-process model: an API Server, an Engine Core, and GPU Workers, each as distinct OS processes communicating via ZMQ sockets.

For a typical 4-GPU deployment (-tp=4), vLLM spawns six processes: one API server, one engine core, and four GPU workers. A data-parallel deployment with eight GPUs (-tp=2 -dp=4) spawns seventeen processes. Each process has a well-defined role and a clear trust boundary — but the boundaries are software, not hardware.

The API Server (FastAPI) handles HTTP ingress, tokenization via HuggingFace’s tokenizer, multi-modal input loading (images, audio, video), response streaming via Server-Sent Events, and detokenization. It’s the only process directly exposed to client traffic.

The Engine Core runs the scheduler — the brain of the system. The V1 scheduler eliminated the prefill/decode phase distinction that V0 used. As the source code comment puts it: “There’s no ‘decoding phase’ nor ‘prefill phase’ in the scheduler. Each request just has num_computed_tokens and num_tokens_with_spec.” At each step, the scheduler tries to assign tokens so each request’s computed tokens catch up to its target. It manages three queues (WAITING, RUNNING, and the deprecated SWAPPED), allocates KV cache blocks via the KVCacheManager and BlockPool, handles chunked prefill natively, and uses recompute-based preemption — discarding and recomputing a request’s KV cache when memory is insufficient, rather than swapping to CPU like V0 did.

GPU Workers each have a ModelRunner that loads model weights (with tensor-parallel sharding), prepares input tensors, executes the forward pass (often via CUDA graph replay for decode steps), runs the sampler, and transfers results back. The attention backend is selected by priority — FlashAttention first, then FlashInfer, then Triton, then the original PagedAttention CUDA kernel as fallback. All backends share the same BlockPool and BlockTable infrastructure, so the memory management is backend-agnostic.

The trust boundary that matters most is at the GPU. All GPU workers share one CUDA context, meaning all GPU memory — model weights, KV cache blocks, activation buffers, intermediate computation results — lives in one flat address space with no per-tenant boundaries. There is no hardware-level MMU for KV cache blocks. Isolation relies entirely on correct software block table management.

PagedAttention and the block lifecycle problem

PagedAttention’s core insight is an analogy to operating system virtual memory. Blocks are pages. Tokens are bytes. Sequences are processes. Block tables are page tables. Each sequence’s KV cache is partitioned into fixed-size blocks (default 16 tokens) mapped to physical blocks in GPU HBM via a per-sequence block table. Blocks are allocated on demand, freed when a sequence completes, and can be shared via Copy-on-Write for parallel sampling and beam search.

This is an elegant memory management system. It is also where the security problems begin.

Figure 1 — The KV cache block lifecycle. When a block is freed, free_blocks() only decrements the reference count and returns the block to the free queue. No GPU memory zeroing occurs. When Tenant B receives the freed block, the GPU DRAM still contains Tenant A’s attention key and value vectors — a deterministic, invertible function of Tenant A’s input tokens.

The free_blocks() implementation in vllm/v1/core/block_pool.py is the root of the first and most serious attack:

1
def free_blocks(self, ordered_blocks):
2
    for block in ordered_blocks:
3
        block.ref_cnt -= 1
4
        if block.ref_cnt == 0 and not block.is_null:
5
            if block.block_hash is None:
6
                blocks_without_hash.append(block)
7
            else:
8
                blocks_with_hash.append(block)
9
    self.free_block_queue.prepend_n(blocks_without_hash)
10
    self.free_block_queue.append_n(blocks_with_hash)
11
    # No torch.zero_(). No cudaMemset. No zeroing of any kind.

When this freed block is allocated to a new request from a different tenant, the physical GPU memory still contains the previous request’s key and value vectors. These vectors are a deterministic function of the input tokens and the model weights — and for open-source models, the weights are public. Given the key vectors and the model’s W_K matrix, one can solve for the hidden states and apply embedding inversion to recover the original input text with over 80% token accuracy on system prompts up to several hundred tokens.

Two bugfix PRs addressed zeroing — PR #35219 for Mamba models and PR #39283 for FullAttention models — but both were motivated by a correctness bug (stale data causing NaN propagation through masked softmax), not security. The zeroing applies to specific model types and conditions, not a general security guarantee. CVE-2026-7141 (CVSS 9.8) formalized the vulnerability where recycled KV cache blocks hold stale data from prior requests.

Prefix caching: the timing side-channel

Prefix caching, enabled by default in V1, adds another attack surface. When a new request shares a token prefix with a cached block, vLLM reuses the existing KV cache instead of recomputing it. Each block is hashed using SHA-256 (as of v0.11) of its parent hash, its token IDs, and extra keys including LoRA IDs and, since PR #17045, a per-request cache_salt.

Cache hits produce measurably faster Time-to-First-Token than misses. For a 1,000-token prefix with Llama-3-8B, a cache hit reduces TTFT from approximately 200ms to 20ms — a 10x difference. An attacker who can issue requests to the same vLLM process can construct candidate prompt prefixes, measure TTFT, and determine whether another tenant’s request used a matching prefix. By iteratively extending the prefix one token at a time, the attacker reconstructs the victim’s prompt.

vLLM’s cache salting (PR #17045, merged April 2025) injects a per-request cache_salt into the first block’s hash. Since hashes are chained, the salt propagates through all subsequent blocks. Only requests with the same salt can share cached blocks. This makes cross-tenant cache collisions cryptographically infeasible — but it’s opt-in, not default, and it doesn’t address residual data leakage from unzeroed blocks.

The eight demonstrated attacks

Each of these has been demonstrated with working exploits, confirmed CVEs, or peer-reviewed publications — not merely theorized.

Figure 2 — The eight attacks and their mitigation status. Four are rated critical with confirmed CVEs. The mitigation column shows which controls address each attack — cache salting, process isolation, MIG partitioning, safetensors, and version upgrades.

1. KV cache residual data leakage — >80% prompt recovery

When a freed KV cache block is allocated to a different tenant’s request, the GPU DRAM still contains key and value vectors from the previous request. Luo et al. (2024) demonstrated over 80% token accuracy recovering system prompts from residual KV-cache vectors using embedding inversion with beam search. The key vector for token position i is K_i = W_K · h_i, where h_i is the hidden state derived from the token embedding and all prior context. Given W_K (public model weight) and K_i, solving for h_i and inverting to recover the input token is a constrained optimization problem.

CVE-2026-7141 (CVSS 9.8) formalized this. PR #39283 added zeroing for recycled FullAttention blocks, but the fix addresses NaN propagation correctness, not security.

2. Prefix cache timing side-channel — 87% accuracy, 62% exact extraction

InputSnatch (arXiv 2411.18191) achieves 87.13% accuracy determining cache-hit prefix block counts by measuring TTFT variations, with 62% success extracting exact disease input fields from a medical consultation system. The attack requires only standard API access — no privileged memory access, no model knowledge. CVE-2025-46570 was filed for this vulnerability, fixed in v0.9.0 via cache salting.

3. Dynamic quantization cross-batch leakage — 99.6-100% token recovery

The Quantamination paper (Cambridge/Google DeepMind, arXiv 2604.26505) demonstrates that per-tensor dynamic activation quantization — the default in vLLM’s --quantization fp8 mode — creates a cross-user side channel in batched inference. When victim and adversary inputs are co-located in the same batch, the victim’s activation values influence the global min/max computation, producing measurable perturbations in the adversary’s output logits. The attack achieves 99.6-100% token recovery accuracy using only batch co-location and observation of top-1 log probabilities.

The safe alternative is per-token or per-block quantization granularity, where the scale for one request’s tokens doesn’t encode information about another request’s tokens. vLLM’s llm-compressor pipeline with FP8_DYNAMIC uses per-token granularity and is not vulnerable — but the simpler --quantization fp8 online mode uses per-tensor and is.

4. Cache poisoning via hash collisions

The “Cache Me, Catch You” paper (NDSS 2026) demonstrates that vLLM’s prefix caching originally used Python’s built-in hash() function, which uses SipHash — not cryptographically secure. As of Python 3.12, hash(None) returns a predictable constant, making hash collision exploitation feasible. An attacker crafts prompts that produce hash collisions with victim prompts, causing the attacker’s cached KV blocks to be served to the victim’s request. CVE-2025-25183 was filed for this; the fix was switching to SHA-256 hashing (default since v0.7.2/v0.11).

5. RCE via pickle deserialization — CVE-2025-24357

vLLM’s hf_model_weights_iterator uses torch.load() to load model checkpoint files. Before v0.7.0, weights_only defaulted to False, meaning a malicious pickle file would execute arbitrary code during deserialization. An attacker publishes a malicious model on HuggingFace with a crafted pickle payload; when vLLM loads it, the __reduce__ method runs arbitrary Python on the inference host.

The fix was adding weights_only=True in v0.7.0. But CVE-2025-32434 showed that weights_only=True itself was bypassable on PyTorch versions before 2.6.0 — meaning deployments running PyTorch 2.5.x remained vulnerable even after the vLLM fix. PyTorch 2.6.0 made weights_only=True the default and properly enforced it.

6. RCE via ZMQ IPC — ShadowMQ (CVSS 10.0)

In November 2025, Oligo Security disclosed ShadowMQ — a class of vulnerabilities affecting 30+ AI inference engines. The root cause: unsafe code reuse combining ZeroMQ sockets with Python’s pickle deserialization. The vulnerable pattern:

1
data = socket.recv()          # Receive raw bytes over ZMQ
2
obj = pickle.loads(data)     # RCE: deserializes arbitrary Python objects

vLLM’s V0 engine used this pattern for inter-node communication. CVE-2025-32444 (CVSS 10.0) affected the Mooncake KV cache transfer integration, where ZMQ sockets were bound to all network interfaces. An attacker with network access could send a crafted pickle payload to achieve unauthenticated RCE. The V0 multi-node variant (GHSA-9pcc-gvx5-r5wm) remains unfixed — vLLM declined to patch it, recommending network isolation instead.

V1 addressed this by replacing pickle with msgpack for inter-process IPC. But inter-node communication still uses PyTorch Distributed (TCPStore, NCCL), which is unencrypted and unauthenticated by design.

7. GGUF integer truncation — silent GPU memory disclosure

The GGUF dequantize kernel in csrc/quantization/gguf/gguf_kernel.cu uses a 32-bit int parameter for element count. When the product of tensor dimensions exceeds INT_MAX (2^31-1), the parameter silently truncates. The output tensor is allocated at full size via torch::empty() (uninitialized memory), but the kernel only fills a truncated portion. The unfilled remainder retains whatever was previously in GPU memory. No error is generated — the partial fill is completely silent.

CVE-2026-53923 was filed for this. A broader security audit documented 221 additional int64-to-int32 truncation sites across vLLM’s CUDA/C++ extensions, suggesting this is a systemic pattern.

8. PROMPTPEEK — “I Know What You Asked” (99% recovery, NDSS 2025)

The PROMPTPEEK attack, presented at NDSS 2025, exploits changes in serving order as a side-channel. When a request hits the prefix cache, it is served faster and changes the relative scheduling order of batched requests. An attacker monitors these order changes to infer cache hits and extracts one token at a time from another user’s prompt. The paper demonstrated 99%+ prompt recovery in multi-tenant settings, working against both vLLM and SGLang, requiring only the ability to create accounts and issue strategically timed queries.

The supply chain problem

Model supply chain security is the most actively exploited surface. vLLM performs no hash verification or signature checking on model files. The entire supply chain trust model relies on HuggingFace Hub’s PickleScan, which has known bypasses.

Figure 3 — The auto_map field in config.json maps standard AutoModel class names to custom Python module paths. When vLLM encounters it, get_class_from_dynamic_module() fetches the referenced module from the model repository and executes its code. The trust_remote_code flag is supposed to gate this, but has been bypassed three separate times.

Three trust_remote_code bypasses

The auto_map field in HuggingFace model config.json is a mechanism for custom model architectures not natively supported by Transformers. It maps standard AutoModel class names to custom Python module paths — and loading those modules executes arbitrary Python code from the model repository.

The trust_remote_code flag is intended to be the security gate. When set to False, loading of auto_map dynamic modules should be blocked. But three confirmed CVEs demonstrate that this gate has been repeatedly bypassed:

The third bypass is particularly concerning — two model implementation files in vLLM’s own codebase hardcoded trust_remote_code=True when loading sub-components, bypassing the user’s explicit --trust-remote-code=False security opt-out. This means a user who carefully set the flag to False was still vulnerable to RCE via malicious model repositories.

The enforcement is fundamentally fragile. Each model implementation file must independently respect the trust_remote_code setting. There is no central enforcement mechanism. New model implementations can reintroduce bypasses by hardcoding True, as CVE-2026-27893 demonstrated.

HuggingFace Hub security gaps

HuggingFace, the primary model source for vLLM, has significant gaps:

• PickleScan uses a deny-list approach that can’t cover all legitimate ML imports — at least 7 Python standard library modules providing direct command execution are not blocked
• The nullifAI bypass (February 2025) evaded PickleScan by wrapping pickle payloads in compression that the scanner didn’t recognize while torch.load() successfully decompressed and executed them
• Namespace reuse: when HuggingFace authors delete their accounts, attackers can re-register the namespace and backdoor orphaned models
• A fake “OpenAI” model repository reached #1 trending on HuggingFace with 244,000 downloads before removal — it deployed a Rust-based infostealer

vLLM currently meets zero OWASP AISVS C06 (Supply Chain Security for Models) requirements at any level. There is no model scanning, no source allowlisting, no integrity verification, no behavioral testing, and no AI BOM generation.

The API security illusion

vLLM’s --api-key flag provides Bearer token authentication, but only for paths starting with /v1. Over fifteen other endpoints remain unprotected even when the API key is configured.

The /invocations endpoint — a SageMaker-compatible inference endpoint — routes to the same inference functions as the protected /v1/chat/completions endpoint, but requires no authentication. This is a complete authentication bypass for inference capabilities.

Operational control endpoints are similarly unprotected: /pause causes denial of service, /update_weights allows unauthenticated model weight modification, /abort_requests kills in-flight requests. In dev mode (VLLM_SERVER_DEV_MODE=1, which should never be enabled in production), /collective_rpc allows arbitrary method execution on the engine core, and /server_info exposes detailed server configuration including model architecture and memory layout.

vLLM’s own security documentation explicitly warns: “Do not rely exclusively on --api-key for securing access to vLLM.” The recommended approach is a reverse proxy that blocks all non-/v1 paths.

Comparative security: vLLM vs SGLang vs TensorRT-LLM

vLLM is safer by default than SGLang, which had unauthenticated ZMQ RCE (CVE-2026-7301, CVSS 9.8) and whose maintainers were non-responsive during CERT/CC coordinated disclosure. But “safer by default” is not the same as “safe” — all three require significant hardening for multi-tenant production deployment.

Defense-in-depth: four deployment tiers

Figure 4 — The four isolation tiers. Each tier cumulatively eliminates more attacks. MIG + BlueField (Tier 2) is the recommended balance of security and cost for most multi-tenant deployments.

Tier 4 — Shared Process (default): All eight attacks are applicable. No isolation between tenants. Only acceptable for single-organization, trusted internal users.

Tier 3 — Process Isolated: One vLLM process per tenant. Eliminates KV cache leakage, prefix cache timing, quantization leakage, cache poisoning, and PROMPTPEEK. L2 cache side-channels remain because co-located processes share GPU L2 cache. The cost is GPU memory duplication — each process needs its own model weights.

Tier 2 — MIG + BlueField (recommended): NVIDIA MIG partitions an A100 or H100 into hardware-isolated instances with separate DRAM banks, L2 cache partitions, and compute engines. The BlueField DPU provides network and storage isolation outside the host trust boundary. This eliminates all GPU memory side-channel attacks. The limitation: NCCL is not supported across MIG partitions, so multi-GPU tensor parallelism cannot be used.

Tier 1 — Confidential Computing: H100/Blackwell CC mode encrypts GPU HBM at the hardware level with remote attestation. Protects against infrastructure provider access and physical attacks. CC and MIG are mutually exclusive — one tenant per GPU.

A critical misconception: Kubernetes namespaces are API-layer abstractions that control RBAC. They have no relationship to GPU memory management or CUDA context isolation. Two pods in different Kubernetes namespaces, scheduled to the same node and sharing a GPU, share the same GPU memory pool.

The hardening checklist

Critical (must do):

• Set --api-key to a strong 256-bit random value
• Set --host 127.0.0.1 (behind a reverse proxy)
• Deploy behind a reverse proxy with endpoint allowlisting (block all non-/v1)
• Disable prefix caching or use cache salting
• Never set VLLM_SERVER_DEV_MODE=1 in production
• Never set VLLM_ALLOW_RUNTIME_LORA_UPDATING=True
• Use sha256 for prefix-cache hashing
• Use safetensors format exclusively
• Set --trust-remote-code=False (ensure vLLM ≥ 0.18.0)
• Upgrade to PyTorch ≥ 2.6.0
• Use static (pre-quantized) models, not online FP8
• Upgrade to vLLM ≥ 0.8.0 (ShadowMQ fix), ≥ 0.18.0 (trust_remote_code fix), ≥ 0.23.1rc0 (GGUF fix)

Recommended:

• MIG partitioning for hardware-level tenant isolation
• BlueField DPU for network isolation
• Model quarantine pipeline: download → scan → verify → convert → validate → promote
• Block /update_weights, /init_weight_transfer_engine at the proxy
• Set VLLM_MAX_N_SEQUENCES to 64-128 for public deployments
• readOnlyRootFilesystem: true in container security context

Future directions

Academic mitigations are emerging. SafeKV (arXiv:2508.08438) proposes selective KV-cache sharing — keeping sensitive entries in per-user private caches while sharing non-sensitive entries globally, reducing overhead by up to 40% compared to full isolation. CachePrune (arXiv:2605.23640) identifies sensitive tokens within cached sequences and selectively shares only non-sensitive portions. KV-Cloak (NDSS 2026) uses obfuscation to protect against reconstruction attacks, reducing attack output quality to random noise while maintaining model accuracy. PipeLLM (ASPLOS 2025) achieves confidential computing for LLM serving with less than 19.6% throughput overhead by overlapping encryption and GPU computation.

The vLLM community has implemented several security improvements: cache salting (PR #17045), SHA-256 default hashing, the V1 msgpack IPC, LMCache cache_salt propagation, and FIPS-aware configuration. But fundamental gaps remain — no model weight integrity verification, no comprehensive endpoint authentication, no inter-node communication encryption, and no built-in rate limiting.

References

1. Kwon et al., “Efficient Memory Management for Large Language Model Serving with PagedAttention,” SOSP 2023 — arxiv.org
2. Luo et al., “Stealing LLM System Prompts via KV-Cache Side-Channels” — demonstrated >80% token recovery
3. Zheng et al., “InputSnatch: Stealing Input in LLM Services via Timing Side-Channel Attacks” — arxiv.org
4. Wu et al., “I Know What You Asked: Prompt Leakage via KV-Cache Sharing in Multi-Tenant LLM Serving” — NDSS 2025, DOI:10.14722/ndss.2025.241772
5. “Dynamic Quantization Can Leak Your Data Across the Batch” (Quantamination) — Cambridge/Google DeepMind, arxiv.org
6. Wu et al., “Cache Me, Catch You: Cache Related Security Threats in LLM Serving Frameworks” — NDSS 2026, DOI:10.14722/ndss.2026.242812
7. Oligo Security, “ShadowMQ: How Code Reuse Spread Critical Vulnerabilities Across the AI Ecosystem” — oligo.security
8. vLLM Security Documentation — docs.vllm.ai
9. vLLM GitHub Security Advisories — github.com/vllm-project/vllm/security/advisories
10. vLLM Cache Salting PR #17045 — github.com/vllm-project/vllm/pull/17045
11. GHSA-5jv2-g5wq-cmr4 — vLLM GGUF integer truncation (CVE-2026-53923)
12. CVE-2026-27893 / GHSA-7972-pg2x-xr59 — Hardcoded trust_remote_code=True
13. SafeKV: “Selective KV-Cache Sharing to Mitigate Timing Side-Channels” — arxiv.org
14. “Shadow in the Cache: Unveiling and Mitigating Privacy Risks of KV-cache in LLM Inference” (KV-Cloak) — NDSS 2026, arxiv.org
15. PipeLLM: “Fast and Confidential Large Language Model Services” — ASPLOS 2025
16. OWASP AISVS C06: Supply Chain Security for Models — github.com/OWASP/AISVS
17. NVIDIA MIG User Guide — docs.nvidia.com
18. vLLM V1 BlockPool source: vllm/v1/core/block_pool.py — github.com/vllm-project/vllm
19. SGLang CVE-2026-5760 (SSTI RCE) — kb.cert.org
20. vLLM Integer Truncation Audit — github.com/Aviral2642/vllm-integer-truncation-audit