Inference Runtime Security: The Hidden Attack Surface in LLM Serving

The inference runtime — the software stack between the API endpoint a customer calls and the GPU hardware where a model actually executes — is the critical trust boundary in any LLM cloud platform. When you call an inference API, your request flows through tokenization, scheduling, batch formation, GPU execution, and response streaming, crossing multiple trust boundaries at each step. The runtimes managing this pipeline — primarily vLLM, SGLang, and TensorRT-LLM — were designed for performance and throughput. They were not designed for multi-tenant security.

This report dissects their internal architecture, identifies seven demonstrated cross-tenant attack surfaces (with working exploits, not theoretical concerns), and maps a defense-in-depth strategy using GPU hardware isolation. The research draws on 100+ primary sources including peer-reviewed papers from SOSP, NDSS, and arXiv, confirmed CVEs, official documentation, and source code analysis across 12 inference runtimes.

The runtime landscape

The LLM inference ecosystem spans four architectural philosophies. Compiled runtimes like TensorRT-LLM convert models to optimized binary engines ahead of time, trading flexibility for maximum kernel-level performance. Interpreted runtimes like vLLM and SGLang execute PyTorch models directly with JIT compilation, supporting any HuggingFace model out of the box. Platform-wrapping layers like Triton, Ray Serve, and KServe wrap inference engines with Kubernetes-native management and autoscaling. Local-first tools like Ollama, MLX, and llama.cpp prioritize zero-config or zero-dependency single-machine inference.

vLLM solves memory waste — its PagedAttention innovation reduced KV cache fragmentation from 60-80% to under 4%, enabling 2-4x throughput improvement [1]. SGLang solves prefix reuse — its RadixAttention automatically shares KV cache across requests with common prefixes, achieving up to 6.4x throughput for multi-turn workloads [2]. TensorRT-LLM solves raw performance — compiler-based kernel fusion achieves world records on NVIDIA hardware [3].

But none of them solve multi-tenant security. That’s the gap this report explores.

vLLM: inside the multi-process architecture

vLLM V1 uses a multi-process architecture where an API Server, Engine Core, and GPU Workers communicate via ZMQ sockets. A typical 4-GPU deployment spawns 6 separate processes; a data-parallel deployment with 8 GPUs spawns 17.

Figure 1 — vLLM V1 architecture. The API Server handles HTTP and tokenization, the Engine Core runs the scheduler and KV cache manager, and GPU Workers execute the model. All inter-process communication uses ZMQ (unauthenticated by default). NCCL handles multi-GPU collectives with no built-in security.

The request lifecycle and its trust boundaries

Tracing a request from API ingress to response return reveals where untrusted data crosses trust boundaries — and where attacks are possible.

Figure 2 — Each step of the request lifecycle accesses different memory (CPU RAM, GPU HBM, ZMQ IPC) and crosses different trust boundaries. The most critical boundary is at GPU execution, where multiple tenants’ requests share the same HBM and L2 cache.

PagedAttention: OS virtual memory for KV cache

PagedAttention’s core insight is an analogy to operating system virtual memory. Blocks are pages, tokens are bytes, sequences are processes, and block tables are page tables. Each sequence’s KV cache is partitioned into fixed-size blocks mapped to physical blocks in GPU HBM via a per-sequence block table. Blocks are allocated on demand and freed when a sequence completes, with Copy-on-Write enabling sharing for parallel sampling and beam search [1].

This is an elegant memory management system. It is also where the security problems begin.

KV cache blocks are not zeroed on free. The free_blocks() method in vLLM’s BlockPool only decrements the reference count and returns the block to the free queue. Stale KV data from the previous request remains in GPU memory. Isolation relies entirely on correct software block table management — there is no hardware-level MMU-like protection for KV cache blocks.

SGLang: RadixAttention and the prefix-sharing problem

SGLang’s RadixAttention replaces vLLM’s flat per-request block tables with a global radix tree that spans all requests. When a new request arrives, the system traverses the tree to find the longest matching prefix and reuses its KV cache without recomputation. This handles multi-turn conversations, few-shot examples, and agent workflows automatically — and it’s why SGLang can outperform vLLM by up to 3.1x on prefix-heavy workloads [2].

Figure 3 — PagedAttention uses per-request block tables with hash-based prefix matching. RadixAttention uses a global radix tree with automatic prefix sharing. The security implication: RadixAttention shares KV cache across all tenants by default, creating both a timing side-channel and a data sharing path.

SGLang does provide an extra_key field in its RadixKey class for namespace isolation — the documentation explicitly states that entries with different extra_key values “are intentionally kept disjoint and never share prefix nodes” and that this is “useful to isolate KV cache lines for different LoRA / adapter IDs or separate requests that intentionally should not share state.” But this isolation is opt-in, not default. Without explicit configuration, all tenants share the same radix tree.

TensorRT-LLM: a different attack surface

TensorRT-LLM takes a fundamentally different approach — it’s a compiled runtime, not an interpreted one. Models are converted to optimized TensorRT engines ahead of time through a build step that applies kernel fusion, precision calibration, and static memory planning.

Figure 4 — The compiled approach eliminates the pickle deserialization attack surface at runtime (no torch.load, no auto_map, no Jinja2 rendering). But it introduces a different surface: no built-in integrity verification for compiled engines, an undocumented binary format, and IPC that was previously unrestricted pickle.

This is a genuine security advantage. A compiled engine is a binary that executes CUDA kernels — there’s no torch.load(), no auto_map code execution, no Jinja2 template rendering. The attack surface shifts from “code execution during loading” to “engine tampering and IPC security.” However, NVIDIA’s documentation explicitly states that “users are responsible for verifying model integrity,” the engine format is undocumented, and a security fix (PR #11683) revealed that IPC weight handles were previously deserialized with unrestricted pickle [4].

The seven demonstrated attacks

Each of these has been demonstrated with working exploits and confirmed CVEs — not merely theorized.

1. KV cache residual data leakage — >80% prompt recovery

When a request completes, its KV cache blocks return to the free pool without zeroing. When Tenant B receives a freed block that previously held Tenant A’s data, the GPU DRAM still contains Tenant A’s attention key and value vectors — a deterministic, invertible function of Tenant A’s input tokens. Luo et al. demonstrated over 80% token accuracy recovering system prompts from residual KV-cache vectors using embedding inversion with beam search [5].

Mitigation: Zero KV cache blocks on free (estimated 2-5% throughput cost), or use per-tenant process isolation.

2. Prefix cache timing side-channel — 62% exact input recovery

Two peer-reviewed papers demonstrate this attack. InputSnatch achieves 87% accuracy determining cache-hit prefix lengths by measuring time-to-first-token, with 62% success extracting exact disease inputs from a medical consultation system [6]. PROMPTPEEK (NDSS 2025) provides the first systematic investigation of KV-cache sharing security risks [7]. The attacker needs only standard API access — no privileged memory access. Prefix cache hits produce measurably faster TTFT than misses.

Mitigation: vLLM’s in-progress “cache salting” (PR #17045) injects a per-request salt into the block hash. SGLang’s extra_key provides namespace isolation. Disabling prefix caching eliminates the attack but sacrifices major throughput.

3. Dynamic quantization cross-batch leakage — 99.6-100% token recovery

A paper from University of Cambridge and Google DeepMind demonstrates that per-tensor dynamic activation quantization — the default in vLLM’s --quantization fp8 mode — creates a cross-user side channel in batched inference [8]. When victim and adversary inputs are co-located in the same batch, the victim’s activation values influence the global min/max computation, producing measurable perturbations in the adversary’s output logits. The attack achieves 99.6-100% token recovery accuracy. At least 44 popular ML frameworks have at-risk configurations.

Mitigation: Use per-token (not per-tensor) dynamic quantization. SGLang’s w8a8_fp8/int8 modes default to per-token and are not vulnerable.

4. Cache poisoning via hash collisions

The “Cache Me, Catch You” paper (NDSS 2026) demonstrates hash collision attacks on vLLM and SGLang prefix caching [9]. Attackers craft prompts that produce hash collisions with victim prompts’ prefix cache entries, causing the system to serve the attacker’s cached KV blocks for the victim’s request. The attack uses a Meet-in-the-Middle birthday attack on Python’s tuple hash. Confirmed CVEs: CVE-2025-25183, CVE-2025-1953, CVE-2025-46722.

5. RCE via pickle deserialization

Multiple confirmed RCE paths during model loading, across all major runtimes:

The SGLang SSTI vulnerability is particularly notable — it’s the first practical malicious-model-file RCE that doesn’t rely on pickle at all. A chat_template field in GGUF metadata is rendered as a Jinja2 template, and the unsandboxed environment allows arbitrary Python execution [10].

6. ShadowMQ — RCE via ZMQ IPC

Oligo Security disclosed a chain of critical RCE vulnerabilities stemming from unsafe ZMQ recv_pyobj() with Python pickle deserialization over unauthenticated TCP sockets [11]. The vulnerability propagated through code reuse — SGLang’s vulnerable file literally began with “Adapted from vLLM.” Confirmed CVEs across vLLM (CVE-2025-30165), TensorRT-LLM (CVE-2025-23254, CVSS 9.3), Meta Llama Stack (CVE-2024-50050), and SGLang (CVE-2026-3059). Thousands of exposed ZMQ sockets were found on the public internet.

7. GGUF integer truncation — silent GPU memory disclosure

Integer truncation in vLLM’s GGUF dequantize kernels causes partial tensor processing (GHSA-5jv2-g5wq-cmr4). The output tensor is allocated via torch::empty (uninitialized memory), but the CUDA kernel processes only a truncated number of elements when tensor dimensions exceed INT_MAX. The unfilled portion retains whatever was previously in GPU memory. No error is generated — the partial fill is completely silent [12].

The supply chain problem

Model supply chain security is the most actively exploited surface. HuggingFace, the primary model source for vLLM and SGLang, has significant security gaps:

• PickleScan is structurally incomplete — a deny-list approach that can’t cover all legitimate ML imports
• The nullifAI bypass (February 2025) evaded PickleScan by wrapping pickle payloads in 7-Zip compression — the scanner didn’t recognize 7z magic bytes while torch.load() successfully decompressed and executed the payload [13]
• 352,000 unsafe or suspicious issues across 51,700 models were found by Protect AI’s Guardian scanner [14]
• 59% of serialized model files still use unsafe pickle-based formats [14]
• A fake “OpenAI” model reached #1 trending on HuggingFace with 244,000 downloads before removal — it deployed a Rust-based infostealer targeting browser credentials and crypto wallets [14]
• Namespace reuse: when HuggingFace authors delete their accounts, attackers can re-register the namespace and backdoor orphaned models referenced from Google Vertex AI and Azure AI Foundry [14]

The safetensors format is the primary mitigation — it’s a pure data format with no execution path on load, externally security-audited, and now the default in PyTorch 2.6.0. But safetensors only protects weight tensor data. The config.json auto_map field, GGUF chat_template Jinja2 rendering, and tokenizer files all provide code execution paths independent of the weight format. Safetensors is necessary but not sufficient.

Hardware-level isolation: the strongest defense

Software-level isolation (block tables, process boundaries) is necessary but not sufficient. GPU memory has no per-process MMU within a single CUDA context — all KV cache blocks live in the same address space, accessible by any kernel. Hardware-level isolation provides the strongest guarantees.

NVIDIA MIG — physical partitioning at the silicon level

MIG (Multi-Instance GPU) creates up to 7 isolated instances with dedicated memory controllers, L2 cache slices, and streaming multiprocessors. Unlike MPS (Multi-Process Service), which only offers percentage-based SM allocation, MIG provides physical partitioning with memory bandwidth QoS, error isolation, and cache partitioning. Supported on A100, H100, H200, and B200.

The critical limitation: NCCL is not supported with MIG, meaning multi-GPU tensor parallelism cannot be used across MIG partitions. Each partition runs a separate, independent inference server. Models must fit within a single partition’s memory budget.

BlueField DPU — isolation outside the host trust boundary

NVIDIA BlueField DPUs run their own Linux OS on embedded Arm cores with full control over the network adapter. Even if the host is compromised, the DPU’s security functions remain intact. Three core functions: offload infrastructure tasks from the server CPU, accelerate networking and security in hardware, and isolate the control plane in a separate trust domain.

For multi-tenant inference: SR-IOV provides per-tenant network isolation with GENEVE encapsulation, DOCA Vault enforces zero-trust file access (ensuring each tenant can only read authorized model weights), and DOCA Argus performs zero-copy memory analysis of the host for runtime threat detection without host-based agents [15].

Confidential Computing — TEE-protected GPU execution

NVIDIA Confidential Computing on H100 and B200 creates a Trusted Execution Environment that encrypts GPU HBM memory, register state, and all data in transit over PCIe. It protects against host-level attacks including hypervisor compromise, with hardware-rooted attestation verifying GPU integrity before accepting work. CC and MIG are mutually exclusive — one tenant per GPU in CC mode [16].

Figure 5 — The four isolation tiers. Tier 1 (same process) is vulnerable to all seven demonstrated attacks. Tier 2 (separate processes) eliminates KV cache, prefix cache, and quantization attacks but shares L2 cache. Tier 3 (MIG + BlueField) provides hardware-isolated compute, memory, cache, and network. Tier 4 (Confidential Computing) adds TEE-protected encrypted HBM and attestation.

Defense-in-depth architecture

No single runtime is safe by default for multi-tenant inference. All three primary runtimes have confirmed CVEs enabling cross-tenant attacks. The question is which is safest after hardening — and the answer depends on the deployment context.

Figure 6 — Recommended Tier 3 architecture. Each tenant runs in a separate microVM with its own vLLM instance on a dedicated MIG partition. The BlueField DPU provides hardware-enforced network and storage isolation below the host trust boundary.

Hardening checklist

vLLM:

• Set load_format="safetensors" (never auto — prevents fallback to pickle)
• Set trust_remote_code=False and audit for hardcoded bypasses
• Enable cache salting (PR #17045) or disable prefix caching for untrusted tenants
• Deploy behind a reverse proxy with endpoint allowlisting
• Never set VLLM_SERVER_DEV_MODE=1 in production
• Disable /invocations, /pause, /resume, /update_weights endpoints
• Run PyTorch ≥ 2.6.0 (where weights_only=True is the default)
• Use per-token quantization (not per-tensor FP8)

SGLang:

• Configure extra_key for per-tenant KV cache namespace isolation
• Use SandboxedEnvironment for Jinja2 template rendering
• Replace ZMQ pickle IPC with msgpack

TensorRT-LLM:

• Sign compiled engines and verify signatures before deployment
• Enable HMAC authentication for MPI/IPC communication
• Use profiling_verbosity=NONE for production engines

Observability for security detection

Standard API metrics (request count, latency, 4xx rate) are insufficient for inference security. A prompt injection attack generates HTTP 200 with correct token counts. Model extraction campaigns run at volumes indistinguishable from legitimate use. Security-relevant signals that should be instrumented:

Log hashed prompt identifiers (SHA-256 truncated) and metadata, not raw prompt content. Otherwise you’re building a PII database in your observability infrastructure.

Edge deployment considerations

For distributed edge deployments, current inference runtimes face challenges they weren’t designed for: resource constraints (T4, L4, or no GPU), intermittent connectivity, model synchronization across hundreds of sites, and communication overhead for pipeline parallelism.

vLLM is the strongest candidate for edge deployment due to its broadest hardware support (including CPU fallback), sleep mode for model swapping on constrained GPUs, richest observability for centralized security monitoring, and highest security patch velocity. llama.cpp serves as a complementary option for the most constrained nodes — zero-dependency C/C++ with the smallest footprint and CPU+GPU hybrid inference for models exceeding VRAM.

A centralized security control plane should scrape all /metrics endpoints via Prometheus federation across edge sites, collect OpenTelemetry traces via an OTLP collector, ingest structured logs via Fluent Bit or Loki, and deploy DCGM Exporter as a DaemonSet on every GPU node for GPU-level telemetry.

Where the sources disagree

SGLang vs vLLM performance: The 3.1x performance gap reported by LMSYS is from mid-2024 and may not reflect current versions. vLLM has adopted similar techniques (prefix caching, chunked prefill), narrowing the gap. SGLang’s RadixAttention provides structural advantages for prefix-heavy workloads, but for workloads without prefix sharing, the difference is smaller. The 3.1x figure should be treated as a historical benchmark.

MIG side-channel resistance: NVIDIA’s documentation claims physical L2 cache partitioning eliminates cache timing attacks. Academic research (Naghibijouybari et al., 2018) has demonstrated GPU side-channel attacks on shared resources. MIG significantly reduces but does not provably eliminate all side-channel risks — shared memory controller arbitration and PCIe bandwidth could theoretically create covert channels. For most practical scenarios, MIG is sufficient. For hyper-sensitive workloads, Confidential Computing provides stronger guarantees.

Safetensors as a complete solution: HuggingFace correctly states that safetensors has no code execution path and passed external security audit. But security research shows that config.json auto_map, GGUF chat_template Jinja2 rendering, and tokenizer files all provide independent code execution paths. Safetensors is necessary but not sufficient — defense-in-depth requires safetensors plus trust_remote_code=False plus sandboxed Jinja2 plus sandboxed loading.

The devil’s advocate

This report may overstate the risk. Several caveats deserve attention:

Attack practicality: Many demonstrated attacks require specific conditions. The KV cache residual data attack requires GPU memory access via a driver bug or physical access — not achievable through a standard API. The prefix cache timing attack requires the attacker to know the victim’s prompt prefix structure. The quantization attack requires top_logprobs API access and batch co-location. In a well-architected platform with API gateways, rate limiting, and orchestration-layer isolation, some attacks may be impractical.

CVE severity vs. real-world impact: Having a CVE doesn’t mean a vulnerability is exploitable in all deployments. The pickle RCE requires loading an attacker-controlled model — in a managed platform where model loading is operator-controlled, this is not directly exploitable by tenants. The ShadowMQ ZMQ RCE requires network access to ZMQ sockets — in a properly firewalled deployment, these are not externally accessible.

Rapid patch velocity: vLLM and SGLang have extremely active communities with rapid security patching. Several CVEs were patched within weeks of disclosure. A deployment running current versions may not be vulnerable to many of these issues.

The counterfactual: If a platform runs one vLLM process per tenant (or per security tier), the KV cache leakage, prefix cache timing, and quantization attacks are all eliminated. The cost is GPU memory duplication, but for 7B-13B models on H100s with MIG, this may be acceptable. In this model, runtime-layer vulnerabilities are less critical than orchestration-layer isolation. If tenants cannot specify arbitrary model IDs (only select from a curated, signed catalog), pickle RCE and trust_remote_code bypass attacks are not directly exploitable.

References

Primary sources (peer-reviewed papers & confirmed CVEs)

1. Kwon et al., “Efficient Memory Management for Large Language Model Serving with PagedAttention,” SOSP 2023 — arxiv.org
2. Zheng et al., “SGLang: Efficient Execution of Structured Language Model Programs” — arxiv.org
3. NVIDIA TensorRT-LLM — github.com/NVIDIA/TensorRT-LLM
4. NVIDIA TensorRT-LLM Engine Builder Documentation; PR #11683 (secure IPC deserialization)
5. Luo et al., “Stealing LLM System Prompts via KV-Cache Side-Channels” — demonstrated >80% token recovery
6. Zheng et al., “InputSnatch: Stealing Input in LLM Services via Timing Side-Channel Attacks” — arxiv.org
7. Wu et al., “I Know What You Asked: Prompt Leakage via KV-Cache Sharing” — NDSS 2025
8. “Dynamic Quantization Can Leak Your Data Across the Batch” — Cambridge/Google DeepMind
9. “Cache Me, Catch You: Cache Related Security Threats in LLM Serving Frameworks” — NDSS 2026 (CVE-2025-25183, CVE-2025-1953)
10. CERT/CC VU#915947 — SGLang SSTI RCE via GGUF chat_template (CVE-2026-5760)
11. Oligo Security, “ShadowMQ: How Code Reuse Spread Critical Vulnerabilities” — oligo.security
12. GHSA-5jv2-g5wq-cmr4 — vLLM GGUF integer truncation
13. “nullifAI: Broken Pickles and the Hugging Face Detection Gap” — ReversingLabs
14. OWASP AISVS C06: Supply Chain Security for Models — github.com/OWASP/AISVS
15. NVIDIA DOCA In-Silicon Security — developer.nvidia.com
16. NVIDIA Secure AI Operations Guide — docs.nvidia.com

Official documentation & source code

17. vLLM Architecture — docs.vllm.ai
18. vLLM Security Documentation — docs.vllm.ai
19. vLLM V1 BlockPool Source — github.com/vllm-project/vllm
20. SGLang radix_cache.py Source — github.com/sgl-project/sglang
21. NVIDIA MIG User Guide — docs.nvidia.com
22. DevZero, “Secure Multi-Tenant AI Clouds with NVIDIA DPUs and MIG” — devzero.io
23. NVIDIA NCCL Documentation — docs.nvidia.com
24. OWASP Top 10 for LLM Applications 2025 — genai.owasp.org
25. Safetensors README — github.com/huggingface/safetensors
26. vLLM Cache Salting RFC — github.com/vllm-project/vllm/issues/16016