Serverless LLM Inference Platforms: How They're Architected and What Differentiates Them

Serverless LLM (Large Language Model) inference has become the default entry point for teams deploying generative AI. The pitch is familiar: scale to zero, pay only for what you use, no infrastructure to manage. But underneath that promise lies a sophisticated engineering stack that is fundamentally different from traditional serverless computing — and the differences matter enormously for production deployments.

This report synthesizes findings from eight independent research perspectives, covering the internal architecture of inference platforms, performance optimization techniques, pricing economics, the competitive landscape (Amazon Bedrock, Fireworks AI, Together AI, and a dozen others), developer experience, and the skeptical case against serverless inference for production workloads.

The serving stack: architecture under the hood

A serverless LLM inference platform is not a single monolithic service. It is a layered architecture with three principal components: an API server/router that handles client requests, a scheduler that decides what to batch and how to allocate memory, and GPU workers that execute the model forward pass.

Figure 1 — The three-layer inference serving stack. The overlap scheduler pipelines CPU tasks with GPU computation, critical as GPU execution time drops to ~5ms on modern hardware.

PagedAttention: the foundational memory innovation

The single most important architectural innovation in modern LLM serving is PagedAttention, introduced by the vLLM team at SOSP 2023. The problem it solves: traditional KV (Key-Value) cache allocation reserves contiguous memory blocks sized for the maximum possible sequence length per request. Since most sequences never use their full context window, this leads to 60–80% memory waste through fragmentation.

PagedAttention borrows from OS virtual memory concepts — it stores the KV cache in fixed-size pages, allocated on demand from a shared pool. Memory waste drops below 4%. The result: 2–4× throughput improvement over prior systems with the same latency. TensorRT-LLM extends this with a radix search tree for cross-request cache reuse, prioritized LRU eviction, and KV cache offloading to host CPU memory — a full page cache hierarchy analogous to an operating system’s.

Continuous batching: the throughput foundation

Traditional static batching waits for every sequence in a batch to finish before releasing resources. When generation lengths vary — as they always do in real chat workloads — this causes severe GPU underutilization. Continuous batching (also called in-flight batching or iteration-level batching), introduced by Orca at OSDI 2022, operates at the iteration level: completed sequences are evicted and new sequences injected every single forward pass.

Figure 2 — Continuous batching eliminates the tail latency problem where short sequences wait for the longest one. Benchmarks show up to 23× throughput over naive static batching, 4× over optimized static batching.

Chunked prefill is a critical refinement: it splits long prompt processing into chunks that can be batched with generation-phase tokens, eliminating head-of-line blocking and providing stable Time-To-First-Token (TTFT) across all requests.

Disaggregated serving: separating prefill and decode

The newest architectural pattern separates the prefill (context processing) and decode (token generation) phases onto different GPU pools. The DistServe paper (OSDI 2024) established that colocating these phases causes strong interference — context processing delays token generation, forcing a tradeoff between TTFT and TPOT (Time-Per-Output-Token) optimization. By assigning each phase to different GPUs with independently optimized parallelism strategies, DistServe achieves 4.48× more requests served or 10.2× tighter SLO compliance.

Performance engineering: making inference fast

The quantization landscape

The most comprehensive quantization study to date — Kurtić et al., 2024, with 500,000+ evaluations across Llama-3.1 8B/70B/405B — established clear tradeoffs:

FP8 (E4M3 format, native on Hopper GPUs) is effectively lossless across all model scales, with recovery rates of 99–101% of BF16 accuracy. It’s now the recommended default for H100/H200. INT4 is more aggressive — it cuts model size by 75% and enables 12× more concurrent users on the same GPU — but degrades code generation by 8 points on HumanEval. The key insight: W4A16-INT is most cost-efficient for synchronous (batch-1) deployments, while W8A8 formats dominate in asynchronous continuous batching because 8-bit activation quantization enables faster batch GEMM (General Matrix Multiply) operations that compound at high batch sizes.

Contested finding: The Kurtić study found GPTQ (Generative Pre-trained Transformer Quantization) with MSE-optimal clipping outperforms AWQ (Activation-aware Weight Quantization) on real-world tasks — challenging prior claims that AWQ is superior. This remains contested in the literature, with the discrepancy attributed to hyperparameter choices and calibration data quality.

Speculative decoding and FlashAttention-3

Speculative decoding uses a small draft model to cheaply generate K candidate tokens, then the large target model verifies all K tokens in a single parallel forward pass, achieving 2–3× latency reduction with mathematically equivalent sampling. SpecInfer introduced tree-based speculation, organizing draft tokens in a tree structure to verify multiple candidate continuations simultaneously.

FlashAttention-3 (NeurIPS 2024) exploits Hopper GPU features — WGMMA (Warpgroup Matrix Multiply-Accumulate) for async GEMM, TMA (Tensor Memory Accelerator) for async memory copies, and FP8 support — achieving 1.5–2× speedup over FA2 in BF16 (75–85% H100 utilization vs. FA2’s 35%).

The economics: pricing, utilization, and sustainability

Three pricing models

The market operates on three billing paradigms: per-token pricing (Fireworks, Together, Bedrock — pay for tokens consumed, no idle cost), per-second/per-minute GPU pricing (Baseten at ~$6.50/hr per H100, Modal at per-second rates — pay for wall-clock GPU time), and per-request pricing (Replicate — flat charge per invocation).

For the same model weights — Llama 3.3 70B — per-token prices range from ~$0.12/M tokens (DeepInfra Turbo FP8) to ~$1.05/M (Scaleway). That’s a 9× spread for identical weights, driven by quantization level, batching policy, and loss-leader pricing. A lower per-token price frequently means more aggressive quantization that may degrade output quality.

The utilization crossover

Figure 3 — The utilization crossover. Below ~40-50% sustained utilization, serverless per-token pricing is cheaper. Above ~70%, dedicated GPU-hour economics win. The 50-70% band requires calculation with your specific numbers.

The rule of thumb: below ~40–50% sustained utilization, serverless wins. Above ~70%, dedicated wins. The 50–70% band is where specific numbers determine the winner. Modal’s economic model reduces this to one equation: if the peak-to-average demand ratio exceeds the reservation discount rate, serverless is cheaper — and observed peak-to-average ratios of 5–10× for inference workloads mean serverless wins decisively for spiky traffic.

Cost unpredictability at scale

Per-token pricing is deceptively simple, but actual costs scale non-linearly. Agentic AI workflows consume 5–30× more tokens per task than standard chatbots. The evidence is stark:

• Uber burned through its entire 2026 AI budget in four months. Monthly per-engineer API costs hit $500–$2,000.
• Microsoft cancelled internal Claude Code licenses in May 2026 — token-based billing made costs unmanageable once engineers used the tool at full velocity.
• GitHub moved all Copilot plans to usage-based billing from June 1, 2026, admitting it “can no longer absorb the escalating inference cost.”
• A healthcare enterprise consumed 1 trillion tokens over six months, generating $6 million in unplanned costs.
• 73% of enterprises reported AI costs exceeded original budget projections (FinOps Foundation 2026).

The root cause: flat per-token pricing is wrong by 10–50× per request. A 2,000-token generation on a cold KV-cache uses 10× more GPU compute than a 100-token completion riding a warm cache. The per-token price implies uniform cost, but real GPU cost varies by an order of magnitude depending on context length, cache state, and batch composition.

Deep dive: Amazon Bedrock architecture

Not a proxy — a licensed model marketplace

The single most important architectural fact about Bedrock: it is not a proxy forwarding API calls to Anthropic’s or Meta’s data centers. AWS receives model weights through commercial licensing agreements (with revenue-sharing to providers) and deploys them on AWS-owned infrastructure. When you call Claude through Bedrock, the request is served by a model instance running inside an AWS data center. AWS Trainium2 is reported to handle the majority of inference traffic, alongside NVIDIA GPUs.

Inference tiers and cross-region routing

Bedrock’s on-demand capacity has three tiers: Flex (lowest cost, best-effort, no SLA), Standard (balanced, moderate guarantees), and Priority (highest cost, premium allocation, enhanced SLA). Provisioned Throughput reserves fixed Model Units with no throttling within purchased capacity. Batch inference offers 50% savings with a 24-hour window.

Cross-Region Inference (CRIS) performs real-time capacity checks across Regions, re-routing requests via the AWS network backbone (not public internet), adding “double-digit milliseconds” of latency. CRIS throughput can reach double the default in-region quotas.

Model eviction and cold starts

Bedrock uses an internal eviction policy that removes (unloads) models not invoked for ~1 hour. The next invocation returns ModelNotReadyException during restoration. AWS recommends a heartbeat strategy: a Lambda function pinging the model every 30–50 minutes. This applies primarily to imported/custom models; managed partner models (Claude, Llama) appear to have persistent warm capacity — revealing a two-tier readiness model.

Prompt caching: neural state, not text

Bedrock prompt caching captures the model’s internal computed representation — attention patterns, token embeddings, KV-cache state — at cache checkpoints, not just text. The default TTL is 5 minutes on a sliding window (each hit resets the timer). Cache reads cost ~10% of standard input price (90% discount). Cache is invalidated if even a single character before the checkpoint changes.

What makes Bedrock different

Bedrock is architecturally a governed multi-model marketplace on AWS infrastructure. The customer has zero visibility into or control over the inference engine, instance type, quantization, or serving framework. The value proposition is multi-provider model choice behind one IAM (Identity and Access Management) boundary, enterprise governance, and integration with the broader AWS ecosystem. The trade-off: less optimized latency and throughput than a pure inference specialist. Per-token cost is only ~30% of TCO; engineering time, observability, and compliance are the other 70%.

Deep dive: Fireworks AI architecture

FireAttention: a from-scratch CUDA kernel stack

FireAttention is Fireworks’ custom CUDA (Compute Unified Device Architecture) kernel-based serving engine, built from scratch by former Meta PyTorch engineers — not a wrapper around vLLM or TensorRT-LLM. Four versions have shipped:

The key insight from V1: FP8 floating-point quantization is superior to integer quantization because LLM activations have non-uniform distributions that FP8’s exponent-based format handles better than uniform integer scales.

FireOptimizer and Multi-LoRA

3D FireOptimizer automatically searches across speed, throughput, and quality — the combinatorial space includes model architecture, parallelism strategies, hardware selection, quantization recipes, and speculation strategies, exceeding 100,000 combinations. Adaptive Speculative Execution trains draft models on the customer’s specific data distribution: a generic draft model achieved only 29% hit rate, while a FireOptimizer-trained model achieved 76% hit rate with 2× speedup. Cursor reported ~2× speed improvement for their Fast Apply and Cursor Tab features.

Multi-LoRA enables serving hundreds of LoRA (Low-Rank Adaptation) fine-tuned models on a single base model simultaneously, at the same inference cost as the base model. Cross-Model Continuous Batching processes requests from multiple LoRAs simultaneously, with Dynamic LoRA Adapter Loading (not constrained by GPU memory). Cresta deployed 1,000s of LoRAs achieving 100× cost-efficiency vs. GPT-4.

GPU infrastructure: multi-cloud, not owned

Fireworks rents GPU capacity from 8 cloud providers across 18+ regions, processing 5T+ tokens/day at 100,000+ requests/second. For enterprise customers, BYOC (Bring Your Own Cloud) runs the inference engine inside the customer’s VPC (Virtual Private Cloud).

The competitive landscape: five tiers

Figure 4 — The serverless LLM inference market has crystallized into five structural tiers. Each tier competes on a different axis: speed (custom silicon), serving engine (GPU platforms), procurement (hyperscalers), flexibility (serverless GPU), or optionality (aggregators).

Within the GPU-platform tier, a clear positioning rule has emerged: Fireworks optimizes latency, Together AI optimizes catalog breadth (200+ models), Baseten optimizes enterprise polish (Truss packaging, per-minute billing, $5B valuation), Modal optimizes Python-native developer experience, and DeepInfra optimizes price (cheapest per-token via aggressive quantization).

The compliance gap between hyperscalers and pure-play providers has narrowed: Fireworks, Together, Baseten, and Nebius all carry SOC 2 (System and Organization Controls 2) Type II + HIPAA (Health Insurance Portability and Accountability Act). The remaining hyperscaler differentiators are FedRAMP (Federal Risk and Authorization Management Program) and full parent-cloud compliance depth.

Developer experience: building on these platforms

Nearly every platform has converged on OpenAI-compatible Chat Completions as the primary developer interface. Fireworks, Together, and Baseten support pointing the stock OpenAI SDK at their endpoints with a 2–3 line change. Bedrock is the outlier — its primary SDK is boto3, and its InvokeModel API is a well-known pain point that lacks parameter schema discovery.

LoRA (Low-Rank Adaptation) support varies dramatically:

API abstraction covers only ~20% of switching cost. “The other 80% is buried in places no one looked: system prompts written around a model’s quirks, response format expectations, tool-use schemas, retry logic tuned to a specific provider’s rate limits, and eval suites calibrated to one model’s output distribution.”

The skeptic’s case: where serverless falls short

The cold start problem is structural

Serverless LLM inference cold starts are not the manageable 200–300ms familiar from traditional serverless functions. Production deployments routinely report cold starts exceeding 40 seconds before the first token appears, against ~30ms per token once warm — a 1,000× latency gap. The model weights are the cold state. A 70B model in FP16 requires ~140GB to traverse the PCIe bus into VRAM. Scale-to-zero converts a steady dollar cost into a spiky latency cost: in a workload where inference takes 5 seconds but cold start takes 30 seconds, 85% of wall-clock time goes to waiting rather than processing.

Latency variability: the noisy neighbor problem

Serverless inference shares GPU capacity across tenants, creating noisy-neighbor effects that cannot be eliminated:

• Continuous batching: “Your decode step sits on the same kernel launch as a stranger’s. There is no fair-share scheduler on the inside of a transformer block.”
• KV-cache contention: Long-context tenants reshape queueing dynamics for everyone on the shard. A GPU that can serve 50 concurrent short-context users can be fully saturated by two users running extended research workflows.
• PCIe fabric contention: 10–15% of tail latency burns on bus contention alone.

Independent measurements show p95/p50 ratios of 1.8× to 5×. A Bedrock benchmark showed DeepSeek V3.2 with TTFT of 17.6 seconds on Run 1, 1.2 seconds on Run 2 — a 14× swing within the same session.

When serverless wins vs. loses

The mature production pattern is hybrid: dedicated capacity for baseline traffic (80–90%), serverless for burst and experimental models. But most teams underestimate the operational complexity of managing both modes by 3–5× in the first year.

Where the sources disagree

The utilization crossover point varies from 30% (per-minute billing) to 73% (specific DigitalOcean H200 pricing). It’s not a universal constant — it must be calculated per workload. The directional consensus: below ~40–50%, serverless wins; above ~70%, dedicated wins.

AWQ vs. GPTQ quantization quality remains contested. The Kurtić study found GPTQ superior; prior studies found AWQ superior or tied. Neither is universally better — quality depends on clipping strategy, calibration data, and model architecture.

Cold start latency is contested between marketing (milliseconds via GPU memory snapshotting) and independent reports (15–40+ seconds). The honest answer: warm workers achieve near-instant cold starts, but the first request after scale-to-zero faces full cold-worker latency.

Fireworks’ performance claims (4× faster than vLLM) are self-reported. Independent benchmarks confirm Fireworks has the lowest p95 TTFT among GPU-based hosts (320ms vs. 450ms for Together), but specific multipliers are best-case figures.

Devil’s advocate: what this report misses

The Chinese inference ecosystem is entirely absent — SiliconFlow, ModelScope, and Volcengine serve billions of tokens daily with their own innovations. Small-model inference (under 7B parameters) is underrepresented; for those models, the serverless vs. dedicated question becomes nearly irrelevant. Edge and on-device inference (Apple’s MLX, Qualcomm’s AI Engine) represents a growing alternative to cloud-based inference entirely.

The 23× throughput improvement from continuous batching is measured against naive static batching — a baseline no production system uses. Against optimized static batching, it’s 4×. The cost unpredictability cases (Uber, Microsoft) may reflect organizational maturity gaps rather than inherent platform problems.

All provider performance claims are largely self-reported. The inference landscape evolves rapidly — findings may have a 6–12 month half-life. Failed inference startups might offer cautionary lessons that funded survivors don’t surface.