DFlash vs MTP: Two Paths to Breaking the Autoregressive Decoding Bottleneck

Large Language Model (LLM) inference is bottlenecked by a physics problem, not an algorithmic one. In autoregressive (AR) decoding — generating one token at a time, left to right — each step requires moving the entire model’s parameters from High-Bandwidth Memory (HBM) to the GPU’s compute cores, only to produce a single token. At batch size 1, the GPU’s compute units sit mostly idle, starved for data. A 70B parameter model in FP8 on an NVIDIA B200 caps at roughly 115 tokens per second — not because the math is hard, but because the memory pipe is narrow.

Speculative decoding broke this bottleneck by borrowing a trick from CPU architecture: just as processors speculatively execute predicted branches and discard mispredictions, a small “draft” model can propose multiple candidate tokens that a large “target” model verifies in a single parallel forward pass — costing roughly the same memory bandwidth as generating one token, but potentially accepting many. The approach is lossless: the output distribution is provably identical to standard AR sampling.

Two approaches now dominate the frontier, and they represent fundamentally different philosophies:

• DFlash uses a lightweight block-diffusion model as the draft model, generating an entire block of 16 draft tokens in a single parallel forward pass via iterative denoising. It requires no modification to the target model. It claims up to 6× lossless single-stream speedup and, on NVIDIA Blackwell GPUs, up to 15× throughput at fixed interactivity.
• Multi-Token Prediction (MTP), productionized by DeepSeek-V3, integrates additional prediction heads directly into the model’s training. The model learns to predict multiple future tokens during pre-training, and these heads can optionally serve as a native draft model at inference time. DeepSeek-V3’s MTP achieves 85–90% acceptance rate, delivering 1.8× speedup.

The central tension: DFlash parallelizes drafting but requires a separate model; MTP serializes prediction but is baked into the target at training time. Neither is universally superior. Both face a fundamental theoretical ceiling that no amount of engineering can break.

Figure 1 — DFlash generates an entire block of 16 draft tokens in one parallel forward pass via block diffusion. MTP uses sequentially chained prediction modules where each depth depends on the previous one.

How Each Mechanism Works

MTP: Sequential Prediction Heads Trained With the Model

DeepSeek-V3’s MTP is not the parallel multi-head design from the original Gloeckle et al. paper (Meta FAIR, ICML 2024). It is a deliberate departure: sequential prediction using chained Transformer modules that maintain a complete causal chain at each depth.

The k-th MTP module consists of four components: a shared embedding layer (shared with the main model), a shared output head (also shared), a dedicated Transformer block unique to that depth, and a projection matrix. The input to the k-th module combines the representation from the previous depth and the embedding of the predicted token, projected through the matrix:

1
h'_i^k = M_k [RMSNorm(h_i^(k-1)); RMSNorm(Emb(t_{i+k}))]

The training objective adds a weighted average of cross-entropy losses across all depths:

1
L_MTP = (λ/D) Σ_{k=1}^{D} L_MTP^k

This densifies the training signal — each position contributes to D+1 predictions instead of 1. DeepSeek-V3 uses D=1, predicting the next 2 tokens total. At inference, the MTP modules can be discarded entirely (the main model functions independently) or repurposed for speculative decoding.

Key distinction: Gloeckle et al.’s original MTP uses n independent output heads on a shared trunk — all predict simultaneously. DeepSeek-V3 chains modules sequentially, preserving the causal chain but making drafting inherently sequential.

DFlash: Block Diffusion as a Draft Model

DFlash’s draft model is a lightweight version of the Block Discrete Denoising Diffusion Language Model (BD3-LM) architecture introduced by Arriola et al. at ICLR 2025. The mechanism works in three layers:

Block decomposition. The token sequence is divided into blocks of size B (typically 16). Within each block, tokens are generated via discrete denoising diffusion — starting from a fully masked state and iteratively unmasking tokens. Across blocks, generation proceeds left-to-right (semi-autoregressive), with each block conditioning on the clean previous blocks.

Three-component attention mask. The architectural core combines: (1) a Block Diagonal Mask allowing bidirectional self-attention within a noised block (enabling parallel denoising); (2) an Offset Block-Causal Mask allowing cross-attention from noised tokens to clean context blocks; (3) a Block Causal Mask for standard causal attention between clean blocks. This is what makes single-pass parallel drafting possible — all positions within a block can attend to each other, unlike in AR drafting where each token depends only on its leftward context.

KV injection. DFlash’s critical innovation over prior diffusion-based drafters is KV injection — it extracts hidden features from a few layers of the target model and injects them into the Key-Value (KV) cache of every draft layer. The target model’s hidden features “implicitly contain information about multiple future tokens,” so the drafter doesn’t need to reason from scratch. This is conceptually similar to EAGLE’s feature-level conditioning, but applied to a diffusion draft model rather than an AR one.

Single-forward-pass drafting. Unlike AR drafters that require B sequential forward passes to draft B tokens, DFlash generates all B tokens in one pass. The draft tokens are then verified by the target model in a standard lossless verification step.

The Fundamental Difference

DFlash parallelizes drafting — the entire block is generated in one forward pass, so drafting cost is flat regardless of block size. MTP serializes prediction — each depth depends on the previous, so drafting is inherently sequential. This has a profound consequence: DFlash can afford a deeper, more expressive draft model (5 layers) than EAGLE-3 (1 layer), because its drafting cost doesn’t scale with the number of draft tokens. A multi-layer DFlash generating 16 tokens has lower latency than a 1-layer EAGLE-3 generating 8 tokens.

The Performance Numbers

DFlash: Single-Stream and Production Serving

On Qwen3-8B with greedy decoding on a single NVIDIA H200, DFlash achieves an average speedup of 4.86× across 7 benchmarks, with a peak of 6.08× on MATH-500. EAGLE-3 averages only 1.76× at tree size 16 — DFlash is approximately 2.4× faster. On SGLang with FA4 on a single B200, DFlash achieves up to 5.1× at concurrency 1, declining to 2.8× at concurrency 32.

The “15× on Blackwell” — Context Matters

NVIDIA measured DFlash on 8× DGX B300 (Blackwell Ultra) for gpt-oss-120b. At an interactivity target of 500–600 tokens/sec per user, DFlash serves >15× more concurrent users than AR decoding at the same per-user interactivity, and ~1.5× more than EAGLE-3.

The “15×” is a throughput-at-fixed-interactivity metric, not a single-stream latency speedup. At batch size 1, DFlash “more than doubles interactivity on Blackwell” (~2×). This is a vendor-reported number (NVIDIA on NVIDIA hardware), with no independent reproduction. Comparing it directly to single-stream MTP numbers without noting the different operating regime is misleading.

MTP: Production Deployment

SGLang reports MTP delivering 1.8× at batch size 1 and 1.5× at batch size 32 on H200 TP8. A production deployment (16× H200) showed +60% output throughput with 3-token MTP. AMD’s benchmark on 8× MI300X showed 2.11× at concurrency 1, degrading to 1.25× at concurrency 64.

Head-to-Head: No Single Winner

The fairest comparison comes from an independent JarvisLabs benchmark on a single H100 using vLLM:

The key finding: MTP has higher per-position acceptance rates on both models, but DFlash wins on the MoE model despite lower acceptance — because MoE forward passes are already cheap (3.8B active parameters), making draft cost and verification overhead more important than acceptance length. On dense models, acceptance length matters more, and MTP’s higher acceptance wins. DFlash won 10 of 11 categories on the MoE model; MTP won all categories on the dense model.

Figure 2 — Speedup declines monotonically with concurrency for all methods as the GPU transitions from memory-bound to compute-bound. MTP wins on dense models; DFlash wins on MoE.

The TTFT Penalty

Both approaches degrade Time-To-First-Token (TTFT) under load, but DFlash is worse. On Gemma 4 31B dense at C=16: baseline TTFT was 192ms, MTP was 1,551ms, DFlash was 3,381ms — 17.6× worse than baseline. On MoE models, the penalty is much smaller because each forward pass is cheaper. This matters for interactive products where users wait for the first word; for batch/offline jobs, it’s irrelevant.

The Skeptical View: Limitations and Failure Modes

DFlash’s Lossless Guarantee Breaks at Non-Zero Temperature

A confirmed bug (GitHub Issue #74 on z-lab/dflash) reveals that the draft sampler ignores the user-supplied temperature while the target sampler receives it. For temperature > 0, the two paths sample from different distributions, so the speculative-decoding distribution guarantees do not hold. A fix PR (#77) was opened, but until it’s confirmed merged, the lossless guarantee holds only for greedy decoding (temperature=0). This is a critical correctness issue for any non-greedy production deployment — and most chatbot serving uses temperature > 0.

DFlash’s High Domain Variance

The AdaFlash paper (Nanjing University / Huawei, ICML 2026) demonstrates that DFlash suffers substantially higher variance than AR drafters. Acceptance length fluctuates from 3.40 on ShareGPT (chat) to 7.09 on MathQA — a 2.1× ratio, versus EAGLE-3’s 1.2× range. The root cause: bidirectional attention captures holistic patterns during training, making the drafter more sensitive to domain shifts at inference time. DFlash’s headline 4.86× average is heavily lifted by math/code tasks; for chat workloads, the realistic speedup is closer to 2.5–2.8×.

Both Degrade Below Baseline at High Concurrency

At C=128, DFlash achieves only 0.76× speedup — slower than standard AR decoding. MTP’s gains shrink to 1.25× at C=64. The DFlash paper reports results up to C=32 but omits C=64 and C=128 where the decline becomes critical. Most production serving operates at higher concurrency than benchmarks report.

MTP’s Production Distribution Mismatch: 14.93% Acceptance

A confirmed vLLM bug report shows DeepSeek-V3.2 MTP achieving only 14.93% acceptance rate in production — far below the ~90% in the paper. The root cause: the MTP module was trained during pre-training but frozen during Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF) post-training. When chat templates are enabled, the main model’s output distribution shifted to chat-formatted text, but the MTP module still reflects the pre-training distribution. Nebius and Perplexity independently confirmed this, requiring them to fine-tune custom MTP heads.

The NTP Specialization Barrier

A systematic study (Mehra et al., Sony/EPFL, 2025) demonstrates that LLM hidden layers become strongly specialized for Next-Token Prediction (NTP) during pre-training, discarding information crucial for MTP. Even joint fine-tuning of the backbone and MTP heads cannot fully overcome this barrier. The paradox: a good NTP model that specializes strongly is more confident in token predictions, but this same specialization makes MTP amortization difficult. MTP works best when baked into pre-training, not bolted on after.

Training Economics: The Cost Comparison

MTP and DFlash are complementary, not mutually exclusive. A model trained with MTP could serve as the target for a DFlash drafter, potentially achieving even higher speedup than either alone. The dominant cost for DFlash is not GPU-hours for draft training, but GPU-hours for data generation — running the target model to produce self-distilled responses on a large, diverse, deployment-matched corpus. As one practitioner found: “training accuracy plateaued at ~0.18-0.20 while the loss kept making small new lows. That pattern means I hit the ceiling of the training set, not the ceiling of the epoch count.”

The Theoretical Ceiling: What No Approach Can Exceed

The “Speculative Decoding Speed-of-Light” paper (Pankratov & Alistarh, EACL 2026) establishes the first tight lower bounds on speculative decoding runtime:

1
E[X] ≈ log(P) / μ

where X is accepted tokens per iteration, P is verification parallelism, and μ is the expected entropy of the target’s output distribution. No matter how good the drafter is, speedup is logarithmic in parallelism and inversely proportional to entropy. Increasing parallelism P exponentially only yields a linear improvement in accepted tokens.

With an imperfect drafter, the bound becomes E[X] ≈ log(P) / μ_CE, where μ_CE is the cross-entropy between target and drafter. This is where the key architectural difference matters:

• AR drafters (EAGLE-3, MTP) suffer the “gradual collapse of trust” — each draft token conditions on previously unverified draft tokens, so discrepancies recursively amplify. The KL divergence between drafter and target grows with draft depth, effectively inflating μ_CE.
• Diffusion drafters (DFlash) avoid this: the proposal at position i is independent of previously drafted tokens. The mismatch is determined solely by how well the drafter matches the target at that context, keeping μ_CE approximately constant across depth.

This is why DEER (a related diffusion-drafting approach) achieves acceptance lengths up to 32 tokens versus 7-8 for EAGLE-3. But there’s a catch: verification cost for eliminating sampling error in parallel decoding grows as exp(TC), where TC is the intra-block total correlation. Larger blocks give more parallelism but exponentially higher verification cost.

The practical implication: the entropy-speedup relationship means code and math tasks (low entropy) fundamentally benefit more than creative writing and chat (high entropy). This is not a limitation of either approach — it’s a limit of speculative decoding itself.

The Practitioner’s Decision Framework

Based on all the evidence, here is the practical decision tree:

Figure 3 — The right approach depends on concurrency, model type, training budget, workload, and hardware. No single method wins everywhere.

If your concurrency exceeds 64–128: Turn off speculative decoding entirely (or use n-gram/suffix, which have zero overhead and never hurt under load). The GPU is compute-saturated and speculation steals memory bandwidth from useful generation.

If you’re serving a MoE model at low-to-medium concurrency: Use DFlash. Its flat drafting cost (one forward pass for 16 tokens) is more valuable when the target’s forward pass is already cheap. DFlash won 10 of 11 categories on the JarvisLabs MoE benchmark and beat native MTP by 1.5× on Qwen 3.5 397B-A17B.

If you’re serving a dense model with native MTP: Use MTP. It has lower overhead, higher per-position acceptance, and scales better at higher concurrency. It’s built into the model — no separate draft model to train or load.

If you’re serving a dense model without native MTP: Use EAGLE-3. It’s the best all-around performer with the most stable acceptance behavior, <10% memory overhead, and broad compatibility. It requires training a small draft head but doesn’t modify the target.

If you cannot train anything: Use n-gram or suffix decoding. They’re training-free, zero-extra-memory, and provide modest speedups without increasing workload during peak traffic — the only methods that don’t hurt under load.

Break-even points: Monitor spec_decode_draft_acceptance_rate in production. Below 0.5, you’re adding latency — turn it off. Above 0.65, you’re seeing genuine benefit. Temperature is the single biggest lever: at temperature=0, acceptance rates are ~0.8+; at temperature=1.0, they can drop to ~0.38, which is net-negative.

How We Got Here: Four Generations of Speculative Decoding

Figure 4 — Four generations of speculative decoding, with block diffusion emerging as the fourth. The parallel track of diffusion text generation converged with the speculative decoding lineage in 2025-2026.

The entire story maps cleanly to processor architecture. CPU branch prediction: the draft model “guesses” future tokens; the target “verifies”; wrong guesses are discarded. Instruction-Level Parallelism (ILP): AR decoding is a serial dependency chain; speculative decoding breaks it through speculation, just as out-of-order execution breaks instruction dependency chains. Compression: language modeling is fundamentally compression, and speculative decoding exploits the fact that most of a text sequence is highly predictable — the acceptance rate directly measures the “predictability” of the sequence.

There were dead ends along the way. Non-Autoregressive Translation (NAT, 2018) tried full parallelism without verification and failed for open-ended generation — the lesson being that full parallelism without a correctness guarantee is insufficient. Massive 7B+ diffusion drafters (DiffuSpec, SpecDiff-2) were too slow without target conditioning. DFlash’s KV injection — feeding the target’s hidden features into every draft layer — is what makes a small diffusion drafter viable.

Where the Sources Disagree

Is DFlash truly “lossless”? The paper says yes, but GitHub Issue #74 shows the draft sampler ignores temperature, breaking the guarantee for any temperature > 0. Reconciled: valid in principle, broken in implementation for non-greedy settings — a code bug, not a theoretical flaw.

Which is faster? The DFlash paper says DFlash roughly doubles MTP’s speedup. JarvisLabs says MTP wins on dense, DFlash wins on MoE. Reconciled: both agree the winner depends on model architecture.

MTP acceptance: 85-90% or 14.93%? Both are correct under different conditions. The 85-90% is pre-training-like data; 14.93% is production chat serving with distribution mismatch from frozen MTP heads.

15× on Blackwell: speedup or throughput? Both framings are accurate but measure different things. The 15× is throughput-at-fixed-interactivity; at batch size 1, it’s ~2×.

The Case That This Report Is Overstated

DFlash’s advantages may be ephemeral. The 2.4× advantage over EAGLE-3 comes from eliminating the sequential drafting bottleneck, but EAGLE-3 uses a single Transformer layer. If AR drafters adopted multi-layer architectures with better feature conditioning, the gap could narrow — DFlash’s own ablation shows KV injection alone accounts for a meaningful portion of the gains, independent of the diffusion mechanism.

No independent party has reproduced DFlash’s 6× single-stream numbers. JarvisLabs showed 3.03× on a dense model. One independent desktop audit reported only 12% acceptance on Qwen 3.6 27B FP8. Both papers measure throughput but largely ignore the serving-system-level costs: dual KV caches, rollback mechanics, and verify-phase bandwidth spikes that may add 15–45% more HBM consumption.

The high-concurrency collapse is underreported. Both papers stop at C=32 but omit C=64/128 where DFlash falls below baseline. Most production serving operates at higher concurrency than benchmarks report. MTP’s 14.93% acceptance is a deployment issue fixable by fine-tuning; DFlash’s high domain variance is a structural property of bidirectional attention that cannot be fixed without changing the architecture.

The theoretical ceiling is closer than it looks. For a typical LLM with μ ≈ 2-3 nats/token and P = 16-64, the theoretical maximum is ~2-4× — not far from what current methods already achieve. Further improvements may yield diminishing returns. And neither paper reports results on structured output generation (JSON, XML, strict-syntax code) — a critical gap for production tool-use that increasingly dominates real workloads.

References

1. DFlash: Block Diffusion for Flash Speculative Decoding — Chen, Liang, & Liu, ICML 2026. arXiv:2602.06036
2. DeepSeek-V3 Technical Report — DeepSeek-AI, Dec 2024. arXiv:2412.19437
3. Better & Faster LLMs via Multi-token Prediction — Gloeckle et al. (Meta FAIR), ICML 2024. arXiv:2404.19737
4. Block Diffusion: Interpolating Between AR and Diffusion LMs — Arriola et al., ICLR 2025 Oral. arXiv:2503.09573
5. Speculative Decoding Speed-of-Light — Pankratov & Alistarh, EACL 2026. arXiv:2512.11718
6. DEER: Draft with Diffusion, Verify with AR Models — Cheng et al., Dec 2025. arXiv:2512.15176
7. Generation Order and Parallel Decoding in Masked Diffusion Models — Zhang et al., ICML 2026. arXiv:2602.00286
8. Fast Inference from Transformers via Speculative Decoding — Leviathan et al. (Google), ICML 2023. arXiv:2211.17192
9. Medusa: Simple LLM Inference Acceleration with Multiple Decoding Heads — Cai et al., ICML 2024. PMLR
10. EAGLE-3: Scaling up Inference Acceleration via Training-Time Testing — Li et al., NeurIPS 2025. arXiv:2503.01840
11. On Multi-Token Prediction for Efficient LLM Inference — Mehra et al. (Sony/EPFL), 2025. arXiv:2502.09419
12. AdaFlash: Adaptive Speculative Decoding via On-Policy Distilled Diffusion Drafters — Nanjing University / Huawei, ICML 2026. arXiv:2607.19223
13. Boost Inference Performance up to 15× on NVIDIA Blackwell Using DFlash — NVIDIA Developer Blog
14. Accelerating SGLang with Multiple Token Prediction — LMSYS Blog, July 2025
15. The Next Generation of Speculative Decoding: DFlash and Spec V2 — LMSYS Blog, June 2026
16. Benchmarking Gemma 4 MTP vs DFlash on a Single H100 — JarvisLabs
17. Efficient LLM Serving with MTP on AMD Instinct GPUs — AMD ROCm Blog
18. DFlash Issue #74: draft sampler ignores temperature — GitHub
19. vLLM Issue #33497: Low acceptance rate for DeepSeek-V3.2 MTP — GitHub
20. Why chat templates reduce DeepSeek MTP accept rate — ROCm/ATOM Issue #609
21. Training a DFlash drafter on a B200 — Jared Frost
22. SpecDecode-Bench: Speculative Decoding — Performance or Illusion? — Project
23. Speculative Decoding Is a Memory Problem — Blog
24. D3PM: Structured Denoising Diffusion Models in Discrete State-Spaces — Austin et al., NeurIPS 2021. arXiv:2107.03006
25. TiDAR: Think in Diffusion, Talk in Autoregression — Liu et al. (NVIDIA), MLSys 2026 Oral. arXiv:2511.08923