Inference-Time Compute Scaling and the Overthinking Problem: How Adaptive Budget Allocation Is Reshaping Reasoning Models

For five years, the dominant strategy for improving AI language models was straightforward: train bigger models on more data. The scaling laws established by Kaplan et al. (2020) and refined by Hoffmann et al. (Chinchilla, 2022) told the industry that performance was primarily a function of training compute — parameters and data. Inference was a fixed downstream cost.

That assumption has been overturned. The industry has shifted to inference-time compute scaling: spending more compute at generation time — letting models “think longer” via extended chain-of-thought (CoT), tree search, parallel sampling, and iterative revision — rather than building ever-larger models. The shift was catalyzed by Snell et al. (2024), who demonstrated that test-time compute can substitute for model parameters, and commercialized by OpenAI’s o1 and DeepSeek-R1, which use reinforcement learning (RL) to train models that generate long internal reasoning chains.

But this paradigm shift revealed a paradox: more thinking is not always better. Multiple independent studies confirm an inverted U-shaped accuracy curve — accuracy rises with reasoning token count, peaks at an intermediate budget, then declines. Models second-guess correct answers, waste tokens on trivial problems, and enter unproductive reasoning loops. This “overthinking” phenomenon has spawned a new research subfield focused on adaptive budget allocation: frameworks that dynamically decide not just how to think harder, but when and how much.

Figure 1 — The trajectory from Kaplan’s scaling laws (2020) through Chinchilla, o1, DeepSeek-R1, the overthinking discovery, and the adaptive allocation wave (2026).

The Historical Shift: From Training Compute to Inference Compute

”Scale Is All You Need” (2020–2022)

In January 2020, Kaplan et al. at OpenAI published “Scaling Laws for Neural Language Models,” demonstrating that language model loss follows smooth power-law relationships with model size, dataset size, and training compute. The message was clear: invest in larger models and more training data. Inference was a fixed cost. GPT-3 (175B parameters) embodied this philosophy.

The Chinchilla paper (Hoffmann et al., 2022) corrected a critical misreading: models were “significantly undertrained” relative to data. For compute-optimal training, model size and training data should scale equally. The 70B Chinchilla model outperformed the 175B GPT-3 despite being less than half the size. But this was still a training-time correction — the paradigm of “spend more at training” remained unchallenged.

The Seeds of Inference-Time Scaling (2022)

Two papers from Google Research planted the seeds:

• Chain-of-Thought (CoT) prompting (Wei et al., January 2022) showed that prompting LLMs to generate intermediate reasoning steps dramatically improved performance without any model retraining. The same parameters, given a different prompt, could reason better. Cited over 32,000 times.
• Self-consistency (Wang et al., March 2022) introduced horizontal scaling: sample N diverse reasoning paths at non-zero temperature, then majority-vote the final answer. This made inference-time compute a tunable knob with predictable, monotonic accuracy gains.

The Turning Point (2024)

In August 2024, Snell et al. at Google DeepMind published “Scaling LLM Test-Time Compute Optimally,” making the paradigm shift explicit. The central finding: for a given compute budget, spending it on inference-time computation could be more effective than spending it on training a larger model. Test-time compute can substitute for up to a 14× larger pretrained model on easy and medium questions.

Weeks later, OpenAI launched o1, the first major commercial product built on inference-time scaling. o1’s headline feature was not a larger model but a model that “spends more time thinking before it responds.” o1 ranked in the 89th percentile on competitive programming and exceeded human PhD-level accuracy on a physics benchmark.

Democratization and the Backlash (2025–2026)

In January 2025, DeepSeek published DeepSeek-R1, proving that o1-style reasoning models could be built without proprietary infrastructure. Using Group Relative Policy Optimization (GRPO) — an RL algorithm that computes relative advantages within groups of sampled outputs rather than requiring a separate critic model — with rule-based verifiable rewards, DeepSeek showed that reasoning behaviors emerge from RL alone, without human-annotated reasoning data. The paper has been cited over 10,400 times.

But researchers noticed an uncomfortable pattern: these models thought too much on simple problems. Chen et al. documented that when asked “what is 2+3?”, QwQ-32B-Preview produced 13 separate solution attempts spanning 901 tokens — when the first 39 tokens already contained the correct answer, an outcome efficiency of just 4.3%. This was the turning point where the narrative shifted from “more inference compute is always better” to “inference compute must be calibrated to problem difficulty.”

The adaptive allocation wave of 2025–2026 is “Chinchilla for inference-time compute”: the realization that the optimal allocation is not “more of everything” but “the right amount for each problem.” The same cycle that corrected parameter-heavy training (Chinchilla) now corrects uniform inference scaling.

The shift echoes the CPU clock speed → multi-core transition (2004–2006): when a dominant performance lever hits diminishing returns, the field pivots to a previously under-exploited lever, initially over-uses it, and then learns to calibrate.

The Scaling Laws: Mathematical Foundations

The Chinchilla Baseline

The foundational training-time scaling law models loss as:

1
L(N, D_tr) = E + A/N^α + B/D_tr^β

where N is parameters, D_tr is training tokens, with fitted coefficients A = 406.4, B = 410.7, E = 1.69, α ≈ 0.34, β ≈ 0.28. Both parameters and data contribute as separate power-law terms with exponents < 1, meaning diminishing returns.

The Training-Inference FLOPs Exchange Rate

The FLOPs (floating-point operations) approximation provides the exchange rate between training and inference compute:

• Training: 6 × N × D_pretrain FLOPs (forward + backward passes)
• Inference: 2 × N × D_inference FLOPs (forward pass only)

The 3:1 ratio arises because training requires both forward and backward passes. Sardana & Frankle showed that the amount of test-time compute you can “afford” to substitute for a larger model depends critically on R = D_inference / D_pretrain — the ratio of inference tokens to pretraining tokens.

No Analytic Solution for Inference-Aware Scaling

Sardana & Frankle proved that when inference costs are included, the compute-optimal model size and training token count cannot be solved analytically from the Chinchilla parametric form — unlike the training-only case, which has a clean closed-form solution. The optimization must be solved numerically.

Their analysis reveals a concrete pattern: as inference demand increases, optimal models become smaller relative to Chinchilla-optimal but are trained on proportionally more data. For a 70B-Chinchilla-quality model with 2T inference tokens, the compute-optimal model is 41.6B parameters trained on 7.92T tokens. This is exactly the LLaMA training strategy: Llama 3 uses ~1,875 tokens/parameter vs. Chinchilla’s ~20.

Test-Time Scaling is Difficulty-Dependent

The most important finding from Snell et al.: test-time compute scaling does not follow a single universal power law. Unlike training-time scaling, effectiveness is critically modulated by question difficulty:

S-Curves, Not Power Laws

The Epoch AI analysis reveals that inference-time compute via search follows S-curves (sigmoidal) rather than power laws — meaning it has a hard ceiling and saturates. A critical distinction: pass@k (unlimited trials, oracle selection) can span 6+ orders of magnitude, but n@k (limited trials, imperfect verification — the realistic case) saturates below 100% with a much smaller tradeoff span. The theoretical ceiling depends on whether perfect verification is available — and in practice, it rarely is.

When accounting for real-world hardware utilization (Model FLOP Utilization, or MFU), the picture worsens: generation MFU is typically ~1% vs. training MFU of ~50% — a 50× gap. FLOP-based scaling laws systematically underestimate the value of inference compute.

The Strategies: How to Spend Inference Compute

Figure 2 — The two axes of inference-time compute strategies: parallel (multiple independent paths) and sequential (extended single-chain reasoning).

Self-consistency (Wang et al., 2022) is the cheapest robust approach: sample N diverse CoT paths, majority-vote the final answer. The Bag of Tricks benchmark (2025) — 20,000+ A100 GPU hours across 1,000+ experiments — found it to be the top performer across 8 reasoning tasks under a fixed token budget, outperforming both Best-of-N and MCTS (Monte Carlo Tree Search). It requires no external reward model, making it the most deployment-friendly strategy.

Process Reward Models (PRMs) predict correctness at each intermediate reasoning step, enabling fine-grained search. Lightman et al. (2023) showed PRM-guided Best-of-N outperforms outcome reward models on MATH. However, PRMs can cause “performance inflation” — one PRM led to decreased performance on MATH as the budget increased, because the reward model failed to distinguish truly correct answers from superficially correct ones.

DeepSeek-R1 represents a fundamentally different strategy: the model learns to generate long CoT traces as an emergent behavior of RL. Using GRPO with outcome-based rewards, the base model develops reflection, backtracking, and longer chains of thought — without external search infrastructure. No controlled study has directly benchmarked this against external search strategies on the same base model; the two approaches are complementary.

A crucial practical finding from the Bag of Tricks benchmark: implementation details have outsized impact. Optimal temperature (~0.8) yields 2.3–4.8% improvement; self-evaluation by the LLM “frequently fails to assess solution quality effectively” and can perform worse than random selection. The gap between methods in benchmarks may be smaller than the gap introduced by implementation choices.

The Overthinking Phenomenon: When More Thinking Hurts

Figure 3 — The inverted U-shaped accuracy curve. Easy problems peak early (~2K tokens) and overthink sooner; hard problems benefit up to ~8K tokens but still eventually decline.

The Inverted U-Shaped Curve

Multiple independent studies confirm that accuracy follows an inverted U-shaped curve with CoT length. Zhou et al. (2026) evaluated DeepSeek-R1-32B and s1-32B across AIME, MATH-500, and GPQA Diamond at budgets from 500 to 16,000 tokens. Accuracy peaks at ~12K tokens, then declines. The drop from 12K to 16K was -0.9% (95% bootstrap CI: [-1.4%, -0.4%]) — genuine, not noise.

The overthinking threshold varies 7.5× across difficulty levels. Level 1 problems peak at ~1K tokens; Level 5 problems benefit up to ~7.5K. Uniform compute allocation is systematically suboptimal — overthinking is most severe precisely where it’s most wasteful.

Negative Flips: The Primary Mechanism

Zhou et al. introduced “flip event tracking”: a positive flip is incorrect→correct (beneficial thinking); a negative flip is correct→incorrect (overthinking). The crossover where negative flips exceed positive flips occurs at ~7K tokens for R1-32B. Of 80 sampled negative-flip cases, 67.5% involve genuine overthinking — the model explicitly reconsiders and rejects a correct answer:

“Wait, I should double-check by considering an alternative approach… Actually, I think I may have overcounted. Let me reconsider…”

The Variance/Entropy Explanation

Ghosal et al. (NeurIPS 2025) provide a theoretical explanation: as the model reasons longer, output distribution entropy increases. Initially, this broader distribution improves coverage — an illusion of improved reasoning. Beyond a critical point, the “dilution effect” dominates: probability mass spreads to regions far from the correct answer. Extending thinking tokens from 385 to 6,136 on GSM-8K produced a 12× increase in entropy. Accuracy rose from 82.2% to 87.3%, then crashed to 70.3% at 15,980 tokens — a 17 percentage point decline from peak.

Verbose vs. Harmful Overthinking

Caldarella et al. (2026) introduce a critical distinction:

• Verbose overthinking: reaches correct answer early, continues generating redundant reasoning — wasteful but harmless.
• Harmful overthinking: reaches correct answer, continues, then deviates to incorrect — a reliability failure.

This matters because common efficiency strategies (early stopping, shorter traces) reduce verbose overthinking but do not reduce harmful overthinking. Early stopping with K=5 reduced reasoning length from 17.2 to 8.9 utterances but increased the harmful overthinking rate from 10.1% to 18.4%.

Optimal stopping — stopping at the first correct prefix — outperforms the model’s default behavior by up to 21% on individual benchmarks. Averaged across models, the gap is ~10% — larger than the gain from reasoning-oriented post-training itself. Current LRMs are “limited not only by whether they can reason, but also by whether they can stop reasoning at the right time.”

Reasoning trajectories are non-monotonic: after the first correct prefix, the probability of remaining correct drops sharply, plateauing around 0.2 (only ~20% probability of staying correct) after ~100 intermediate steps. This violates the assumption that correctness is “absorbing” — once a model reaches the right answer, further reasoning can move it away from correctness.

Adaptive Budget Allocation: The Emerging Solution

Figure 4 — The three stages of adaptive budget allocation: difficulty detection, budget allocation, and termination/routing.

The “Reasoning on a Budget” survey (Alomrani et al., 2025) provides the field’s organizing framework: L1 (controllable) methods operating under fixed user-specified budgets, and L2 (adaptive) methods dynamically scaling inference based on difficulty or confidence.

Difficulty-Aware Compute Allocation

Solve-then-Learn (Zhai et al., 2026) formalizes adaptive allocation as constrained optimization: maximize accuracy subject to a compute budget, using Lagrangian relaxation. A lightweight XGBoost classifier predicts oracle budgets in real-time. It achieves up to 12.8% relative accuracy improvement over uniform allocation, tracking the oracle with 91–99% imitation accuracy. The oracle exhibits an inverted-U allocation: minimal budget for both easy and intractable questions, high budgets for medium-difficulty “responsive” questions.

DiffAdapt (Liu et al., 2026) discovers a U-shaped entropy pattern: high entropy on easy problems (overthinking), low entropy on medium problems, high entropy on hard problems. An MLP (Multi-Layer Perceptron) probe classifies inputs into Easy/Normal/Hard tiers. It achieves 22.4% token reduction with comparable accuracy across five models and eight benchmarks, with no LLM retraining required.

Learn-to-Stop Mechanisms

LearnStop (Dong et al., 2026) provides a three-regime decision map:

1. Learned stopping wins on free-form math: 33.8–51.4% token savings
2. Calibrated scalar exits win on multiple-choice: where 44–53% of instances are already solved at the first checkpoint
3. No aggressive policy works on small hard sets: where 68% of trajectories never become correct

A crucial finding: the serving regime determines whether savings are real. The same policy saving 32% of tokens under KV-cache (Key-Value cache) forking costs 121% extra under black-box repeated prefilling — the deployment infrastructure matters as much as the stopping policy.

RL-Based Adaptive Allocation

CODA (Wu et al., 2026) uses group success rate from rollouts as a zero-cost difficulty signal. Dual-gated reward shaping penalizes verbosity on easy tasks and rewards deliberation on hard ones. It reduces average token cost by 15–21%, with 60–87% reductions on easy tasks while preserving compute on hard tasks.

Compute Routing

Damani et al. (ICLR 2025) formalize routing between cheap “System 1” and expensive “System 2” paths. Their method matches the strong decoder’s performance while calling it only 50–75% of the time, and sometimes outperforms it. Commercial systems have adopted this: OpenAI’s o-series offers “reasoning effort” (low/medium/high), and Claude supports a “thinking token budget.”

The Economics: Who Profits, Who Pays

The DeepSeek-R1 Cost Story

DeepSeek-R1 matched o1’s benchmark performance (79.8% vs 79.2% on AIME 2024) while pricing output tokens at $2.19/MTok vs. o1’s $60/MTok — a 27.4× gap. Epoch AI concludes this is “probably mostly due to OpenAI’s large markups.” The famous “$5.6 million training cost” refers only to the final pretraining run of DeepSeek-V3; IBM’s analysis notes it “likely is a snapshot of a single, pristine training run” that excludes all experimentation, personnel, and infrastructure costs.

The Inference Compute Shift

Deloitte’s 2026 TMT Predictions — the primary source for the “two-thirds” claim — states: “Inference will make up two-thirds of AI compute by 2026.” Inference went from ~1/3 of AI compute in 2023, to half by 2025, to a projected 2/3 in 2026. Over a model’s lifecycle, inference represents 80–90% of total compute costs.

Per-token costs have collapsed ~280× in two years (from $20/MTok to ~$0.07/MTok for GPT-3.5-equivalent performance), yet enterprise inference bills are climbing because volume growth dwarfs price declines. Three structural drivers: agentic loops require 10–20 LLM calls per task, RAG (Retrieval-Augmented Generation) pipelines send thousands of pages of context, and always-on monitoring agents consume compute 24/7.

Value Capture

Anthropic’s gross margins on inference increased from 38% to over 70% even as they cut prices 3×. Margin expansion came from hardware (Blackwell GPUs: ~30× more tokens/second) and software optimizations (14× throughput on same hardware). Meanwhile, Nvidia and TSMC have not fully repriced to capture the surge in token value — a gap SemiAnalysis calls “venting vast value into every vertical of the ecosystem.”

Production Systems: Deploying Reasoning at Scale

The Thinking-Token Explosion

Reasoning models generate 5–50× more output tokens. A model emitting 8–32K thinking tokens holds a GPU decode slot for 10–120 seconds — a slot that would have served 50–100 chat requests in the same window. Decode now dominates 90–98% of cost, and a 32K-token trace on a 70B model consumes ~84GB of KV cache per request.

Routing Before Reasoning

The dominant production pattern: a fast classifier (BERT-class, <12ms) routes easy queries to non-reasoning models, reserving reasoning for the 10–30% that benefit. This cuts 50–70% of reasoning cost with negligible quality loss. Systems should default to the premium tier when classifier confidence falls below 0.75, because routing errors concentrate in the most consequential query types.

Budget Semantics Are Non-Portable

OpenAI’s reasoning_effort (low/medium/high) is not a hard cap. Anthropic’s budget_tokens is closer to a hard cap. DeepSeek-R1 via official API exposes no budget — the model decides. Assuming portability across vendors leads to 5–10× cost surprises. The right default for most products is medium effort.

Reliability and Load Shedding

Reasoning models exhibit doom loops (repeating “Wait, let me reconsider…” until the context window exhausts), premature commitment, and trace-answer disagreement. Standard chat load-shedding (reject when full) is wrong for reasoning — by the time overload is detected, requests are 30+ seconds into multi-minute traces. The 2026 best practice: shed by lowering thinking budgets, not by rejecting requests. Databricks’ platform saved >80% in GPU costs with cost-aware load balancing.

Agent Systems Use Reasoning Selectively

The 2026 production pattern routes 5–20% of agent turns to reasoning models. Using reasoning for every turn creates 30–100× cost inflation. Leading code-agent products split 20–40% reasoning / 60–80% non-reasoning, with routing logic continuously tuned against user-completion metrics.

Where the Sources Disagree

Devil’s Advocate: The Case That This Report Is Overstated

• The overthinking problem may be smaller in practice than in benchmarks. Most research uses math/science benchmarks with binary correctness. In open-ended production use (creative writing, code generation, analysis), the inverted-U curve may be less pronounced or absent.

• The adaptive allocation frameworks are mostly academic. None have been deployed at production scale. Solve-then-Learn requires expensive offline data collection. DiffAdapt’s 6× latency reduction was measured under specific conditions.

• The economics may be driven by temporary market dynamics. R1’s 27× price advantage may reflect below-cost pricing. The inference-dominance trend could reverse if new training methods make training relatively more important.

• The historical parallel to multi-core may be misleading. CPU clock speed hit a physical wall (thermodynamics); LLM parameter scaling faces economic limits, not physical ones. The shift could be temporary if new architectures reduce training costs.

• Several key claims rest on thin sourcing. The Zylos survey is AI-generated, not peer-reviewed. Several ACL 2026 papers were referenced from abstracts, not full text. The “4× efficiency” result from Snell et al. was on a single base model and benchmark.

• Publication bias may inflate the phenomenon. ACL 2026 alone features 11+ papers on overthinking. This concentration reflects a fashionable topic, not necessarily its practical importance relative to other challenges.