The Architecture of Intelligence: LLMs, 2024 → 2026

Between mid-2024 and mid-2026 the frontier large language model stopped being a single dense transformer that got bigger every year. It became a sparse, reasoning-trained, inference-optimized object wrapped in tooling: a Mixture-of-Experts core with latent or sparse attention, trained partly by verifiable-reward reinforcement learning, served through speculative decoding and aggressive caching, and judged less by its benchmark score than by what it can do inside an agent loop.

This is a field report across five lenses — architecture, reasoning, training, production, and multimodality/agents — researched with a STORM-style multi-perspective methodology, every claim tied to a source in the references at the end.

The two-year rebuild, in one paragraph

DeepSeek-V3 [1] proved a frontier-class MoE could train for about $5.6M; R1 [10] proved reasoning could emerge from reinforcement learning alone; Llama 4 [5] and Gemini 3 [33] made multimodality native and context windows absurd; and GLM-5.2 [2] compressed all of it into an open-weights model built for long-horizon agents. The economics inverted with it: a roughly 600-fold fall in token price since 2020 [14], an open-versus-closed output-price spread of around 640× [15], and a growing consensus that the harness, not the model, is now the leverage [27].

The headline of the period is not “models got smarter.” It is compute moved: from one giant pretraining run, into sparse parameters, into test-time thinking, into the agent harness around the model — and the price collapsed as each layer got engineered.

01 · Architecture: make it sparse, then make the attention forget on purpose

The dense transformer did not die, but it stopped being the shape of the frontier. Four mechanisms define the 2024–2026 architecture: MoE (most parameters dormant per token), latent/sparse attention (the KV cache stops scaling with context), SSM–attention hybrids (a small dose of attention recovers what state-space models lose), and multi-token prediction baked into training so inference can speculate.

• Mixture-of-Experts is now the default frontier shape (high, [1]) — DeepSeek-V3 runs 671B total / 37B active with 256 routed + 1 shared expert, 8 active per token, and an auxiliary-loss-free load-balancing scheme that uses a routing bias instead of a balancing loss [1]. Llama 4 Maverick is 400B / 17B active with 128 experts [5]; GLM-5.2 is ~753B / 40B active [2]. Total capacity balloons while per-token compute stays in the tens of billions.
• Multi-head Latent Attention (MLA) compresses the KV cache (high, [1]) — DeepSeek-V3 joint-compresses keys and values into a 512-dim latent (queries into 1536-dim) with decoupled RoPE keys, so generation caches a small latent vector per token instead of full K/V heads — the single biggest reason long-context stopped being ruinous [1].
• GLM-5.2 IndexShare: reuse one indexer across four sparse-attention layers (high, [2]) — Building on DeepSeek Sparse Attention, IndexShare places the top-k indexer once and reuses its indices for the next three layers, cutting per-token FLOPs ~2.9× at 1M context by deleting redundant indexer dot-products and topk in three of four layers [2][4]. KVShare plus rejection-sampled MTP lifts speculative-decoding acceptance length about 20% (4.56→5.47 tokens) [2].
• Llama 4 iRoPE: interleave NoPE layers among RoPE chunked-attention (high, [5]) — Every fourth layer is a no-positional-embedding full-attention layer over the entire context; the other three use RoPE with 8K chunked attention plus inference-time temperature scaling to stop softmax fading. The “i” nods to “infinite” context — it is what lets Scout claim a 10M-token window [5][6].
• SSM–Transformer hybrids work — with about 8% attention (high, [19]) — Jamba interleaves one attention layer per seven Mamba layers plus MoE [17]; NVIDIA’s controlled 8B study found a Mamba-2 hybrid (7% attention) beat a pure transformer on all twelve tasks at up to 8× faster inference [19]. Pure SSMs still lose on in-context learning and copying — even one attention layer per eight restores induction heads.
• Multi-Token Prediction moved from research trick to trained-in feature (high, [24]) — DeepSeek-V3 trains a depth-1 MTP head (loss weight 0.3→0.1) that doubles as a speculative decoder at inference (~85–90% acceptance, 1.8× tokens/sec) [1]. The standalone line — Medusa (extra heads, tree attention) and EAGLE (feature-level, lossless 2.7–3.5×) — converged on the same idea: predict several tokens, verify in one forward pass [24][25].

Figure 1 — The modern MoE stack: latent/sparse attention and sparse MoE FFN blocks, an MTP head as built-in speculative decoder, and the agent harness that increasingly carries the gains.

02 · Reasoning: reinvented as an RL problem

OpenAI’s o1 [20] made “think longer at inference” a product. DeepSeek-R1 [10] then showed the thinking could be trained with verifiable rewards and no neural reward model, and open-sourced the recipe. By 2026 the field had also learned the limits: longer chains are not always better, and “thinking harder” is not “thinking longer.”

• RLVR: rewards you can verify, not rewards you have to learn (high, [10]) — R1-Zero applies GRPO directly to the base model with rule-based rewards — answer correctness plus format tags — and no SFT, hitting 71% pass@1 on AIME (from 15.6%) and spontaneously developing self-verification, reflection, and the famous “aha moment” [10]. Verifiable binary rewards avoid the reward-hacking failure mode of learned reward models.
• GRPO drops the critic (high, [12]) — Group Relative Policy Optimization estimates the baseline from a group of sampled rollouts (whitened advantages) instead of a value model, cutting the RL machinery roughly in half. 2025 theory showed GRPO provably amplifies success probability above the reference policy, and that mirror-GRPO (KL to the previous iterate) guarantees monotone improvement [12].
• RLVR genuinely extends reasoning — not just sampling efficiency (medium, [13]) — A 2025 debate asked whether RLVR merely reroutes Pass@K. The rebuttal introduced CoT-Pass@K (credit only if both answer and reasoning are correct) and showed RLVR raises the reasoning boundary for math and code, with gains visible from early training and generalizing to unseen questions [13].
• o3-mini thinks harder, not longer (high, [21]) — On Omni-MATH, o3-mini beats o1-mini without longer chains; accuracy actually declines per 1,000 extra reasoning tokens (3.16% o1-mini → 0.81% o3-mini-h). The high-effort setting buys ~4% accuracy for more than 2× the tokens — sharply diminishing returns [21].
• Parallel sampling beats sequential self-revision (medium, [23]) — Across o1-like models, longer chain-of-thoughts add “Wait / Alternatively” self-revisions whose successful-revision rate is under 10% — they more often flip correct to incorrect. Parallel sampling plus aggregation (the proposed “Shortest Majority Vote,” score = count / log length) outperforms sequential scaling [23].
• Test-time compute buys robustness too (medium, [20]) — OpenAI found attacker success on o1-preview/mini often decays toward zero as inference-time compute grows, and the robustness is not from adversarial training, suggesting it generalizes to unforeseen attacks [20]. A real caveat: cost nondeterminism — reasoning models burn wildly varying tokens for the same correct answer [22].

03 · Training: train cheaply, distill downward, and let RL finish the job

Pretraining data has crossed 30T tokens as a floor, FP8 mixed precision is standard at frontier scale, and the post-training pipeline has become a multi-stage RL-plus-distillation assembly line. The most counterintuitive result of the period: for small models, distilling from a big reasoner beats doing RL on the small model directly.

• DeepSeek-V3 trained a frontier MoE for about $5.6M (high, [1]) — 2.788M H800-hours, FP8 mixed precision (first validated at this scale), DualPipe for compute-communication overlap, no irrecoverable loss spikes. The number re-set expectations for frontier training cost — and is the architectural root of the US–China training-cost gap (MoE, not cheaper compute) [1][14].
• Qwen3: 36T tokens, 119 languages, thinking fused into one model (high, [8]) — A unified thinking/non-thinking framework with a thinking budget and /think /no_think flags — one model replaces the old “fast model versus reasoning model” split. Four-stage post-training: cold-start CoT SFT → reasoning RL (GRPO) → thinking-mode fusion → general RL. Apache-2.0, dense 0.6B–32B plus MoE 30B-A3B / 235B-A22B [8][9].
• Distillation beats RL for small models; RL beats distillation for the frontier (high, [10]) — R1’s six distilled models (1.5B–70B on Qwen2.5/Llama3, 800K samples) beat o1-mini at 32B. DeepSeek’s own conclusion: distillation from a big model beats RL on small models, but pushing the capability boundary still needs a strong base plus large-scale RL [10].
• GLM-5.2 agentic RL: critic-based PPO plus an anti-hack module (medium, [2]) — Post-training used the “slime” framework for parallel Online Policy Distillation merging 10+ expert models in about two days, moving from group-wise optimization to critic-based PPO with token-level advantages — plus an anti-hack module that blocks coding agents from curl-ing solutions off GitHub [2].
• The PRM and MCTS bets largely lost (medium, [10]) — Process Reward Models hit reward hacking and annotation bottlenecks; Monte Carlo Tree Search blew up the search space and strained value models. Verifiable outcome rewards plus GRPO became the pragmatic default [10].
• Co-distillation and MetaP: train the teacher and student together (high, [5]) — Llama 4 Maverick was co-distilled from Behemoth via a loss that dynamically weights soft and hard targets through training [5]; MetaP transfers per-layer learning rates and init scales across widths, depths, and token counts. Post-training order itself was reordered (SFT→RL→DPO) because SFT and DPO were over-constraining exploration [5].

04 · Production: the price collapsed; the routing layer became the asset

Inference economics are the headline story of the period. A March-2026 dataset of 318 models documents a ~600× decline in token price since GPT-3, with a structural break in May 2024 — the DeepSeek price war — after which competition, not technology, drove the curve [14]. By June 2026 there is a ~640× spread between the cheapest and priciest output token [15].

• Software, not silicon, is doing the deflating (medium, [14]) — The pricing study attributes ~103.7% of cost reduction to software and architectural innovation (MoE, Flash Attention, speculative decoding) and −0.9% to GPU hardware. Economy-tier prices have a 1.10-year half-life — 82% faster than Moore’s law [14].
• The reasoning premium is the last holdout (medium, [14]) — Flagship and reasoning prices defy exponential decay (R²=0.031) at a ~31.5× premium over non-reasoning — until a competitor enters and compresses it fast (DeepSeek-R1 at $0.55/M versus o1 at $15/M). Reasoning rents are real but short-lived [14].
• Open weights are now “price insurance” (medium, [16]) — Hosted open-weight models are 5–30× cheaper per token than proprietary APIs in mid-2026, and the best open models (Qwen 3.7 Max, GLM 5.1) match proprietary mid-tiers but still miss the true frontier (Opus 4.8, Gemini 3.1 Pro, GPT-5.5) [16]. Self-hosting only wins above ~10M tokens/day; otherwise the exit right — not the serving — is the value.
• Caching and batch are the real cost levers (medium, [15]) — Every major provider offers 50–90% cache-read discounts and 50% batch discounts; DeepSeek’s cache-hit multiplier (~0.02×) is the most aggressive. Anthropic dropped its 200K+ long-context premium entirely; Gemini and OpenAI still tier — and Anthropic’s new tokenizer emits ~30–35% more tokens for the same text, so raw dollars-per-million-tokens misleads [15][16].
• Speculative decoding went to production scale (high, [26]) — EAGLE (feature-level, lossless 2.7–3.5×) and Medusa (extra heads, tree attention) established the no-separate-draft-model paradigm [24][25]; Meta shipped EAGLE-style spec-decode for Llama at scale (1.4–2.0× even at large batch) in August 2025 [26]. GLM-5.2’s trained-in MTP makes the model its own draft [2].
• Market structure: from one ruler to several (medium, [14]) — OpenAI share fell 65%→34%; DeepSeek rose 0%→13% in 18 months; the Herfindahl-Hirschman index dropped 4,558→2,086 (highly to moderately concentrated). The routing layer — pick the cheap model for 80–85% of calls, the frontier for 10–15% — is now treated as the core architectural asset [14][16].

05 · Multimodality and agents: native multimodality, and the harness that ate the model

Two converging moves: models became natively multimodal (text, image, audio, video fused from pretraining, not bolted on), and the system around the model became the thing that’s actually optimized. Agentic Harness Engineering studies show tools, middleware, and memory carry the gains — system-prompt-only edits actively regress.

• Early-fusion multimodality is now standard (high, [5]) — Llama 4 fuses text and image tokens into one backbone from pretraining (MetaCLIP vision encoder trained jointly with a frozen Llama) [5]; Gemini 3 Pro processes text, images, video, audio, and PDF through one native architecture and does cross-modal reasoning without translating between modalities [33].
• Context windows hit 1M–10M tokens (medium, [33]) — Gemini 3 Pro: 1M in / 64K out at launch (November 2025); Gemini 3.1 Pro extends toward 10M via hierarchical cascaded retrieval attention (O(n²) to ~O(n log n)) and dynamic sparse KV-cache eviction [33][34]; Llama 4 Scout claims 10M via iRoPE [5]; GLM-5.2 ships a “solid” 1,048,576 tuned for coding agents [2].
• The harness, not the prompt, carries agentic gains (medium, [27]) — Agentic Harness Engineering auto-evolves a coding harness via observability; on Terminal-Bench 2 it reaches 77.0% versus 71.9% for a human-designed Codex-CLI harness, and transfers cross-benchmark (75.6% SWE-bench-verified at 12% fewer tokens) and cross-model (+5–10 points) [27]. Ablation: the gain lives in tools, middleware, and memory — system-prompt-only edits regress 2.3 points.
• A 0.8B model can learn the agent interface (medium, [28]) — HarnessBridge trains a tiny Qwen3.5-0.8B as a bidirectional controller between LLM and environment (Pass/Compress/Drop per turn; action projection with trajectory-grounded rejections). It lifts Terminal-Bench 2.0 success 30.3%→33.7% and cuts tokens 42–77%, generalizing from SWE-bench to Terminal-Bench and from small to commercial generators (GPT-5.4: 53.9% success at ~89% fewer tokens) [28].
• Process rewards plus semantic tools stabilize long-horizon agents (medium, [29]) — SWE-TRACE uses a rubric process reward model plus GRPO and step-level test-time pruning (63.5%→71.2% on SWE-bench-verified with a 30B MoE) [29]; SWE-Master gives agents IDE-grade LSP navigation (go-to-definition, find-references), cutting tokens 20–40% at equal resolve rate [30]. Both block git-hacking to protect eval integrity.
• Frontier agents: codebase-scale, computer-use, and export control (high, [31]) — Claude Opus 4.8 (May 2026) shipped “dynamic workflows” running hundreds of parallel subagents for 100k-line migrations and is the best browser/computer-use model (84% Online-Mind2Web) [31]. Fable 5 / Mythos 5 (June 9, 2026) did a 50M-line Stripe Ruby migration in a day — then were suspended by US export control on June 12 [32], an early signal that frontier agent capability is now a geopolitically constrained good.

06 · Twenty-four months on a single line

07 · The spec sheet and the price sheet

Frontier architecture, mid-2026

Frontier API pricing, June 2026 (USD per 1M tokens)

The spread between the cheapest output (DeepSeek V4 Flash, $0.28) and the priciest (GPT-5.5-pro, $180) is about 643×. Pricing figures are aggregator-verified (June 11, 2026) and should be re-checked against provider pages before any procurement decision.

08 · Where the sources disagree

Does RLVR create reasoning? — Yue 2025 vs Wen 2025. One camp argues RLVR only improves sampling efficiency (base-model Pass@K catches up at large K); the other introduces CoT-Pass@K and shows the reasoning boundary genuinely extends. Reconciled read: RLVR extends reasoning when intermediate steps are verified; pure answer-correctness Pass@K can mask it [13].

Longer thinking equals better? — o1 marketing vs 2025 studies. OpenAI frames more thinking tokens as strictly better; three independent studies find accuracy declines with excessive reasoning tokens and that parallel sampling beats sequential self-revision [21][22][23]. Reconciled read: thinking helps up to a point and per-generation; scaling it sequentially past that hurts.

GLM-5.2 parameter count — blog vs community note. Z.ai’s blog and HuggingFace list ~753B total / 40B active; an earlier community digest cited 744B. Reconciled read: treat the exact count as unconfirmed until a peer-reviewed tech report; the 40B-active and MIT-license facts are consistent across sources [2][3].

Open versus closed frontier gap — aggregators disagree on size. One analysis says open weights “match proprietary mid-tiers but don’t reach the true frontier”; GLM-5.2’s own benchmarks trail only Opus 4.8 on several coding-agentic tasks [2][16]. Reconciled read: the gap is now task-dependent — effectively closed or reversed on coding-agentic work, still present on the broadest knowledge and reasoning suites.

09 · Devil’s advocate: the case that this report is overstated

Before you take any of this to a roadmap.

• Source provenance is uneven. The architectural claims rest on primary tech reports (DeepSeek-V3, Qwen3, R1, Jamba, EAGLE/Medusa). But several 2026 specifics — Gemini 3.1’s hierarchical attention, exact June-2026 pricing, GLM-5.2 internal training details, the AHE/HarnessBridge numbers — come from aggregators and blog posts, not peer-reviewed papers. A few arXiv IDs in the 2603–2606 range are dated only by web search and could not be independently fetched here.
• Benchmarks are games, especially agentic ones. SWE-bench, Terminal-Bench, and FrontierSWE have known contamination and overfitting risks; vendors route specifically for them. GLM-5.2’s “anti-hack module blocking curl” and SWE-Master’s “git-hacking prevention” are evidence the evals are being gamed — which means leaderboard deltas overstate real-world deltas.
• The “harness greater than model” thesis is a vendor-interest story too. AHE and HarnessBridge come from groups building harness tooling; the ablation that “system-prompt-only edits regress” conveniently elevates the sellable layer. It is suggestive, not settled.
• Deflation math can mislead. The 600× figure compares GPT-3 (2020) to a 2026 flash model at a different capability tier; same-capability-bar curves show ~10–50× per year, still steep but not 600×. The 643× price spread mixes reasoning and non-reasoning tiers — apples to oranges.
• SSM hybrids did not conquer the frontier. Jamba and the NVIDIA study prove hybrids can win at 8B, but every 2026 frontier model here is still transformer-based MoE. The hybrid story is a real efficiency result, not a paradigm shift at the top.
• Geopolitics is now a first-order variable. Fable 5 / Mythos 5 were suspended by US export control three days after launch [32]. Any production roadmap assuming stable access to a given frontier model is making a political bet, not just a technical one.

Colophon

References

1. DeepSeek-V3 Technical Report (2024) — arxiv.org/2412.19437
2. GLM-5.2: Built for Long-Horizon Tasks (2026) — z.ai/blog/glm-5.2
3. zai-org/GLM-5.2 model card — huggingface.co/zai-org/GLM-5.2
4. Raschka, GLM-5.2 IndexShare note (2026) — sebastianraschka.com
5. Meta, The Llama 4 herd (2025) — ai.meta.com/blog/llama-4
6. HuggingFace, Welcome Llama 4 (2025) — huggingface.co/blog/llama4-release
7. Llama 4 model card — github.com/meta-llama
8. Qwen3 Technical Report (2025) — arxiv.org/2505.09388
9. Qwen3 blog — qwenlm.github.io/blog/qwen3
10. DeepSeek-R1 (2025) — arxiv.org/2501.12948
11. DeepSeek-R1, Nature (2025) — nature.com/s41586-025-09422-z
12. Mroueh, GRPO dynamics (2025) — arxiv.org/2503.06639
13. Wen et al., RLVR implicit reasoning (2025) — arxiv.org/2506.14245
14. Tiered Super-Moore’s Law token pricing (2026) — arxiv.org/2603.28576
15. Developers Digest, Frontier pricing June 2026 — developersdigest.tech
16. TokenRate, Open vs Proprietary costs 2026 — tokenrate.dev
17. Jamba: Hybrid Transformer-Mamba (2024) — arxiv.org/2403.19887
18. AI21, Introducing Jamba — ai21.com/blog/jamba
19. NVIDIA, Empirical Study of Mamba LMs (2024) — arxiv.org/2406.07887
20. OpenAI, Learning to Reason with LLMs (o1) (2024) — openai.com
21. Ballon et al., o3-mini Thinks Harder (2025) — arxiv.org/2502.15631
22. MSR, Inference-Time Scaling (2025) — microsoft.com/research
23. Zeng et al., Revisiting Test-Time Scaling (2025) — aclanthology.org
24. EAGLE (2024) — arxiv.org/2401.15077
25. Medusa (2024) — arxiv.org/2401.10774
26. Meta, Speculative Decoding for Llama at Scale (2025) — ai.meta.com/research
27. Agentic Harness Engineering (2026) — arxiv.org/2604.25850
28. HarnessBridge (2026) — arxiv.org/2606.12882
29. SWE-TRACE (2026) — arxiv.org/2604.14820
30. SWE-Master (2026) — arxiv.org/2602.03411
31. Anthropic, Introducing Claude Opus 4.8 (2026) — anthropic.com
32. Anthropic, Claude Fable 5 & Mythos 5 (2026) — anthropic.com
33. Gemini 3 Pro launch guide — llm-stats.com
34. Gemini 3.1 Pro deep dive (2026) — explore.n1n.ai
35. Gemini Embedding 2 (2026) — arxiv.org/2605.27295