Distillation: The Attack Vector Frontier Models Can't Hide From

In early 2026, two of the three U.S. frontier providers went public with something the industry had suspected but rarely named: foreign labs were extracting their models’ capabilities at industrial scale. Anthropic reported that DeepSeek, Moonshot, and MiniMax generated over 16 million exchanges with Claude through roughly 24,000 fraudulent accounts [1]. OpenAI, via an internal memo surfaced by Reuters, stated that DeepSeek had trained on U.S. models by distillation and bypassed its restrictions [2].

The mechanism behind these campaigns is not a novel exploit. It is distillation — the same legitimate technique labs use to ship smaller, cheaper versions of their own models. The dual-use nature is the whole problem: the mechanism is identical whether you are building GPT-4o-mini or cloning your competitor’s frontier model without consent. Only authorization, intent, and downstream use separate product from attack.

This post is a defensive threat-intelligence analysis of model distillation as an attack vector against frontier large language models. It maps the vector to shared taxonomies, scores variants for risk, and ends with concrete, prioritized defender guidance. Throughout, I keep the line between documented fact, demonstrated research, and informed speculation visible — because the difference matters when you are deciding where to spend a defensive dollar.

The economics are the threat

Before any taxonomy, the single number that frames everything: the cost asymmetry is staggering.

Foundational research by Carlini and colleagues demonstrated that nontrivial model internals can be pulled from production APIs for under $20. They extracted the entire embedding-projection matrix of OpenAI’s ada and babbage models at that price, recovering the exact hidden dimensions (1024 and 2048) for the first time. Recovering the full projection matrix of gpt-3.5-turbo was estimated at roughly $2,000 [3]. On the harder, no-logprob side of the same work, a well-tuned student model can end up about 130× faster and 25× cheaper to run than the frontier teacher it replicates [4].

Frontier training runs, by contrast, are in the hundreds of millions to billions of dollars [5]. Distillation transfers comparable task-specific competence for orders of magnitude less.

The strategic point is not that an attacker can bit-for-bit steal your weights. They cannot — no black-box query extraction replicates a model 100% [6]. The point is that partial fidelity is sufficient for capability transfer, and the price of that partial copy is vanishingly small next to the price of building the original.

This asymmetry, not the prospect of weight theft, is the core of the threat. And it widens every year as student-training efficiency improves.

What we are (and are not) talking about

Precision matters here, because distillation sits next to a cluster of related vectors that share query access but have different objectives.

Adversarial model distillation / extraction — the subject of this analysis — is an adversary using query access to a hosted frontier “teacher” to generate input–output pairs, recover token probabilities or logits, or elicit reasoning traces, then training a cheaper “student” that replicates the teacher’s capabilities, knowledge, or behavior, without authorization.

Benign knowledge distillation is the legitimate, consented counterpart. Frontier labs distill their own models into smaller commercial variants; OpenAI sells distillation tooling to enterprise customers; open-weight developers like the Allen Institute for AI use it building models such as OLMo 3 [5]. The mechanism is identical. Only authorization, intent, and downstream use distinguish attack from product.

Then there are the adjacent vectors, which share the query surface but should not be conflated with distillation:

• Model inversion (AML.T0024.001) reconstructs representative inputs or training data from outputs, rather than replicating the function.
• Membership inference asks whether a specific record was in the training set.
• Training-data extraction recovers verbatim training text.
• Prompt / system-prompt extraction recovers the hidden instructions shaping behavior.
• Weight exfiltration via infrastructure compromise — direct theft of weights through insider access, misconfiguration, or supply-chain attack — is a different threat class. It requires white-box or infrastructure access, not the black-box query surface studied here.

In scope for this analysis: black-box, query-based attacks against hosted frontier models via API and chat interfaces. Out of scope: anything requiring white-box weight access, and pure insider exfiltration.

The attack taxonomy

Distillation-as-extraction is a family, not one attack. The variants differ along three useful axes.

Signal fidelity separates hard-label attacks — harvesting only the final text outputs, the “self-instruct” path — from soft-label / logit-based attacks that recover token probabilities or full logit vectors and carry far more information per query.

Target separates capability distillation (maximize accuracy on a domain), behavior distillation (maximize fidelity — replicate style, refusal patterns, reasoning style), and vertical distillation (concentrate on a narrow high-value domain like coding or agentic tool use).

Pipeline stage is the most strategically illuminating axis. Distillation can feed any phase of the adversary’s training: synthetic-data generation for pre-training or SFT, chain-of-thought extraction for SFT, and — critically — using the frontier model as a reward model to grade a student’s RLVR rollouts [5].

The diagram below shows how four extraction methods feed an adversary’s full training pipeline, and why the cycle compounds.

Figure 1 — The adversarial distillation cycle. Four methods extract capabilities from a frontier teacher across the training pipeline. Synthetic data and CoT feed pre-training and SFT; the teacher-as-reward-model shapes RLVR. Capability gains compound back into pre-training, letting the adversary climb the ladder in larger steps than independent development allows [5].

A few variants deserve special attention because they are where the strategic damage concentrates:

Safety/alignment stripping distills a model’s competence while discarding its guardrails, often via a jailbreak-then-harvest pipeline. Anthropic’s framing is blunt: illicitly distilled models “are unlikely to retain safeguards,” so dangerous capabilities proliferate “with many protections stripped out” [1]. And this can happen unwittingly — research on response-based distillation for multilingual jailbreak prevention found it can “compromise safety” even when that was never the intent [7].

Reasoning-trace (CoT) harvesting is the highest-leverage behavioral variant. A model that exposes full reasoning traces gives the attacker the learning signal directly — not just the answer, but the policy that produced it. Many providers now hide chains of thought for exactly this reason. Anthropic observed attackers working around it by prompting Claude to “imagine and articulate the internal reasoning behind a completed response,” effectively synthesizing CoT training data at scale [1].

The comparison table captures the full set:

Who is doing this, and why

The threat actors fall into a small number of profiles with very different resource profiles.

The nation-state-linked profile is the one driving the documented campaigns, and it is worth understanding why it is economically rational for them. Export controls on advanced semiconductors have put Chinese developers at a large and growing compute deficit [5]. Distillation is a way to convert query budget — which they can buy or steal — into capability, which they cannot easily compute their way to. The 16 million exchanges Anthropic reported are estimated at 150–400 billion tokens by Nathan Lambert’s methodology, exceeding DeepSeek-R1’s entire SFT dataset (~6.4 billion tokens per Epoch AI) “by orders of magnitude” [5].

There is a subtlety defenders and policymakers should not miss: distillation is not free for the adversary either. Absorbing distilled outputs into a stronger model still requires substantial compute — which is precisely what export controls constrain. This is why the vector reinforces rather than nullifies the rationale for chip controls. It does not let an attacker skip the compute step; it lets them get more capability per compute dollar by standing on a frontier teacher’s shoulders.

What the evidence actually shows

A caveat first: the cases below are provider allegations and attributions made with stated confidence, not independently adjudicated fact. Where attribution is contested or unproven, I flag it.

Anthropic published attribution “with high confidence” via IP correlation, request metadata, infrastructure indicators, and industry-partner corroboration [1]. The specifics:

• DeepSeek — over 150,000 exchanges, targeting reasoning, reward-model grading, and “censorship-safe alternatives” to politically sensitive queries (likely to train their own models to steer away from censored topics).
• Moonshot — over 3.4 million exchanges, targeting agentic reasoning, tool use, coding, and computer-use agents, later escalating to reasoning-trace extraction.
• MiniMax — over 13 million exchanges, targeting agentic coding and tool orchestration. Anthropic detected this campaign before MiniMax released the model it was training, and watched MiniMax pivot within 24 hours when a new Claude release landed.

One proxy network alone ran more than 20,000 fraudulent accounts simultaneously [1].

OpenAI, per Reuters/Bloomberg reporting on an internal memo, stated that DeepSeek “trained its AI by distilling US models” and that employees “bypassed OpenAI’s restrictions” [2]. This is a provider allegation reported by reputable outlets — attribution is OpenAI’s claim, not independently verified here.

Google’s Threat Intelligence Group documented over 100,000 prompts in a single chain-of-thought extraction campaign without naming specific labs [5].

What is demonstrated in research is more settled: Carlini et al.’s production-API extraction (with responsible disclosure, after which OpenAI and Google added mitigations) [3]; Tramèr et al.’s foundational 2016 prediction-API theft work [9]; and the Alpaca self-instruct replication at ~$500 [8].

What remains plausible but speculative: that named labs’ public releases are materially dependent on distilled data (CNAS notes this “cannot be cleanly attributed” — internal resource allocation produces identical signatures [5]); that the three named campaigns were coordinated rather than merely parallel (Anthropic’s “similar playbook” language is explicitly hedged); and that transfer-station traffic over China Telecom/Unicom networks implies state direction (circumstantial, not a proven command relationship).

Where the taxonomies fall short

MITRE ATLAS is the primary frame here, and it captures the mechanics reasonably well:

• AML.T0024 — Exfiltration via AI Inference API is the parent technique.
• `AML.T0024.002 — Extract AI Model is the closest fit: “repeatedly querying the victim’s inference API, the adversary can collect the target model’s inferences into a dataset.”
• AML.CS0056 is a notable milestone — the Anthropic/DeepSeek/Moonshot/MiniMax campaigns are now a formal ATLAS case study, the first distillation campaign entered into the framework [10].
• Relevant mitigations include AML.M0002 (Passive AI Output Obfuscation), AML.M0004 (Restrict Number of AI Model Queries), and AML.M0024 (AI Telemetry Logging).

On the OWASP side, LLM10: Model Theft explicitly enumerates both query-based extraction and the self-instruct “functional replication” path, and the ML Security Top 10 carries it as ML05:2023 Model Theft [6].

But three behaviors are real and observed yet lack a clean, named entry in either framework — a gap that mirrors how agentic orchestration outran classic ATT&CK:

1. Reasoning-trace / CoT harvesting is subsumed under generic extraction, but its objective (stealing the reasoning policy, not the answer) and its trigger (CoT-elicitation prompts that reconstruct hidden reasoning) are not named. AML.T0024.002 is a structural fit but loses the semantic distinction.
2. Reward-model abuse — using a frontier model as a grader for an independent student’s RLVR — is a distinct functional extraction with no dedicated technique. CNAS elevates it to a first-class method [5].
3. Safety/alignment stripping — distillation that intentionally discards guardrails — is not captured as a safety-degradation technique distinct from theft. OWASP LLM10 covers theft; OWASP LLM01 covers jailbreak bypass; the distillation-for-stripping combination is unnamed.

The recommendation for taxonomy bodies is straightforward: add named sub-techniques for CoT/reasoning-trace harvesting, ML-as-a-reward-model abuse, and alignment stripping via distillation.

Ranking the variants by concern

To prioritize, I propose a transparent, additive score — in the spirit of Anthropic’s ARiES model (Threat + Vulnerability + Impact) rather than a multiplicative one. I call it the Distillation Variant Concern Score (DVCS). Four dimensions, each scored 1–5:

• T — Threat actor reach & intent (1 = academic, 5 = nation-state scale, documented)
• V — Vulnerability / feasibility (1 = expensive/fragile, 5 = cheap & demonstrated)
• I — Impact if successful (1 = nuisance, 5 = frontier-capability + safety-stripping + proliferation)
• D — Detectability deficit (1 = easily detected, 5 = routinely evades controls)

DVCS = T + V + I + D, range 4–20. It measures concern and enablement to a defender — how much a variant should worry you and be prioritized — not the probability an attack succeeds.

Figure 2 — The DVCS heatmap. Reasoning-trace harvesting scores highest at 19, followed by reward-model abuse and vertical distillation at 18. The top tier is dominated by behavioral extraction — the things frontier models are most valuable for and hardest to detect.

The interpretation is the important part. The top tier (DVCS 17–19) is dominated by behavioral extraction — reasoning traces, safety stripping, reward-model abuse, vertical capability. These win because they exploit what frontier models are most valuable for and what is hardest to detect. The classical parametric extraction (logit-based, hard-label) scores lower — 14 — largely because providers have already closed the easiest doors. After Carlini’s responsible disclosure, OpenAI and Google removed the logit-bias-plus-logprob combinations that made those attacks ten times cheaper [3]. The cheap, structural theft is already mitigated; the expensive, behavioral theft is not.

Detection: the operational center of gravity

Detection is the primary lever providers have, because most prevention degrades utility. The signals cluster into four categories [5]:

• Prompt signals — content designed to elicit synthetic data, reasoning traces, grading, or domain capability at scale. Anthropic built classifiers and “behavioral fingerprinting” specifically for “detection of chain-of-thought elicitation used to construct reasoning training data” [1].
• Identity signals — account age, payment method, billing address, credential ownership. The most-exploited onboarding routes are educational, startup, and security-research pathways [1].
• Network signals — IP origin, residential vs. corporate, geographic consistency, anonymizing routing. Transfer stations concentrate in Hong Kong and Singapore; some traffic flows over China Telecom and China Unicom networks [5].
• Behavioral signals — request volume, templating across many accounts, synchronized timing, and the telltale rapid pivot to a newly released model (the MiniMax <24-hour signature).

Each technique has real limits. Query-pattern anomaly detection catches blunt campaigns but generates false positives against legitimate bulk use (synthetic data generation, eval harnesses). CoT-elicitation classifiers collide with legitimate “explain your reasoning” requests. Output watermarking is a provenance aid, not prevention, and its robustness under fine-tuning and quantization is still an open problem.

The structural blindness is the hardest part. No single defender sees the whole campaign. A campaign distributed across a developer API, AWS Bedrock, Azure OpenAI, and commercial token mixers (Eden AI, OpenRouter, ShareAI) fragments every signal category. Cloud privacy commitments can blind the developer entirely. Self-hosted mixers like LiteLLM make attribution absolute — the adversary controls the intermediary. Prompt content remains visible to whoever serves the model — the one persistent detection surface [5].

This is why no single actor can counter the vector alone. It is also why detection must tolerate false positives; the alternative is to let the hydra clusters dilute every signal into noise.

Defenses and their honest tradeoffs

Defenses divide into three layers. Each is assessed here honestly: most degrade legitimate use, which is exactly why detection, not prevention, carries the load.

Figure 3 — The defense stack. The architectural layer (P0) is the only zero-utility-regret move: it removes the highest-value signal before it is ever emitted. Detection (P1) is the operational center of gravity. Access-side and output-side controls (P2/P3) are necessary but lossy.

Architectural defenses — what not to expose. This is the only layer with no legitimate-use regret. Do not expose raw logits, and never co-expose logit-bias with logprobs — that single design choice neutralized the cheapest extraction class [3]. Do not expose full reasoning traces by default; provide summaries only. Default-deny agentic and tool-use surfaces for unvetted accounts, because agentic traces are the richest behavioral signal. There are deeper architectural tricks too: split the final layer (h→t→l with a nonlinearity, where t > l) so the low-rank property the attack exploits no longer holds, or post-hoc expand the apparent hidden dimension with orthogonal noise vectors to mislead dimension-recovery [3].

Output-side defenses are lossy. Logit truncation removes constrained generation and logprob-based evaluation. Output perturbation degrades determinism and quality. Reasoning-trace hiding reduces transparency. Watermarking adds latency and is not robust to fine-tuning. Use these sparingly.

Access-side defenses trade friction for safety. Rate limiting slows extraction but is Sybil-defeatable across accounts. Tiered access and KYC close the most-exploited onboarding routes but limit researcher and startup access — the same democratization the field is trying to expand. ToS enforcement and legal remedy (CFAA and wire-fraud theories, Microsoft’s Storm-2139 action) deter but are jurisdiction-limited; transfer stations reconstitute under new identities faster than blacklists update [5].

The honest summary: prevention is lossy, detection is leaky. The defensible posture is architectural minimization of the exposed surface, layered with detection that tolerates false positives, and cross-provider coordination that closes the fragmentation gap. The goal is not to make extraction impossible — it is to raise attacker cost above the value of the marginal extracted token.

The trajectory

Three forces shape where this goes.

Reasoning transparency is the leading edge of risk. As models expose richer reasoning and agentic traces, information-per-query rises, and the behavioral variants at the top of the risk ranking get cheaper. Providers hiding CoT are responding, but elicitation-and-reconstruction persists [1].

Student-training efficiency improves, lowering absorption cost. Better distillation recipes, cheaper SFT, and more efficient RLVR mean a given teacher-signal budget yields a stronger student each year, widening the asymmetry.

Detection is an arms race the attacker currently leads. Campaigns are “maturing,” evading new protections within release cycles, and pivoting to new models in under 24 hours [1][5]. The defender’s durable advantage is prompt-content visibility; the attacker’s is identity, network, and behavioral obfuscation.

The plausible end-state is that any openly queryable frontier model is, by default, partially distilled by some determined actor. The question stops being “can we prevent distillation” and becomes “can we make the absorbed capability weaker, traceable, and safety-preserving — and can we slow the absorption rate below the rate at which we generate new advantage.” Export controls remain load-bearing here precisely because they gate the absorption compute, not the query compute.

What defenders should do, prioritized

P0 — Architectural. Do this first.

1. Never expose raw logits, and never co-expose logit-bias with logprobs. If you need constrained generation, use a token block-list instead of logit bias [3].
2. Do not expose full reasoning traces by default; provide summaries only. Treat CoT as a controlled, gated capability.
3. Default-deny agentic, tool-use, and computer-use surfaces for unvetted accounts — these are the richest behavioral signal and the primary observed target [1].

P1 — Detection. The operational center of gravity.

4. Deploy CoT-elicitation and rubric-grading (reward-model-usage) classifiers in the prompt pipeline. These target the highest-concern variants and are currently under-built [1].
5. Build account-graph correlation across identity, network, and behavioral signals to map hydra clusters — not just ban single accounts. Account-level enforcement loses to rotation [5].
6. Instrument rapid-pivot detection: alert when a coordinated cohort redirects more than N% of traffic to a newly released model within 24–48 hours [1].
7. Treat watermarking and fingerprinting as attribution infrastructure, not prevention.

P2 — Access controls and onboarding.

8. Harden KYC on educational, startup, and security-research pathways — the most-exploited onboarding routes [1].
9. Tiered access: gate the highest-value surfaces behind verification while keeping a usable free tier for genuine research. Be explicit about the democratization tradeoff.

P3 — Coordination and policy. Closes the structural blindness.

10. Establish an industry-led signal-sharing forum (a GIFCT-for-distillation) for hashed, non-content signals across developers, cloud providers, and token mixers. Clarify via DOJ/FTC and Stored Communications Act guidance that such sharing is lawful [5].
11. Share attribution intelligence with government (an NSA Cybersecurity Collaboration Center model) to map transfer-station infrastructure companies cannot see alone [5].
12. Pursue legal deterrence: support model-theft deterrence legislation and graduated IEEPA-based services-prohibition and sanctions on designated intermediaries [5].
13. Sustain export controls on advanced compute. They gate the absorption step that makes distillation strategically valuable, and are therefore complementary to, not redundant with, anti-distillation measures [5].

P4 — Taxonomy and measurement. Enable the above.

14. Petition MITRE ATLAS and OWASP to add named sub-techniques for CoT/reasoning-trace harvesting, reward-model abuse, and alignment stripping via distillation.
15. Fund controlled distillation-gain experiments (Epoch-AI-style reference points across methods and scales) to quantify impact even where full replication is impossible [5].

A note on framing

This analysis describes attack mechanisms at the conceptual level required for defense. It does not provide a usable extraction playbook, optimized query strategies, or step-by-step guardrail-stripping instructions. Where evidence is thin or contested — attribution of named labs, coordination among actors, the dependence of public model releases on distilled data — the uncertainty is stated rather than resolved. The goal is to help providers and security teams understand and mitigate the vector, and to stop there.

Sources

Foundational research

• [3] Carlini, N. et al. Stealing Part of a Production Language Model. 2024. arXiv:2403.06634. — Production-API extraction; <$20 ada/babbage, ~$2,000 gpt-3.5-turbo projection matrix; logit-bias/logprob defenses.
• [9] Tramèr, F. et al. Stealing Machine Learning Models via Prediction APIs. USENIX Security 2016.
• Jagielski, M. et al. High Accuracy and High Fidelity Extraction of Neural Networks. 2020. — accuracy vs. fidelity framing.
• [7] Response-Based Knowledge Distillation for Multilingual Jailbreak Prevention Unwittingly Compromises Safety. 2026. arXiv:2602.11157.

Provider reports and case studies

• [1] Anthropic. Detecting and Preventing Distillation Attacks. 2025. — DeepSeek/Moonshot/MiniMax; >16M exchanges, ~24,000 accounts; hydra clusters; CoT elicitation; pivots.
• [2] Reuters. OpenAI says China’s DeepSeek trained its AI by distilling US models, memo shows. Feb 12, 2026.
• Bloomberg. OpenAI Accuses China’s DeepSeek of Distilling US AI Models to Gain an Edge. 2026.
• Google Threat Intelligence Group — chain-of-thought extraction campaign (>100K prompts); referenced via CNAS.

Policy and threat-intelligence analysis

• [5] Remler, D. & Hayum, B. Adversarial Distillation. Center for a New American Security (CNAS), 2026. — taxonomy, supply chain, NSTM-4, H.R. 8283, Lambert token estimates, DeepSeek-R1 SFT ~6.4B tokens.
• Lawfare. U.S. Vows to Fight Distillation Attacks. 2026.

Taxonomy references

• [10] MITRE ATLAS. AML.T0024, AML.T0024.002, AML.T0024.001, case study AML.CS0056; mitigations AML.M0002, AML.M0004, AML.M0024.
• [6] OWASP Top 10 for LLM Applications — LLM10: Model Theft. OWASP ML Security Top 10 — ML05:2023 Model Theft.

Economics and demonstrated replication

• [8] Stanford CRFM. Alpaca: A Strong, Replicable Instruction-Following Model. 2023. — ~$500 self-instruct distillation from text-davinci-003.
• [4] tianpan. Knowledge Distillation Economics: When Compressing a Frontier Model Actually Pays Off. 2026. — ~130× faster, ~25× cheaper; domain-specific and brittle.