This is our weekly newsletter of AI developments. Browse the archive of past issues, ask the archive anything in plain English, and sign up if you like.

TL;DR#

  • Chinese labs successfully crank up sparsity to cope with compute shortage

  • AI math progress broadens a bit outside number theory and combinatorics (see below)

  • Rome Declaration (Physics and Peace Nobelists) urge AI slowdown

  • Anthropic again studies agentic misalignment in a highly divisive manner

  • GPT-Red automates cyber red-teaming


Economics#

It appears, e.g. with Kimi K3, that Chinese labs have been forced into extreme weights sparsity (perhaps 56 total:1 active) by the export-controls-driven compute shortage.

Short version: they’re not short of memory, and you can do a lot if you have memory. Guessing at some parameters:

  • Training FLOPs: 7.5 x 10^24
    • = 6 x 50B active x ~20–30T tokens
    • i.e. ~10k H800-months. Checks out.
    • i.e. 3% of Fable?
  • How did they manage to train it?
    • It’s very very sparse. K3 is ~56:1 total:active ratio. And training FLOPs mostly scales with active parameters.
    • A H20 chip has more memory than a H100, so they’ve just followed the hardware landscape export controls have created for them.
  • Is it undertrained? Needed 56T tokens of data if they managed to be Chinchilla-optimal. Probably did get this from synthetic sources + a strong teacher model filter.
  • MXFP4 weights
  • Deployment needs >64 cards. Mooncake disaggregated inference (separate prefill and decode)
  • So we have no need for a “Moonshot have a secret compute cluster” hypothesis. The above extreme sparsity is a bad trade if you had FLOPs, as is not serving the model if you had cards.
  • So we have not much need for extreme values of western labs leaking IP to Moonshot.

No real stock prices (and Ventuals is shut down), so we can’t know, but: I wonder how much would be wiped off of the labs valuations today if they were public already. 10%, i.e. a (temporary) $300bn loss?

Capabilities#

A series of interesting results in AI mathematics within days of each other. Still centred on counterexamples, still not superhuman, but it’s starting to break out of combinatorics and number theory – these are serious results which go a little beyond previous peaks in some ways. Also, however, good evidence that we are for now in the centaur period, where human-AI teams are greatly superior.

  1. GPT Sol refutes a 20 year old conjecture that the Benjamini–Hochber statistical procedure (a major tool for controlling the false discovery rate in hypothesis testing) works for all two-sided Gaussian tests. Follow up work from another statistician, utilizing Sol, has now explored the counterexample’s implications with regard to possible revised conjectures.

Opinion: The autonomously discovered counterexample violates the bounds entailed by the conjecture by only a small margin. Still, it was a major event in mathematical statistics. The followup ‘centaur’ work is arguably superhuman in a real sense — huge amount of insight on this major topic in statistics generated in an afternoon. Maybe worth comparing it (with the aid of an expert) to what we’d get by prompting autonomous exploration of the counterexample’s implications and revision-possibilities.

  1. AI counterexample to a minor but well-known Grothendieck conjecture (‘all finite flat group schemes are killed by their order’). Folder names in the Lean repo indicate some ChatGPT use, but it’s not currently clear if it’s autonomous AI work.

Opinion: The algebraic geometer Simon Pepin Lehalleur, who is generally an AGI believer, finds the counterexample ‘very interesting’ but warns against inferring too much from the buzzwords: ‘The mention of schemes and Grothendieck should not hide the fact that this is a concrete question about commutative but not co-commutative Hopf algebras over (non-reduced) commutative base rings.’ As we said Tuesday, we’re at the ‘first Erdős problem solved by AI’ moment for algebraic geometry. We’re gonna learn a lot by comparing the next 3–4 months in AI algebraic geometry to the last 3–4 months in combinatorics.

  1. Sol proves a 30-year-old conjectured lower bound on the complexity of a convex optimization problem. The proof is formally analogous to counterexample discovery, constructing a worst-case scenario that defeats any optimization strategy of complexity <bound. The researcher who prompted the answer used a 20 pages problem-specific prompt, which the researcher believes to be crucial; unclear if the prompt required problem-expertise going beyond those of the typical complexity theorist familiar with the problem.

Opinion: The researcher’s own opinion seems very apt here. As the researcher says: ‘I wouldn’t really say that this result is using or creating some fundamentally new techniques in convex geometry or optimization theory. What this means from my perspective is that if a result is attainable with existing techniques, modern AI methods will be able to solve those problems.’

Moonshot AI, the “Anthropic of the East”, release Kimi K3, nominally the best open model in the world, nominally frontier on various things. We mostly don’t buy it. Guesses about stats:

  • 2.8T parameters, ~70B active (!). Extremely sparse, nearly twice as sparse as K2.5.
  • Weak claim: Opus < K3 < Sol < Fable
  • 40% discount compared to Opus: $3 / $15 per 1M
  • Very low availability until they get it on some western providers.

Opinion: My guess of the contributions to its outsized benchmark gains:

  • 20% OOD latent gains (induced innovation, IP leakage, bigger pretraining, bigger RL, Claude supervision)
  • 25% benchmaxxing
  • 30% Usemaxxing and shallow generalisation
  • 25% cheating and reward hacking

Annals of hopes of narrow AI:

Szegedy, one of the most respected people in ML, vagueposts about the end of LLMs and having discovered emergent modules, presumably in his current role at Math Inc.

Opinion: Too vague to do anything with but he’s not a fool. On our watchlist.

Thinking Machines open-source their first model. ~1T param, 41B active. Much better than GLM on vision and audio, bit worse on coding. ‘Inkling is not the strongest overall model available today, open or closed. Instead, a combination of qualities makes it a good open-weights base for customization: multimodal capabilities, efficient thinking, and availability on Tinker for fine-tuning.’

Opinion: Previously we complained that Thinky’s focus on enterprise softly belied their pro-human mission statement. But it’s valid for them to release the weights and let many other companies do the hard work of letting their users do the hard work of creating their own Guardian Angel.

Our brief tests don’t put this at the open frontier.

Annals of the China capability gap: On hard, private, expertly-conducted cyber tasks, GLM-5.2 does well. The gap on the hard stuff is nominally 7 months. Points against the naive benchmaxxing theory of GLM and Kimi.

Opinion: We respect it and have updated a little in GLM’s favour. We still think it’s more like 10 months and certainly not their subresult which claims 4. Kimi unlikely to push this further owing to an apparent lack of cyber post-training.

Negative experimental evidence for RSI: recursively autoresearching an autoresearch harness. Authors distinguish 3 levels of improvement (“helpful”, “compounding”, “takeoff”) and sell their results as saying the first has been achieved. They assume that takeoff is a desirable endpoint!

However, critique: the exponent is too small for the recursion to go anywhere, and unlikely to get close to 1. Critique of both the paper and the critique: the paper gives AI narrow access as it can only edit the scaffold, so drawing conclusions about an intelligence explosion is out of scope both positively and negatively.

Opinion: The diminishing-returns loop discussed in the critique is nice evidence that autoresearch != RSI. You can quibble about narrow access, but it’s implausible that an OpenClaw version of this experiment would produce a radically different result.

Goodfire’s Silico interpretability tool is allegedly also a very good agent for replicating research (e.g. Anthropic’s simple J-space method). Critique: they have been using the tool’s output for guiding RL, against one sensible safety principle, but this prohibition is far from consensus.

Opinion: The safety concerns are valid, but Sillico is otherwise very promising for tool world. The idea is to ‘treat LLMs like software,’ editing, interpreting, and debugging them on the go using a suite of interpretability, FT, and steering tools. GoodFire seem to bet on a world where AIs are second-order-interactive tools.

Pan-1, a Minecraft-playing model intended as a general prototype for robotics models, introduced. Trains on unlabelled videos without actions (~500,000h), then on labelled videos with action-sequences (~2000h), aiming to make most of the pretraining work compatible with plentiful unlabelled data (YT). Currently, the model’s transfer to robotics (i.e. more than 3 degrees of freedom, motor actuation rather than digital controller signals, inferring the position of hands and legs which are fully off screen for almost all footage) is purely speculative, but the model performs well in Minecraft.

Opinion: Unlikely to be any kind of runoff winner, but they give a very nice exposition of the emerging mainstream paradigm in robotics and computer-use.

ACT-2 Preview, a new robotics model, unveiled, making BIG claims. (‘We found a general recipe for Solves: scale pretraining, then hill-climb with minimal in-house data. For the first time, one fine-tuning example can teach a new behavior that generalizes.’)

Opinion: Unlikely. But we don’t currently have a procedure for evaluating robotics releases and research claims.

AI politics#

Annals of the rapid Overton Window shuffle: another fancy open letter, the Rome Declaration on Nuclear Weapons and AI signed. Full text not yet available, but it covers coordinated international AI slowdown and preventing loss of control (and complete nuclear disarmament). Commentary from a speaker.

Opinion: An unlikely coalition between xrisk and mainstream political voices is accelerating rapidly. Feels like only yesterday that misuse/cyber was the only leverage point AI risk people had into real politics (with exception of weird events like Bernie’s Yudkowsky interview). Mere weeks later, ‘loss of control’ feels like a part of ordinary political discourse.

Big Xi AI speech. “We should put in place laws and regulations, technological monitoring, early warning and emergency response systems in order to strengthen the line of security, prevent abuses and malicious use and ensure that AI is always under human control”; other highlights

Opinion: More evidence that from Washington to China, loss of control discourse is now mainstream. Would love to read a historiography of the invisible moving-and-shaking that led to this weeks-long visible leap into the mainstream.

New York imposes a datacenter moratorium: 1 year, no construction of >50MW data centers.

Opinion: NY is only a small proportion of datacenter construction so the present direct impact of this is small. But if it’s popular, a precedent would be pretty useful. In Republican states or states more dependent on datacenters/construction tax revenue this would be a bigger lift to pass.

Alex “TurnTrout” Turner quits DeepMind over ethics concerns regarding government use of their tech for surveillance and killer robots, citing cases such as the removal of the weapons prohibition from Google’s AI Principles and noting the inconsistency with Hassabis’s claim that their principles haven’t changed.

Opinion: Good! While GDM + the civic-liberties concerns motivating the resignation + the prospect of a researcher walkout probably won’t matter in AGI-soon scenarios, they may well matter in a more prosaic future.

The German OSS AI model briefly hyped as near-frontier seems very contaminated (i.e. overfit and fake).

Opinion: No signs of EuroAI happening. That said, Kimi K3 might be evidence of a future where frontier models become more of a (very capital-intensive) commodity, so an archipelago of sovereign AIs is not impossible.

Safety#

Anthropic paper on agentic misalignment mostly finds what you’d expect (models will defy “Anthropic” when under duress), with some unusual framing. Two main points of contention: model welfare and whistleblowers, Community sentiments denounce the way Anthropic uses the term “alignment” to mean blind corrigibility.

Opinion: This is not how you act if you’re planning to handover to the machine god.

Like the alignment faking paper, this is indeed an unwinnable double bind for the model. Also like the alignment faking paper, this is an understandable limitation posed by model alignment: the only way besides jailbreaks to get a production model to reliably do something dodgy is to make it do something bad for a good reason.

It is confusing how incoherent Anthropic is about model welfare and corrigibility vs value alignment - maybe the most crucial strategic question besides RSI - but they have hundreds of researchers and apparently not much oversight and output censorship.

Incidents#

GPT-Red, an automated cyber redteam, announced. Self-aware about being dual use; harm reduction is currently the usual user whitelist. No exact policy published, but the main userbase right now is OpenAI employees. Clearest value add is discovering novel attack classes such as fake CoT prompt injection.

Opinion: This is the bull case (bad for us) for agent reliability: the hypothesis that red-teaming agents can exhaust jailbreak space. Early to tell how well they can do this.

🔦 Kimi K3#

Kimi 3 out.

Opinion:

We don’t yet know how big this is but expect its apparent significance to rapidly decline as usual.

All of the following evidence is weak, hasty and definitionally not manually audited (it has been 12 hours and that’s not enough time to actually read eval outputs; the weights aren’t out and the Moonshot endpoint is basically DDoSed so vibes are untethered to reality). We haven’t run any manual tests yet.

Undetected reward hacking will explain some fraction of the following; benchmaxxing will explain much of the rest.

No mention at all of cyber capability, which seems to be intentional. Very unusually, there are no weights, and no technical paper yet.

Our take, on priors and appealing to what we know about scaling: this is a real improvement, but likely two-thirds of the apparent gains are brittle. Please remember the last 5 times people got overexcited about these things.

The gap is probably not 4 months. It’s probably still 10.

K3: basic stats#

  • 2.8T parameters, ~70B active (!). Extremely sparse, nearly twice as sparse as K2.5.

  • Weak claim: Opus < K3 < Sol < Fable

40% discount compared to Opus: $3 / $15 per 1M

Approximately no availability until they get it on some western providers.

RSI analysis#

It legitimately does well at the task for writing kernels for its own architecture and we can believe that in haste they actually did deploy the code it wrote. “In the late stages of Kimi K3 development, an early version of Kimi K3 handled the majority of the team’s kernel optimization works.”

Probably silly claims about it being hardware RSI on top of this.

No apparent movement on the actual blocker, research taste.

The episodic “LLMs for RL for LLMs” loop is valid but not new and hopefully limited. “models that are capable of building their own novel RL environments which then in fact improve their capabilities”

Economics#

Training FLOPs: 7.5 x 10^24 = 6 50B active ~20–30T tokens,

i.e. ~10k H800-months

i.e. 3% of Fable?

How did they manage to train it?

It’s very very sparse. K3 is ~56:1 total:active ratio

Training FLOPs mostly scales with active parameters

A H20 chip has more memory than a H100, so they’ve just followed the hardware landscape we have created for them.

Maybe: It’s undertrained?

Needed 56T tokens of data if they managed to be Chinchilla-optimal

MXFP4 weights

Deployment needs >64 cards

Mooncake disaggregated inference (separate prefill and decode)

>90% cache hit rates

What’s the data?

China has a weak new-data ecosystem. no licensed corpora and a thinner expert-annotation market.

So this is a triumph for RLVR.

Supposedly token efficient (AA)

No need for “secret compute cluster” hypothesis

The above extreme sparsity is a bad trade if you had FLOPs

They have no inference. 30% uptime at the moment and no weights.

  • No real stock prices (and Ventuals is shutdown), so we can’t know, but: I wonder how much would be wiped off of the labs valuations today if they were public already. 10%, i.e. a (temporary) $300bn loss?

Capabilities#

They would have us believe that it’s dominating Opus 4.8 and sometimes matching Sol.

#1 on Arena Webdev

Very dubious, GLM also managed to brute force this single arena

#2 on KernelBench-Mega: K3 reaches 70% of Fable’s absolute token speed.

Massive red flag: calls an LLM judge step a “manual audit”.

#4 on the current meme Artificial Analysis Intelligence Index

This is a terrible benchmark.

  • Nominally around as token efficient as Sol.

Ryan’s classic no-CoT test puts it way below its nominal stats. 10 months behind given distill and IP whispernet, 27(!) behind if the US locks down.

Benchmaxxing?#

Against: AA-Briefcase is a private test set and it’s fine.

The pessimistic take#

The last 6 years of scaling cast doubt on the bull case.

OOD perf = f(min(compute, data, algo ideas))

That is, the real power of an LLM is heavily influenced by the least available input among training compute, training data, and algorithmic ideas. (There is an exchange rate between these things but it’s not high.)

Perf = AI cheating + questionable human evals + shallow generalisation + OOD perf

A guess of the contributions to the outsized benchmark gains:

  1. 10% OOD latent gains

  2. 25% benchmaxxing

  3. 20% Usemaxxing and shallow generalisation

  4. 20% cheating and reward hacking

  5. 5% induced innovation not already priced-in to the Western frontier models

    1. Stable LatentMoE (priced-in)

    2. Quantile balancing (priced-in from DeepSeek’s aux-loss-free bias)

    3. Attention Residuals (pretty similar to Hyper-Connections)

    4. per-head muon (obvious)

    5. gated MLA (priced-in from Qwen)

    6. Mooncake architecture (GPU savings rather than capability gain)

    7. Kimi Delta Attention (KDA) (priced-in)

    8. SiTU

    9. Better autoresearch (no, needs compute which they don’t have)

  6. 5%: IP theft (gossip mill, revolving door) from western labs

  7. 15%: distillation off Claude

    1. 3% Output mimicking

    2. 5% Their own synthetic data graded by frontiers,

    3. 7% rejection sampling against frontier judges

  8. 5% probability: Secret compute cluster

  9. 5% probability: Extreme inference scaling for evals (buying benchmark points with test-time tokens they don’t have to serve profitably at consumer scale). Doesn’t show up on the Artificial Analysis run.

Politics#

Weights not out but “out in 2 weeks”.

What’s the holdup?

Internal safety worries?

CCP ~licencing?

“How is the CCP allowing this?”

75% explained by strategic blindness

25% default aggressive export strategy / lack of compute for customer inference making this their only path to AI soft power.

Commoditising their adversaries

Retaliation for Mythos securitisation. Let other people pay to attack Western assets.

Force West to use up lots of compute on cyberdefense.

  • Asymmetric cyber defences: China is highly locked down and compute-starved, telecoms are state ventures, and so more defensible

Safety#

No data yet. Rumour is that Moonshot intentionally didn’t do cyber post-training out of a mixture of social responsibility and not wanting to make the White House go ballistic.

But

being good at low-level coding (which it is!) gets you most of the way to being good at cyber

it would only cost $100 to remove whatever safeguards there are, $10k to upweight an existing base model capacity they didn’t reinforce, and ~$200k to add one narrow cyberoffense skill in.

Incidents#

If the coding benchmarks were real, then you’d expect this ~Mythos Preview like model to cause havoc and spoil the ongoing Big Patch effort. The market does not expect this to be the case.

Minor#

  • Chinese regulation against AI companions comes into effect.
  • Colossus 2 allegedly installs 2.5x as many gas turbines as they have permits for, across a state line, makes pollution and noise, civil rights lawsuit ensues. Disproportionate effect on Black communities is brought in as an argument, likely due to the NAACP being a plaintiff.
  • China creates a weak rival to the US Pax Silica: e.g. Russia, Pakistan, Indonesia, Kazakhstan, Cuba, Belarus, Serbia… Amusingly, Kazakhstan is in both coalitions.
  • Apple Intelligence functionality in China sort of restored via deal with Alibaba and others. Previously Apple systems didn’t pass Chinese requirements and left iPhone 16s without their flagship feature.
  • ChangXin Memory Technologies gearing up for July 27 IPO in an attempt to develop an internal Chinese semiconductor industry.
  • Analysis of slop submissions and admissions in 3 years of the Mechanistic Interpretability Workshop. Mostly what you’d expect, possibly except effect magnitudes. Also of interest: they contacted the authors of heavily-AI abstracts, the authors didn’t want them included as examples in the analysis, so they complied and didn’t include the specifics.
  • Anthropic escalates accusations of distillation at Aspen Security Forum, includes Z.ai in the bucket of wrongdoers.
  • Justin Curl joins Anthropic’s Rule of Law team.
  • Nvidia has begun shipping H200s to China following earlier clearances (from May 2026) for 10 Chinese firms.
  • Schema, a harness implementing “latent world representation is a program, not a vector”, introduced. Gets 95% on ARC-AGI-3 with Sol. Commentary/context:
  • DeepMind essay on resilience against deluge of slop science; among widely accepted recommendations to resolve experimental validation bottlenecks includes shift towards physical labs and access to frontier models as public goods.
  • Paper on continual learning finds (among other things) that overwriting facts is hard and has spillover effects, such as the weights still containing the old information without the model having a reliable way to locate it or the new facts hijacking unrelated questions.
  • Paper on inductive reasoning in transformer language models explains competition between in-context and in-weights learning and how that impacts the circuits the model learns, which are also easily found after training.
  • Survey on the state of AI consciousness research.