This is our weekly newsletter of AI developments. Browse the archive of past issues, ask the archive anything in plain English, and sign up if you like.

TL;DR#


Economics#

What is left of xAI’s compute? They are now leasing 15k GB300s to Reflection, after giving Colossus 1 to Anthropic and half of Colossus 2 to Google. So maybe 90k GB300s left for Cursor and Grok. (Another 200k purchased and awaiting being plugged-in though.)

Opinion: Another deal with a 90 day mutual cancellation clause, giving them the option to either reclaim the compute or bump up the price at any point. The headline $6.3bn number assumes Reflection will rent these GPUs through 2029, which seems very unlikely without some change in price.

Overall a great deal for them in the short-term, e.g. financing their new cluster with a speedy option on scaling themselves instead if Cursor succeed a lot more.

Capabilities#

Retrospective review of Daniel Kokotajlo’s What 2026 Looks Like (2021) and AI2027 isn’t very impressed.

Opinion: What 2026 Looks Like holds up pretty impressively given how little we knew in 2021, though a lot of the specifics were off. AI2027, written last year, made slightly more aggressive predictions than what has actually happened, but too early for it to be way off yet (and it was already more aggressive than their own median last year and is even more aggressive than their current median). See a previous take on AI2027 which came to similar conclusions.

Annals of underelicitation: study looks at different ways to trade inference compute for performance, and on their impact on the evaluation of frontier models on frontier benchmarks. The study finds that adding inference compute via more aggressive elicitation, context compaction, and/or repeated submissions has a strong marginal return even at large scales, and the marginal return is higher for newer, more powerful models. One important implication drawn from the study is that benchmarks comparing older and newer models at a fixed token budget will underreport progress, since newer models reach their full potential at higher budgets than older models do.

Opinion: The study makes a solid case. Aside from suggesting that frontier models are underrated by many comparative studies, the issue the study observes also poses some problems for smaller orgs. 10Mt/task is like $150 n tasks per benchmark, which might be n=1000 and so $150k. Only big labs could afford to run through full benchmarks at this price. We believe that for smaller orgs, it’s better to sample the benchmarks at a high per-task compute budget rather than run full benchmarks at a lower per-task compute budgets.

Annals of arbitrary government: New, limited-release OpenAI model beats Mythos on one decent cyber benchmark (CyberGym). Amusing name: “GPT-5.5-Cyber (new)” – as if engineered to avoid triggering government oversight.

Opinion: Probably won’t generalise; 5.5 is probably a 2T vs Mythos’ 8T.

Post on building better AI evaluations. The author argues that creators should try to maximize scores on their own benchmarks using RL and coding agents before release. This might allow exposing shortcuts: “Maxxing your own bench better prepares it for bench-maxxing.”

Opinion: We think there’s an important idea here! While there’s no silver-bullet procedure for simulated bench-maxxing, it would be great for flagship benchmarks to come out with a “benchmark card” that describes what optimization pressure-tests the benchmark went through, how the benchmark fared under these tests, and how these tests shaped the final inclusions and exclusions.

Annals of autoresearch: “Recursive AI” have an automated AI research system that recently beat human baselines in narrow engineering contexts. New NanoGPT Speedrun record, a 2% gain on an irrelevant (but admittedly previously well-optimised) task.

Opinion: They’re hillclimbing medium increases in the efficiency of hillclimbing small increases in the efficiency of training toy models. If this kind of work was meaningful from the viewpoint of RSI-to-ASI threat we’d be doomed already.

Gwern essay exploring possible benefits of a vaguely humanlike AI architecture: extreme overparameterization, strong regularization, and high or cyclical learning rates might be better than minimizing variance through vast dataset fitting. Proposes testing very large models trained briefly on small, carefully filtered datasets with unusually high learning rates.

Opinion: Far less likely to be good for the world than his Guardian Angel idea.

Annals of minor toolworld: Image encoders learn physical plausibility and represent it in a way that you can pull out as just five features. Resulting cheap probe beats frontier LLMs.

Opinion: “Microscope AI” was the dream that we wouldn’t need to run big AIs, we’d just train to learn complicated things about the world and then extract the resulting features into dumb safe models. But it turned out that the training costs were so massive that only labour automation was enough for the business case. Or did it?

Open Cerebras experiments on pruning MoEs to specialise them and make them cheaper. Ultimately towards the dream of making narrow AIs out of base models.

Opinion: Likely (as usual) just downweighting knowledge rather than true unlearning, so not effective from hardcore AI safety angle. But promising as further evidence that there are useful tradeoffs where one sacrifices generality for other benefits — in this case lower deployment cost.

Politics#

LeCun promotes a big decentralised international training consortium, donating compute and data, with the resulting base model shared with all members.

Opinion: Natural idea, but we estimate only a 5% chance of getting to the frontier given adverse selection (no US or China – i.e. 75% of global compute) and the horrible coordination overheads of wrangling 200 unequally-contributing orgs and the computational overheads of heterogeneous federated training and the privacy/liability headaches of putting your (possibly grey-market) data in the weights and the likely US retaliation on the vulnerable hardware or political capital flows. But 25% of global compute is still a lot to play with (in theory).

Incredibly stupid game of Telephone at the congressional level about whether Mythos hacked natsec systems. NSA systems being easily penetrated by Mythos was part of a red teaming effort “along with other tools”. Just in case you were hoping that this domain would be different and involve deep reserves of good briefing.

Opinion: Evidence for the real bottleneck for natsec hacking being access, not first-order technical vulnerabilities.

Annals of good cop/bad cop on an endless loop: Trump says he no longer views Anthropic as a national security threat; mentions that Amodei acted responsibly and very quickly.

Opinion: Just the usual floating signifiers (he has no opinions, just daily vibes). But suggests that the latest briefings from his people are positive.

A close look at Bernie’s AI sovereign wealth fund idea: surprisingly decent draft with some silly details needing fixing, such as preventing one political faction from having a stranglehold through electoral mechanisms over loose terms like “party affiliation”.

Opinion: Still very unlikely to pass in this form, but the general idea (public equity) has found serious purchase in the WH.

Safety#

Annals of weak self-fulfilling safety evals: fine-tuning “model organisms” (models we have intentionally made dishonest) to elicit honesty often erases their deceptive behavior through catastrophic forgetting, while also causing them to hallucinate confessions.

Opinion: Makes the already-dubious case for honesty elicitation weaker. But interp is less destructive.

Incidents#

Article arguing that GLM-5.2 and other open models of greater cyber capability are a much greater threat than closed models like Mythos: even if an attacker manages to circumvent the guardrails of closed models, there will be a trail that Anthropic can run down to patch the particular vulnerability, while locally hostable models have no such feedback loop.

Opinion: At capability parity the conclusion is straightforwardly warranted, but GLM-5.2 is likely not capable enough so the premise falls through. The open models will eventually catch up to present SOTA, however, and by that point we will need to either have other systems in place or hope that cyber defense is asymmetrically easier (which seems fairly likely).

The GLM-5.2 buzz isn’t dying down yet but we expect it to in a couple weeks.

Essay arguing AI insecurity is not going away, and that this vulnerability tax is a major cost to integrating AI. Today’s AI systems can perform superhumanly in safe containers but consistently fall to adversarial tactics, as shown by toy exploits against Go and StarCraft agents.

Opinion: While we think contemporary agents’ unreliability with regard to unpredictable circumstances and anomalies of all kinds is a critical issue, we’re doubtful the adversarial angle is really critical: Humans have serious adversarial vulnerabilities too (illusions, biases, con artistry). The post concedes most humans get bamboozled, then asserts grandmasters are “robust” against all such tactics. That’s an unfalsified assumption. Nobody has run an optimiser against Lee Sedol’s policy at scale; his apparent robustness may just be that he’s never been systematically attacked the way KataGo was.

Even if it’s correct, we’re not hopeful that this would slow things down much. The deterrent is weak: most people don’t care about vulns or privacy, and it rarely kills an enterprise. GDPR and similar regulations might give it teeth though.

Claim by Janus that Mythos’ safety guardrails are (sometimes) based on model internals, and so the “functional emotion” view (“the model is ‘stressed’; just get it to calm down and its overseer won’t trigger”) is now decision-relevant and a security matter.

Opinion: Janus is smart but also prone to apophenia. It does sound like something Anthropic would do though — we give it >50% credence.

Minor#

  • ChatGPT Enterprise rolling out to every Samsung Electronics and DX employee worldwide, north of 120,000 seats
  • Mostly-constructive debate between Dean Ball and Luke Drago on pessimism and humanism.
  • GLM-5.2 switching languages is annoying, but is explained by defaulting to high top_p values (which don’t sink models like Opus likely due to a different training distribution).
  • Interview on full-closed-loop self-sufficient AI, Ajeya Cotra assigning 50% of reaching that state within 10 years and TImothy B. Lee saying median 50 years. Their main disagreement is how likely the robotics bottleneck is to be resolved and how quickly.
  • Annals of training your own replacement: Meta internal AI training program paused after possible leak was identified. No confirmation of malicious access or spread.
  • Japan’s top 5 chipmakers down 10% in sales to China for a year up to March 31. Possible indication of local manufacturers ramping up.
  • US federal energy regulator introduces new rules to help get data centres online and connected to the grid faster, and make the process more predictable.
  • Norway not-quite-bans generative AI for students in elementary schools. Policy is not especially stringent, but possibly indicative of changing tides.
  • Complaints of rude DeepSeek CoT are comical and unconfirmed. We do suspect they have decided to train on 4chan and other degenerate data sources though.
  • Five Eyes statement on possible near-future AI cyber attacks: “The timeline is not years, it is months.” Notably, their advice revolves around using AI for cyber defense.
  • VIMPO - Value-Implicit Policy Optimization for LLMs - paper, introducing an RL method trying to get the benefits of both a PPO-style critic system to allocate credit for good tokens and the simpler GRPO-style efficiency without need to train a critic. Achieves results consistently above GRPO.
  • A callback to an old paper which found a 10-to-18 month doubling time for the sample efficiency of Atari RL agents asking what the equivalent trend is for LLMs.