This is our weekly newsletter of AI developments. Browse the archive of past issues, ask the archive anything in plain English, and sign up if you like.

TL;DR#


Economics#

USG buys $9b of AI datacenters for its spies. After infra costs, maybe 100–200k H100e. What for?

Maybe: abundant, accredited Gubmint Compute to host multiple vendors’ models as managed services, so the IC isn’t gated by any one vendor’s capacity or usage policy.

Opinion: Good, on the first order (less GPU for the scalers). Bad, on the second order, if this is a way of subsidising the labs. Terrible, on the third order: this is a loud signal to China of a preemptive “strike” (USG AGI project).

Trump nominally, prospectively hits US AI talent pipeline hard: aims to require e.g. CEOs to leave the US while waiting for their green card or citizenship. Reading between the lines, it’s an anti-Indian thing. But the actual law is about giving agents discretionary power, so it could totally skip AI people.

Opinion: Exactly the kind of shocking thing they announce and then withdraw, or do for two months then undo.

In Q1, OpenAI’s operating margin was –122%, not even counting stock-based compensation

Opinion: Not clear exactly where OAI are spending so much more than Anthropic here. Gross margins on inference are lower (33% vs 40%, mostly because of the cost of serving so many free users) but the biggest items are likely higher staff costs (more employees and higher turnover) plus more compute spent on a wider range of R&D directions than Anthropic (e.g. the now discontinued Sora).

Orbital data centers are unlikely to claim a large share of global compute before 2030 despite their energy efficiency and regulatory freedom, but the heat and maintenance problems are apparently not insurmountable. This is due to their bandwidth limits (making space inference more feasible than space training) and the insane difficulty of fixing hardware failures leading to onboard redundancy.

Opinion: This is actually a positive update. But large error bars on the key question, which is whether shielding keeps the unit fault rate low enough for this to make any sense. SpaceX only wins if this is possible within ~20 years.

Huawei aims for “1.4nm” nodes for GPU silicon in 2031, i.e. only 3 years behind NVIDIA using this one crude marketing-based metric.

Opinion: On face value, it’s little faster than expected but still way off. They have been doing fairly well on the Ascend inference side so I don’t discount this as pure corporate/communist overclaiming but I doubt that they will make this number by 2031 and will still be behind in reliability and stability even if they do.

Capabilities#

Zyphra claim steps towards futuristic replacements for backprop. Extending Equilibrium Propagation to skew-gradient systems with the goal of finding algorithms that don’t involve back-propagation, eventually striving for “neuromorphic silicon, biologically grounded models, and heterogeneous compute”.

Opinion: From any other neolab, this would be dismissible. As it is it’s only unlikely to work out (10%).

Famous ML hacker Geohot comes out against LLM coding. “I’m not saying that AI isn’t useful, it clearly is. It’s definitely a better Google for most searches. And whenever you need a quick prototype and don’t care about polish, it is absurdly fast. But is it a software engineer? Not close to the bar at any company I have worked at. The key aspect is knowing when to use it and when not to.”

Opinion: Surprising, Q1 2025-flavoured opinion. He’s ignoring the state of the median professional coder, on which the best LLMs are a clear improvement even without any senior human steering or cleanup. But it’s of course highly plausible that we aren’t in the superhuman or super-Geohot regime, and it’s somewhat plausible that we won’t get there soon.

Rohit releases BenchBench, which evaluates how well models can create benchmarks for other models. GPT 5.2 wins — beat GPT 5.4 and 5.5! — and Gemini gets honorable mention.

Opinion: Backwards progress is always very interesting, minor as this is.

Politics#

The Pope’s first encyclical is all about AI. Generally views it as a threat to human flourishing. Comments from Chris Olah (stylised scientific facts, flatly contradicting the encyclical in several ways), Dean Ball (anti-Euro libertarian angst), Adam Thierer (opportunity cost of slowing AI, plea for muddling through). “merely regulating [AI] is insufficient; it must be disarmed, welcoming and accessible… Disarming AI means freeing it from the mentality of ‘armed’ competition… freeing technology from monopolistic control and opening it to discussion and debate, therefore making it human-friendly.”

The following was very unlikely on priors: apparent AI usage in the Pope’s encyclical; in the original Italian draft, >10% appears LLM-based.

Opinion: No intellectual progress here, but you could view it as catching a neglected audience up. More broadly amazing PR for Anthropic as part of their play to get out of their assigned faction in the culture war.

Tencent’s former AI head says China’s LLM industry lacks paradigm-level innovation. Liu believes China can still win in AI by focusing on video generation as a stepping stone to real-time world models and by basing operations in Singapore.

Opinion: “World models” is a common refrain (see non-Gemini Deepmind) and it’s not a crazy claim, if vague. Currently I expect the text and image models to continue growing their lead up to whatever the scaling limits turn out to be.

Chinese PLA (Army) article on the official Chinese military perspective on AGI. Authors argue that a major shift toward algorithmic dominance and rapid autonomous decision-making is underway.

Opinion: The kind of noises that the US Army was making 5 years ago, but the tech is ready this time.

Safety#

It’s actually quite hard for labs to collaborate on safety for antitrust reasons. But you could give governmental safe harbor for mutual evals, and mandate it to remove this threat.

Opinion: Would be great, but I would guess lab collaborations are still bottlenecked on trust and race mindset rather than liability.

Model personas appear surprisingly early during pre-training (0.22%) and then get amplified. Some use this to infer their shallowness.

Opinion: Another win for the deflationary “pretraining creates the representations” view of RL.

Apparently contradicting the above: “selfhood” emerges in models during post-training. Post-training gives models a “self-recognition” capability, manifesting as higher confidence when continuing their own text than reading others’ text.

Opinion: This is, I guess, a second mechanism for model selfhood besides persona vectors. Synthesis with past work showing base models can notice when text is not theirs.

Argument that safety debates fixate on the wrong question (“what timelines to which capability”) and proposes instead the “DIAL” distribution — Decision Importance Adjusted for Leverage — i.e. when the most consequential decisions get made, reweighted by how much your effort moves the needle in each. Argues against people starting on credentials (e.g. 2 year Master’s!) now.

Opinion: Sensible given the assumptions, interesting to know how short-term CG are thinking.

Argument that alignment work will soon become much easier via cheap formal verification; extrapolates the current trend of AI math capabilities improving faster than AI R&D.

Opinion: Would be great news but imo the binding constraint on alignment is the pre-formalisation step (turning fuzzy alignment questions into any well-posed maths), which a proof oracle does nothing to relieve.

LLMs reasoning about harder problems tend to have “more direct conceptual trajectories” after you correct for their output length. Evidence against the hypothesis that reasoning models “just” think for longer.

Opinion: Interesting black-box approach using path statistics. If you believed that models didn’t activate differently based on the current problem difficulty, or that CoT RL led to a very shallow change in the model, then stop.

Incidents#

Successful redteaming of Hugging Face Datasets; this specific incident is not noteworthy and only got downloaded 2400 times, but indeed there were no mechanisms to detect this or propagate updates to anyone possibly affected. Key quote: “Cleaned,” “v2,” “filtered” — these are just README strings. I copied them from a real dataset. Nobody verified anything.“

Opinion: The insecurity of models remains a major source of deployment difficulty and near-term risk / fire alarms. This instance is irrelevant except as a prompt to consider which systems handling your sensitive data are similarly harvesting poisoned data.

The post-Erdos era#

Poor Deepmind. They solve 9 Erdos problems at the same time OAI solves a famous one, and get totally ignored. See also a proof that Mythos also manages to prove Erdos #90 in a similar fashion to OAI.

How should we think about ‘the Erdos era’ — the era where what we’ve been calling vertical generalization is tested, demonstrated, debated, and shown-off more or less exclusively in the terrain of Erdos problems?

  • Erdos problems are not by definition especially important, famous, or profound — it’s just a list of every conjecture Erdos ever made in writing. There are currently about 600 open Erdos problems. A minority within that list of 600 problems are considered very difficult (in the sense of ‘very strong specialists spent a lot of time on them’) but arguably ‘shallow’ problem, such as Erdos #90 aka The Unit Distance Problem solved by OpenAI. A much smaller minority are believed to be both very, very difficult and very deep.

  • The original, legitimate role of saying ‘AI solved an open Erdos problem’ was strictly to certify non-triviality. If a proposition/question is an ‘Erdos problem’ it means a strong mathematician (Erdos) thought it’s worthwhile to write it down — and so that it’s at least a little mathematically interesting. If the proposition/question is furthermore an open Erdos problem, you also get defeasible evidence that even in the ‘00s the problem is not so easy that any pro mathematician can do it on the spot. (Since, at the least, the mathematicians running the Erdos problems website gave each problem a quick look.)

  • There was also always a PR side-effect, if not sinister motive, to the ‘Erdos problems’ meme. Many technical people outside pure math assume that because Erdos is so famous an ‘Erdos problem’ (e.g. one of 1200+ conjectures Erdos made) must be comparable to a ‘Hilbert problem’ (e.g. one of the 24 deepest questions allowed by early C20th math) or ‘Millenium problem’.

  • Is anything happening outside of Erdos-land? GDM’s new paper states — and demonstrates — that their new agent proved a result in algebraic geometry, often considered a more truly ‘higher math’ area. However, experts in private consultations told us that the result (‘Log-Concavity of Hilbert Sequences’) was algebraic geometry in name only: it is unmistakably ‘combinatorial’/’Erdosy’ math, and not of interest to the kind of algebraic geometers that gave the subfield its reputation for profundity.

  • GDM are again using a complex hybrid LLM/Lean-verifier system with a carefully tuned domain-specific harness. OpenAI claim that their internal model that cracked Erdos #90 aka The Unit Distance Problem was ‘a general reasoning model’ not specialized for mathematics. Unclear what this means — does it mean no special harness? Not in-harness finetuning? No Lean? (Back in February, harnesses played a critical role in GDP strongly outperforming OpenAI on the First Proof challenge.)

  • It’s unclear if OpenAI’s internal ‘general reasoning’ model’s Erdos #90 solve demonstrates any new rise in the power of ‘general reasoning’ models. We’d need to know exactly what OpenAI’s discovery process for targeting Erdos #90 was to know if their process relied on a jump in model capabilities.

  • Overall we (i.e. the culture) are in epistemic hell when it comes to frontier models cutting-edge-research-math capabilities. We’re short on appropriately contextualized info about results are being churned, and we face a theoretical/philosophical challenge when it comes to evaluating the significance of hard-Erdos-math ability as a leading indicator of “true” intellectual ability.

  • On the theoretical/philosophical side, more mathematicians are now asking themselves whether a certain archetype of mathematical snob — e.g. an algebraic topologist or K-theorist or noncommutative geometrist who always regarded Erdos-style math as unserious, and who regards even Fields medalists like Timothy Gowers or Terry Tao as doing a ‘light’ type of mathematics — was on to something.

  • As an example of what the next frontier looks like: Daniel Litt’s Problems I Like, which contains problems that are pre-registered as serious and substantive. They have crunch to them but would not elicit dismissal as depthless from depth-first mathematicians, though the author’s taste is still a factor.

See also a recent interesting essay on LLM mathematics from a French mathematician:

“…If the proliferation of bogus proofs produced by incompetent AI forces us to convert all mathematics into formal language (including proofs supposedly produced by humans because we can no longer be certain), it will represent a massive setback for our [mathematics’] capabilities.

… [OAI’s Erdos 90 proof was] relatively untechnical, easy to prove, and require no complicated tools, but which still needed to be “thought” to be introduced. (The difficult part of the proof… is a standard technique known to experts. …their silence on the time or computing power is very telling…)

…There is a very credible possibility that AI will simply put an end to the human adventure in mathematics… by destroying the economic foundations of the practice of mathematical research; or at the very least, that it will lead to reserving the practice of mathematics for the wealthiest, who will be able to afford access to top-of-the-line, deluxe AI .

…The LLM seems to have beaten humans on this problem for three main reasons: it wasn’t influenced by Erdős’s intuition, it has an encyclopedic knowledge of mathematics, and it has inexhaustible patience for exhaustively testing avenues combined with a total intention to solve this problem and this problem specifically. On the other hand, there’s no trace of a flash of intuition, a new technique, much less superhuman genius (at this stage). This is roughly how I interpret the current situation. We must neither overestimate this result (current LLMs are absolutely not superhuman[ #20 ] in math), nor underestimate it (they are not stochastic parrots either, as some like to say).

Minor#

  • Sci-Hub launch an LLM query service on their vast corpus. Crypto token and everything. There is still a large gap in the market for things that are too illegal or dodgy for labs
  • Article analyzing China’s chipmaking (SMIC’s) workforce, noting that it’s young and has 3x the turnover rate of TSMC. Postulates that failure to retain mid-career engineers is crippling.
  • Bezos’ Prometheus is not really a neolab: making next-generation version of CAD for designing physical objects
  • Proposal for identifying frontier AI models without relying on a simple FLOP threshold. Effective Compute Index for better handling of missing data, benchmark updates
  • Some thoughts on bottlenecks in AI progress and where money will/should flow in the near future.
  • Cool widget which visualizes chains of conditional probabilities regarding the influence of AI to get to a final existential catastrophe score. Has estimates of how various public intellectuals thinking about AI might evaluate the sub-questions.
  • Can’t even predict capabilities. This is a symptom of desperation as well as technical maturity.
  • Cheap synthetic content (“AI slop”) is on the rise with no relevant intervention from governments or AI labs in sight. Deepfake concerns also rising, but precautions exist in that domain.
  • Nvidia CEO states that Nvidia expects its projected $200 billion CPU market to include China despite uncertainty about US policy.
  • Deepseek lock in previously-temporary 75% discount. Per-token costs 10x-30x cheaper than frontier models (GPT-5.5, Opus 4.7). Less impressive than it might sound; not all tokens are made equal.
  • Cybersecurity report claiming that direct vulnerability exploitation has overtaken phishing as the primary entry point for cyber attacks due to AI assistance.
  • Some Opinion: s shifting towards pessimism about AI coding. “I think that deep learning is still the solution, but real programming agents will need world models”
  • User-inspectable chatbot with built-in steering, Steerling-8B.