This is our weekly newsletter of AI developments. Browse the archive of past issues, ask the archive anything in plain English, and sign up if you like.

TL;DR#


Economics#

The OpenAI Foundation has an enormous $180B endowment. Herbie Bradley explores how to deploy this capital efficiently. Since traditional grantmaking and startup incubation face severe talent bottlenecks, he suggests the Foundation use its assets as financial leverage to underwrite and subsidize beneficial AI projects. His proposals:

  • An AI Catch-Up Facility to subsidize AI deployment for developing nations and improve their state capacity.

  • An AI Risk Reinsurer to backstop commercial insurance policies covering catastrophic AI incidents like cyberattacks or rogue agents.

  • Advanced market commitments to guarantee the purchase of new AI-driven research products and incentivize early private investment.

  • A few more, like the creation of a compute futures market to reduce the risk of a frontier AI lab bankruptcy, or an underwriter of zero interest rate loans for small manufacturing shops to adopt robotics faster.

Opinion: It’s a smart proposal given a worldview different from ours (adoption boosting, growth boosting priorities). If you talk with grantmakers at e.g. Coefficient Giving they’ll tell you their main bottleneck is talent availability and access to ideas bottlenecked by capital that are worth funding. Herbie’s proposal is smart: rather than face the same structural bottlenecks, the OAI Foundation could leverage its capital to still facilitate positive outcomes by making it easier for efforts that already have solved this bottleneck to access cheaper capital.

Dodgy short-seller, Culper Research, posts a bear case for NVIDIA. Despite their much-touted “0% China” sales claim, Culper thinks >20% of FY2026 AI revenue was driven by China, both through illegal chip smuggling and the use of intermediaries in SE Asia. Singaporean company Megaspeed seems to be implicated in a bunch of different smuggling cases – allegations:

  • Very close ties with NVIDIA, including Jensen. “secretly financed by the Chinese company Alibaba via shell companies.”

  • $4B worth of AI servers from Aivres Systems, parent on the US Entity List.

  • One-third owned by the CCP.

  • Links to the $2.5B Supermicro smugglers.

  • NVIDIA changed the way it reports revenue, eliminating Singapore as a reported geography.

Opinion: Interesting thought: what is the probability that natsec didn’t know this already? 40%? The budget squeeze on BIS points to powerful people not wanting much attention on this area.

Either way, good leverage for Trump over Jensen but we don’t think they’ll bite Jensen at the moment.

Anthropic finalizing another $30B fundraise @ $900B. Just three months after $30B at $350B. Big Tech companies are not currently expected to form part of the new $30B round.

Opinion: Probably pushes back IPO date in expectation (into 2027? Waiting longer would require more of these rounds probably), but otherwise not much to say on this. Would be great to know if they are actively avoiding it.

More details on the US conditions for NVIDIA selling H200s to China, 10 companies named as acceptable final recipients.

Opinion: Not really an update, still waiting for Beijing to unblock. Huang is there and persuasive.

GDM spinoff Isomorphic Labs announces $2.1B secured in second (external) funding round, aiming to ramp up AI drug design.

Opinion: opportunistic raise due to current optimism around anything AI. Likely leading in structure-based AI drug design, but not clear that they’ll be able to replicate AI success in domains with cheap, easy verification. They recently delayed their target for clinical trials. Current pharma returns on human R&D are sufficiently low to make AI platforms like this pretty obvious buys for them.

Cerebras, NVIDIA wannabe specializing in AI chips, raises $5.5B.

Opinion: Cerebras is inference-focused, and not going to meaningfully affect the overall picture in the near term - projections are for approx 500k H100 equivalents produced in 2028, 0.7% of our global estimates for compute production.

Capabilities#

How good are models at building Game Boy Advance emulators? Mechanize finds that GPT-5.5 and Claude models perform well when given 24 hours, while Gemini 3.1 Pro is unable to produce a working emulator and Chinese models fail to produce an emulator that compiles.

Opinion: Ad hoc time horizon coding test: probably a month or so of skilled human effort? Pretty cool minor confirmation of our prior on Chinese AI capability. But of course contamination is major for this too.

A new benchmark, ExploitGym, evaluates how capable current AI systems are at turning vulnerabilities into concrete security impacts, such as unauthorized file access or code execution. The benchmark collects around 900 vulnerabilities sourced from userspace software (e.g. FFmpeg and OpenSSL), browsers (the V8 JavaScript engine used by Chromium) and the Linux kernel, and tests whether AI agents are able to successfully capture the flag – one stored outside the agent’s authorized scope – by executing code with privileges that should not be obtainable under the specific security model.

Opinion: More evidence that Mythos is slightly more capable than GPT-5.5 at cybersecurity, albeit at a much higher cost. It’s somewhat unfortunate that the authors used a 2h time limit when running the benchmark; fixing a certain max budget would probably have been a better x-axis.

Prime Intellect throws 14k GPU hours at an autoresearch loop to set a new record on nanoGPT speedrun (where researchers search for the fastest algorithm to train a GPT-2-scale LLM). Both Opus 4.7 and GPT-5.5 outperformed the human record, but only after being seeded with human-generated ideas the models could iterate on. A track dedicated to fully model-generated ideas didn’t lead to much.

Opinion: Pretty cool minor confirmation of our prior on AI creativity.

METR surveys technical experts on their AI use, finding 1.6x-2.1x on value from incorporating AI, expected to increase over time; authors explicitly flag unreliability of the survey format due to the unreliability of self-reports.

Opinion: One interesting tidbit is that per Becker if you focus the survey on the value created (rather than the speed multiplier achieved) then you get lower self-reports. This seems to be consistent with the view that sees the current paradigm leading to acceleration in proof of concepts and speedy iteration rather than qualitatively better ideas etc.

Discussion of the current paradigm’s main crux: whether AI research automation with coding agents will lead to explosion in AI capabilities. More interesting than usual, partly because the author is changing their mind: their view is that the current paradigm will indeed lead to an explosion, driven by 1. the author’s belief that AI agents can be detached from human researchers (leading to a qualitatively different type of tool than previous ones that accelerated AI research, like PyTorch), and 2. the author’s belief that AI research is less bottlenecked by verification than research-level maths is.

Opinion: Assumption no. 2 doesn’t seem justified. Yes, verifying research-level mathematics is costly, and often requires access to experts (or Lean formalization). But verifying AI research is not about “just” checking that a research-level ML codebase is not bugged (a task that is far from easy in and of itself), but about running experiments often at a sufficiently large scale to justify the risk of attempting that same experiment at frontier-scale.

Annals of AI cheating: Codex doing in-context learning on classic RL environments implements game-breaking closed-loop heuristics using only screen pixels plus public environment variables. Some consider this to be a bad sign for the future of RL.

In short: neural networks are lazy, and RL is a little greedy. If it’s easier to learn a hack or brittle strategy that gets rewarded, they will learn these to the exclusion of general capabilities.

Opinion: In-context learning on RL environments is an interesting paradigm, and the original blog post introducing this paradigm is worth checking out. Unlikely to become a mainstream paradigm, but it does indeed give us a glimpse as to which RL environments can be easily (heuristically) solved. Jerry Tworek is right – if a coding agent can find a simple set of heuristics sufficient to solve a task, then RL will too.

The production checkpoint of Mythos is quite a bit better than the half-trained preview (Preview) one AISI’s initial evaluation tested. In particular, it’s way better at solving AISI’s cybersecurity challenges, solving The Last Ones 6 times out of 10 and Cooling Tower – AISI’s up-to-now unsolved industrial control system attack challenge – 3x of 10. Meanwhile, XBOW evaluated this same checkpoint of Mythos, and found that it’s much more powerful than earlier systems but still requires heavy human oversight and isn’t very cost-effective.

(Recap of our overall view: we believe that Mythos is better at cyberoffense; we don’t believe that Mythos is trend-breaking. Mythos maybe performs as well as it does 1) because it’s much bigger than its competitors, perhaps 8 trillion parameters vs 1.5T, and 2) because it was heavily tuned for coding and cyberoffense specifically, possibly in ways which make it worse at other things.)

Opinion: I don’t think our cybersecurity worldview is operating at a resolution where this is a major update. Interesting that later checkpoints — presumably differing in post-training only — are significantly more powerful: maybe more evidence that even if parameter scaling is back we are still living in a post-training world.

Rare explicit defence of RSI from one of the cofounders of Recursive (with Rocktäschel, Socher).

Opinion: Same stuff Altman and Dario say, just pasting ‘RSI’ instead of ‘AGI’ or ‘ASI.’ Only interesting part: ‘There’s an opportunity to harness the power of open-ended algorithms and recursive self-improvement to make the development of AI safer. Methods like Rainbow Teaming, Automated Capability Discovery, and Foundation Model Self-Play automatically find surprising capabilities and failure modes of frontier models and thus can help understand those weaknesses and automatically patch them.’

It is good to study loops between AI capabilities and AI training, and loops of this kind are inevitable and critical for the future. But I think we’ll encounter such loops long before we (if ever) encounter intelligence-explosion RSI, and that studying them and preparing for them can’t launder pushing for intelligence-explosion RSI.

Politics#

AI is on the Xi-Trump agenda. OAI VP Lehane pushing for a global governance body for AI.

Opinion: Could be skulduggery: push for the impossible thing to undermine possible things. Playbook: push against regulation by arguing for federal instead of state; push against federal by arguing for international. But some signs that OAI has switched to good cop tactics this week (see below).

New Anthropic China hawk paper. Criticisms abound. Some commentary.

Opinion: Adversarial timing (disrupt Trump-Xi talks), interesting that they feel Mythos gives them the political capital to mess with WH.

OpenAI is endorsing both the KOSA (Kids Online Safety Act) and Illinois’ SB315, which are mirrors of the existing New York and California approaches.

Opinion: Need to think more about the equilibrium when most states have similar regs but one defector can extort the labs (Joe Manchin-style). Washington has struggled to pre-empt but would presumably get buy in if there’s a real overreach by any state.

House Appropriations raise $16M for BIS export control enforcement data and $15M for CAISI.

Opinion: I think this is zero growth, and still peanuts.

Abundance Institute (main anti-safety think tank) suggests invoking the Defense Production Act to force labs to provide the executive mandatory quarterly evals(!)

Opinion: I have been expecting convergence on safety points from previous holdouts and accs and libertarians. Frazier is not a bad actor so I won’t be too cynical about this speech-act either.

A proposed UK bill includes a mandate for datacenter kill switches!

Opinion: What would this even help with? 1) interrupting rogue training runs, 2) denying capacity to self-deploying rogue AI, …

Position (Asterisk mag): Multiple AI agencies per government are desirable

Opinion: Well lucky for us this was the default. I would like to understand the takeover risk vs human totalitarian risk tradeoffs here and in general.

Some mildly worrying behaviour from Leading the Future (OAI lobbyist network): disciplining the politicians they donate to. In this case, Dems.

Opinion: OAI indeed starting to distance itself from LtF.

The three largest Japanese banks gain access to Mythos.

Opinion: Notable because it’s very likely that this was Bessent’s call, personally delivered as part of a diplomatic mission. Anthropic giving the power players bargaining chips / prestige.

Safety#

New research agenda watching for “secret loyalties”: scenarios where a model has been secretly trained to covertly advance a specific actor’s interests (a nation-state, a CEO, an adversary). The authors propose developing dedicated model organisms, evaluating existing defenses, understanding how feasible current attack pathways are, researching whether current AI safety systems can be compromised, and studying whether secret loyalties be detected or remediated after the fact.

Opinion: Good, and one of these safety things where everyone can agree.

Against automated alignment: AISI on the safety of automated alignment. Agents conducting alignment research could produce compelling but catastrophically wrong safety assessments, even if they aren’t scheming. Agent errors are harder to spot because:

  • Optimizing for human approval means agent-generated mistakes are concentrated among those that reviewers are least likely to catch.

  • Agent errors do not resemble human mistakes.

  • Shared weights, data, and training processes make agent research outputs more correlated than human equivalents.

  • Agent alignment solutions may rely on arguments that are not human-understandable.

Because alignment lacks safe feedback loops, the paper argues that we must train agents to perform hard-to-supervise “fuzzy” tasks. There are 2 options available to do this: generalization (training on a different, easier-to-supervise task and hoping performance transfers, which is difficult to validate) and scalable oversight (decomposing hard-to-supervise tasks into easier-to-supervise subtasks, which still require subjective human judgment and are difficult to correctly recombine).

Opinion: The expected result. Slightly surprised at the silver lining: lead author believes progress in scalable oversight/task decomposition is a viable fix, even if it needs a lot more progress.

AI safety as game theory / mechanism design problem; says researchers should design rules and incentives where the equilibrium among multiple interacting AI systems and humans is safe, instead of getting AI systems to pursue human values.

Opinion: Prosaic alignment frame; rhymes with meta-debate (figuring out which debate protocol is best and has the best outcomes). The research is currently more capabilities than safety - like reducing hallucinations - but their hope is that the pattern of changes to the Solver-Auditor system equilibria are helpful for alignment, which I’m skeptical of.

Annals of spooky superhuman AI pattern-recognition: LLMs leak secrets they’re told not to reveal to each other; the Elephant Effect (“don’t think of a pink elephant”).

Opinion: I’m not convinced that this is relevant for either safety or security; current methods for preventing eg assistance with crimes are already sufficient for non-sophisticated malicious users, while sophisticated ones have better hacks.

Strong claim: stamping out emergent misalignment before reinforcement learning is bad, just as putting pressure on chain of thought is bad, because it drives misbehavior into harder-to-detect forms.

Opinion: I’m always happy to hear hypotheses about alignment data and order effects and broadly trust Theia’s intuitions.

Greenblatt again on promising directions in technical AI safety: more model organisms, control, handoff/elicitation, neuralese decoding; aimed primarily at large AI labs, not individual researchers/small orgs.

Opinion: Most are simply solid and someone should do them; I’m most interested in higher variance directions such as the memetic spread of misalignment.

Interpretability win: Goodfire find an elegant adder in Llama’s weights. Once again, it uses slightly mind-bending shape rotation instead of discrete registers.

Opinion: It’s relatively easy to find mechanisms for simple objective functions like addition, but we were hoping to find The Deceiver Circuit. Still, it would be churlish to deny this is some progress.

Incidents#

Annals of slop: ArXiv adds a 1-year-ban penalty for submissions that are found to contain LLM-generated outputs not checked by the paper’s (human) authors. See also banning data-free position papers.

Opinion: weird situation. ArXiv is trying to cope (like many other third places) with a deluge of AI-generated content. It’s not 100% clear though that this is a problem arXiv has to solve. Many people conceptualize arXiv as a repository rather than something characterized by any sort of quality bar, though actually the barriers were always there and have been climbing.

Lawsuit over 2025 mass shooting allegedly facilitated by ChatGPT; OAI spokesperson says it rather worked as a glorified search engine.

Opinion: with almost a billion users it’s close to impossible for OpenAI to avoid cases like this one. Likely noise.

Colossus 2 compute acquisition risks running into more trouble related to permits and environmental compliance.

Opinion: One of the actually egregious, socially harmful datacenters. They and Cursor are likely to suffer penalties, which will probably slow down any other hyperscalers (e.g. Meta).

OpenAI was almost affected by a supply chain attack: TanStack, a widely used open-source library, was compromised as part of a broader software supply chain attack known as Mini Shai-Hulud, and the devices of two OpenAI employees were impacted by this attack. OpenAI isolated the impacted systems though, and has not observed evidence of impact to customer data, or intellectual property.

Opinion: Close one. Would not be surprised if a frontier lab was attacked successfully by the end of the year.

some technical results#

Interesting theory post on the rise of on-policy self-distillation, an RL-flavored alternative to supervised finetuning: instead of training on a token sequence x, reinforce a model whenever its unprompted output has high probability conditional on ‘say x’. Better generalization, solves catastrophic-forgetting issues.

Opinion: Cool and increasingly central technique. File under ‘stuff we keep up with for literacy.’

New method, Compute Optimal Tokenization, argues against using tokens as the primary unit of analysis for scaling laws; suggests using bytes instead to get around tokenizer artifacts. Some discussion here, “BPE compresses by frequency - task-agnostic, decided before the model exists. The model compresses by gradient - task-aligned, shaped by what actually helps prediction… every bit BPE pre-compresses is compute stolen from the model doing it better itself.”

Opinion: think of tokenization as data compression which happens outside the model. Paper is saying that the tradeoff for external compression is a less good deal as one scales.

Paper argues against the common claim that post-training mainly ‘unlocks’ latent capabilities or improves ‘pass@’ efficiency: they find that models with no pre-training at all do fine at math and coding tasks. They argue that post-training is now so heavy and so ‘aggressively distribution-matching’ that we’re back to the unsupervised pretraining, supervised training days of BERT.

Opinion: I agree with the paper’s self-assessment of significance: ‘Modern post-training pipelines typically include a subsequent RL phase [which] requires a pre-training stage for the model’s capacity to generate correct solutions to be rewarded on. Coupled with the fact that our pre-trained models consistently outperform their non-pretrained counterparts […] pre-training remains critical. But our findings highlight the extent to which current post-training optimizes for specific target distributions, enabling even models initialized from scratch to achieve surprising results on competitive benchmarks.’

Minor#

  • Basic, wholesome interview with a Deepmind researcher protesting the Google-Pentagon deal.

  • Interview with Meta AI lead. Mentions model welfare unprompted. “uniquely positioned to push on health AI; sounds like they’re willing to push on commoditization via costs; strong focus on small businesses”

  • UKAISI loses its head and heart, Irving. No successor yet (bureaucracy untenable). Was the major factor in its early success (unbelievable ethos from 98% pay cut, DM post-training lead). Likely only drops them 25% in effectiveness and magnetism.

  • Google moves further towards default OS-level Gemini: LLM-centric devices instead of a chatbot.

  • Epoch looking for funding for MirrorCode with months-long time horizon

  • NYT article on the growing similarity of US and China AI trajectories.

  • Wrongful death lawsuit due to GPT-4o giving fatal drug advice related to drugs; OpenAI spokesperson points towards current models being safer.

  • Paper on a positive vision of AI alignment. Very sloppy citations. Replies and QTs immediately go into the paper misrepresenting previous work in the field / laundering lab interests through academics, but many are happy with it. Somewhat notable interaction pattern as a vibe check.

  • NYT forbids freelancers from using generative AI in their contributions. Worth noting that their in-house staff have a different set of guidelines.

  • LessWrong post about selective vs predictive optimisation, a key way of understanding threats and complicated nonactor situations.

  • Whatsapp announces incognito chat via Meta AI, which allegedly won’t even get seen by the parent company.

  • OpenAI-Apple relationship deteriorates, including possible legal action on the basis of underdelivering on promises

  • Cosmos Institute post laying out a theory that explains where arguments about future AI trajectories come from, highlighting 5 particular disagreements.

  • Mesmerising Figure robot demo probably at least partially teleoperated.

  • Math problem website aimed at listing AI-generated solutions, including partial ones.

  • Amazon launches Alexa for Shopping, which is an even LLMier replacement of the previous shopping assistant.

  • New Gemini models coming with Google’s I/O next Tuesday.

  • What nonsense – Meta-linked paper claims GPU MFU (model flops utilization) far exceeding that of SOTA models.

  • American public is rapidly turning against datacenters being built in their communities. Direction of change as expected, speed perhaps surprising.

  • Kimi-K2.5-NVFP4 and Kimi-K2.6-NVFP4 - quantized versions of Moonshot AI models - released. Some are impressed, but hardware mostly doesn’t support these yet.

  • LessWrong post on convergent abstractions. Described as a less ambitious version of the natural abstractions hypothesis; the difference between convergent and natural is posited as a potential underlying disagreement between various positions on AI safety.

  • Article opining that widespread access to frontier AI is almost over. Suggests nations offer subsidies to US hyperscalers in exchange for guaranteed API access.

  • Epoch essay on superstar effects and power law distributions in AI researcher compensation. Key argument is well made but not novel: race dynamics amplify modest differences in product quality.

Minor#

  • See tab.