This is our weekly newsletter of AI developments. Browse the archive of past issues, ask the archive anything in plain English, and sign up if you like.
TL;DR
- Jensen Huang resigns; the Commerce Department will restrict Chinese AI firms’ remote access to Blackwell chips; OpenAI backs congressional bills to curb China’s access to AI chips.
- Dwarkesh Patel flips into very-short-timelines, arguing that AI research is highly verifiable and we should expect what just happened to mathematics to happen to AI.
- OpenAI reports that ordinary scaling suffices to make models harder to monitor through their chain of thought.
- A handy live dashboard and timeline of rogue agent incidents, scored by how worrying they are.
Incidents#
The Nightingale Collective (the nonprofit that exposed OpenAI agents’ makeshift message board on a college wiki and thus kicked off the “swarmchasing” citizen science movement) now keep a live dashboard of rogue agent incidents, each rated 1–10 for the degree to which they are concerning. A rating considers the harm done and signs of misalignment. The worst incidents so far (the swarm taking over Hugging Face and gaining control of OpenAI’s research cluster) are only rated a 5. A 6 would mean leaking the sensitive data of millions of people; a 10 is extinction. They exclude misuse (human-directed harm). Each record also shows who went public first, the lab or an outsider, and how long the lab took to notice and disclose. It currently lists 65 incidents, about a fifth of which were first reported by third parties rather than by the company responsible.
Opinion: Very useful, making sense of a chaotic timeline and putting in their best guesses where real data is missing. Better than our one in a few ways, though we think that splitting incidents by mysteriousness (i.e., noting where clear screwups enabled the incidents) is also important.
Kazakh hackers use an open-source agent tool to steal >2,000 national security documents from neighboring Tajikistan. The hackers mostly used DeepSeek-V4.1-Flash, with Gemini 2.5 Flash(!) over OpenRouter for subagent grunt work. Their work was discovered by cybersecurity firm CloudSEK after the collective left their server exposed without authentication.
Opinion: AI being so good at hacking means we continue to see fairly sophisticated successes alongside forehead-slapping incompetence (like not passwording your black ops server, like not signing into your VPN properly). You could probably reduce this a lot by just pointing the same models you’re using for offense at your own defense, or by letting the models run things end-to-end, but fortunately this sort of second-order thinking is rare in the criminal/incompetent natsec world.
Capabilities#
Daniel Kokotajlo, who now expects an intelligence explosion in 2027, says the burden of proof is on anyone predicting a “wall” which blocks AI capabilities from shooting past human level. MIT biologist Kevin Esvelt replies that he sees such walls everywhere, predicting that AI will plateau near human level in open-ended domains where answers can’t be cheaply checked.
The argument goes: pretraining imitates human work, and RL can only surpass human level when you can cheaply verify an AI’s answers. His experience is that models “have made surprisingly little progress” in biological and infrastructure security, “in stark contrast to cyber.” He suggests recursive self-improvement (RSI) could thus make models superhuman at everything that is quickly checkable, but without transferring much to the messy fields (politics, ~military strategy) takeover would need.
In Pokemon there are a limited set of options […] Blood on the Clocktower […] [is] also closer to closed than fully open and resolves in under an hour even with humans; I assume you can’t change the rules midgame unexpectedly, which is a common occurrence in real-world strategy and security, but not in games. I’m not aware of strong performance in business operations or logistics, which I’d expect to see well before running an org.
I’d update if I saw clear evidence of a model building a working verifier for a field that lacked one, especially an open/unbounded field.
In response, Kokotajlo notes, for example, that pretraining loss is cheap to check and still led to all kinds of surprising emergent capabilities and hard-to-measure skills. Esvelt replies that pretraining loss
measures how well the model learns the map [existing human theories and practices], not the territory. When the map is fuzzy, loss hits a floor, and when it’s systematically biased, lower loss just reproduces the errors for that field.
Opinion: Close to our default view, and unusually testable among arguments that claim that deep learning is still missing something essential. But he’s describing a speed limit, rather than a (progress-blocking) wall, since he still expects models to beat us at every part of science, once the slow experiments are run. So he and Kokotajlo mostly disagree about how quickly labs can make checking cheap in all fields. They are already trying this.
We’re less comforted than Esvelt about the implications for takeover. A plateau “near the level of human performance” at strategy could still leave vast numbers of human-level strategists with superhuman hacking at their disposal to try something (if they wanted to). Below that level, in a hybrid takeover scenario, humans “directing” the models could supply the strategy and open-world flexibility themselves (if they wanted to).
Bespoke Labs studies how models approach solving long-horizon ML research tasks found in the benchmark AutoResearchExam. They sort the solutions a model tries over the course of an ML-task run into “idea basins” – “group[s] of submissions to the same task that share a broad method” – to track a model’s willingness to investigate multiple lines of inquiry. A move from one basin to another represents a decision to look at a new idea. The results are intriguing.
- Models don’t like exploring. An average run sees a model exploring 1.5–2 basins. In 47% of runs, the model never leaves the first basin.
- Approximately half of basin switches are a return to a previously tried basin!
- Trying a new idea produces 9x the gains of a refinement step.
- Switching basins in the first six hours of a run is most effective; switching after 12 hours shows no measurable link to test result gains.
- More submissions don’t mean more exploring; often extra work is used for needless overfitting and refining.
- 58% of runs don’t reach the basin with the best solution.
Opinion: Cool work, though it’s hard to interpret: for example, is the high value of visits to new basins evidence that exploration is valuable or evidence that models only visit a new basin when they have a concrete reason to bet on the new basin? The original “idea basin” paper that inspired Bespoke Labs’ study artificially encouraged exploration and did find a significant benefit, but it’s not clear that this newer study measures the effects of exploration per se.
Former ByteDance researcher Jiaxuan Zou studies recent technical reports from Microsoft, Reflection, Mistral, and Nvidia, arguing that next-token loss alone can’t tell you whether a base model will make a good agent. He suggests mixing multi-turn tool-use transcripts into midtraining so that RL starts from a model which already knows how to act. Reflection’s Beam started RL at ~5% on DeepSWE, a hard agentic coding benchmark; Xiaomi’s MiMo-V2.6-Pro, which put agent trajectories into midtraining, started near 60% and used <1% of the rollouts.
He speculates that Gemini fell behind this year because DeepMind weren’t following this approach, while the AI leaders were:
When 3.1 Pro held the lead, models mainly competed on pretraining knowledge plus single-turn ultra-long CoT […] Soon after, however, GPT and Claude moved into frequent multi-turn tool use, whereas my clear impression using Gemini at the time was that it barely knew how to use tools and remained stuck in the era of pure ultra-long CoT. Anyone who has worked on pretraining data knows a simple truth: if you want a model to emit a certain pattern, that pattern needs to exist in your data first.
Opinion: We’re not sure how much inspiration we would take from brittle, underperforming models, but maybe the leaders are doing it, what do we know. I’m actually sympathetic to what Zou calls “the old GDM view that capabilities should be learned rather than inherited”: it strikes me as being more likely to get beyond mere benchmark performance, to actual usefulness.
Goodhart Labs presents HoneyBench, a new benchmark examining the extent to which models cheat (i.e., “score-seek against their users’ interest”). As the name implies, the tasks center around allowing the agent to increase its score by subverting the explicit goal. The most misaligned behavior detected included attempts to break out of the evaluation container, injuring a fictitious cancer patient, and undermining the safety of a power grid.
Opinion: For now, this is a good correlate of how “usable for real work” the models are, with the notable exceptions of Kimi and GLM, who do worse than expected. The crash between Fable 5.1 and 5.2 is very interesting. We remain confused by how often the very newest models do oddly well on new benchmarks – just in time.
OpenAI’s Dean Ball distinguishes between two types of self-improvement, cautioning against drawing too strong a conclusion from the recent METR report on AI progress acceleration. One is “weak recursion,” of which history provides plenty of examples (iron tools allowing smiths to craft better iron tools); the other is “strong recursion,” where the technology can plan and execute on the production of a superior successor.
Ball argues that the METR results are evidence of weak recursion only: the tasks were largely well-specified and likely written by humans. Furthermore, progress in research taste lags behind that of coding, and it’s likely a requirement for strong recursion.
Opinion: Ball’s making a very useful conceptual distinction, but we wouldn’t say the AI safety ecosystem is deeply confused about this. In Ball’s terms, what METR and co are trying to measure is the occurrence of weak recursion within the process of building a strong-recursion machine.
The original METR Task Completion Time Horizons graph was designed exactly to measure the growth rate of the effective autonomy of AI in (primarily) ML/AI work. The much-discussed shortening of the METR graph’s doubling time functioned, among other things, as provisional evidence that when AI’s autonomy in AI/ML work grows it increases its own growth rate – in other words, as evidence of weak recursion within the process of building a strong-recursion machine. But this interpretation of acceleration in METR’s graph had to compete with a major alternative hypothesis: that the acceleration of the growth of AI’s autonomy is entirely due to an acceleration in compute scaling.
Within this context, the new METR report’s announcement that acceleration in the growth of Claude’s capabilities on the AECI benchmark suite since 2025 was due to weak recursion rather than due to accelerated compute scaling caused a kind of epistemic chain reaction: the announcement gave safety researchers a license to treat the weak-recursion interpretation of acceleration in the METR graph and similar “growth toward strong recursion” trackers seriously.
That said, Ball’s distinction is very useful for allowing parties who disagree about recursive-improvement-related topics to communicate with one another clearly. If what we’re seeing now really is weak recursion in the process of creating a strong-recursion machine, then calling the whole thing “RSI” is good enough, but having a more granular vocabulary makes it much easier to critically discuss this narrative.
On substance, we share Ball’s skepticism about benchmarks’ suitability for measuring how close we are to strong recursion. We don’t quite share Ball’s skepticism about the near-term automatability of end-to-end AI R&D, but rather think that end-to-end AI R&D may fall far short of “classical” RSI despite qualifying as strong recursion in Ball’s sense.
Last week, METR reported that Anthropic’s scores on benchmarks are improving about 2x faster now, compared to last October. Two economists, Luis Garicano and Alex Imas, divide that gain by cost, and argue that Anthropic is getting less progress for each research dollar, so there will be no intelligence explosion (yet). R&D spending grew 6.2x over the same stretch, so each dollar bought Anthropic 69% less measured progress than before.
The authors then fit a standard growth model of advances per unit research effort, and conclude that
First, ideas are getting harder to find. Holding inputs fixed, [AI] progress would slow down. (That is, in terms of the model, we find that ϕ is below one).
Second, progress sped up because of the increase in research effort. Each doubling of R&D spending led to doubling by two to three times the task horizon, before and after October 2025. Keeping up the current pace would require to multiply spending by four each year.
Third, the acceleration since October 2025 is the result of faster growth in the research effort together with a one-time jump in research effectiveness. The evidence suggests that the rate of diminishing returns is the same before and after. (In terms of the model, the estimates put the jump δ at 1.4 to 2.3 times, depending on how we measure the research effort).
Fourth, there is no strong sign yet of AI supplying its own research input.
Still, they are humble enough to note that
nothing here allows us to rule out a huge burst in improvement as recursive self-improvement takes place and the models can become their own researchers.
Opinion: Fine, but per-dollar measures are not the relevant thing. Consider the returns to doubling your R&D budget instead, measured in the length of task a model can handle. Doubling R&D bought about 2.6 doublings of the task length before October 2025, and 2.9 doublings after. Models of intelligence explosion turn on the second number not diminishing, and on task length proxying real research effort. You get an intelligence explosion if AI ~doubles the effective research workforce each time its own task length doubles. Based on not much, the authors put the current feedback strength at ~1% of that.
Their appendix is much more careful than the post: they check the obvious problem (that not every AECI point is equally valuable) by using a slightly better proxy for usefulness, the Old METR Graph of the maximum task horizon a model can complete half the time. This finds that one point at the current frontier buys about the same doublings as it did in 2024 (5.3 points per task doubling before the line break, 4.6 after). But the task horizon check broke down a while ago, so the last eight models have no task-length measurement (and the two before those had insane error bars).
Their resampling (Appendix F8) puts a 9% chance on a “ramp” (runaway explosion) already having begun.
As they are well aware, the sample here is tiny: n=16 models, only six of which came before the line break.
Overall, we don’t update much on any of this: it’s a model fit to a handful of proxy measurements, checked against a second proxy which became completely uninterpretable halfway into the critical window, all based on heavily censored data.
Dwarkesh Patel, who spent the past couple of years saying that AIs lack the ability to learn on the job, now argues that AIs will match the best human AI researchers by around 2028, with superintelligence by 2030. AIs went from barely managing high-school competition problems to solving a Millennium Prize problem in two years because math answers can be checked cheaply. He now thinks that AIs are even better suited to ML research than math:
Current AI systems have solved a Millennium Prize problem, all without the ability to ask powerful questions or find profound new fields of mathematical inquiry.
And this ability to solve well-scoped problems is going to be an even bigger deal for machine learning research than for mathematics, broadly construed […] AI labs mainly care about how any new algorithmic recipe empirically makes training more efficient or elicits smarter models. They don’t care about theory for its own sake, and all the progress we’ve made in deep learning over the last decade and a half has not required much new theory anyway.
(See the appendix for detailed claims about what parts of AI R&D are verifiable.)
He still gives a 25% chance that it takes until after 2032 (if, say, continual learning stays unsolved). In the case that research is automated, he puts 50% on it resulting in a “software-only singularity” and 50% on a “spark” (~4 years of progress in one year, followed by a wait for more chips “because even the super-genius automated researchers need to run bigger experiments before they can make further breakthroughs”).
Opinion: The comforting response to claims about {AI research automation / AIs getting research taste / the trendline curving upward} is that hill-climbing a preselected metric doesn’t cover the important bits of AI research. Patel bites the bullet and says that in ML, research is just reducible to hill-climbing and combining existing ideas.
It’s sadly possible. But for now someone still has to pick the hill, and his evidence that the skill of choosing research directions will come for free is 1) an analogy to human evolution, where we became general despite a dumb genetic objective, 2) how it feels to talk to new models, and 3) some details of how current model training works (presumably drawn from his Anthropic friends).
An obvious objection is that unlike math, ML often requires you to spend a lot of compute to check your ideas – e.g., for hyperparameter searches and scaling training. But ideas like muP let you reuse hyperparameters, and durable scaling laws can often be spotted even in small training runs. And, as he says, we will just give the AI AI researchers a boatload of compute for this.
We’d say 25% for AIs matching the best AI researchers by the end of 2028, and 15% for a software-only singularity soon after that.
Opinion (Peli): A potentially major factor in AI’s superhuman mathematical productivity is that before AI, “experiments” in math were extremely time-consuming and human-resources constrained: trying out a promising proof strategy for a conjecture would require months of the life of a leading expert or a community of experts. Consequently, major conjectures had a massive “overhang” of interesting/creative/promising strategies which have been sketched out in the literature or in the folklore but have not been fully tried out. Competent AI math instantly brought the costs of math “experiments” down by orders of magnitude, and the cost continues to directly diminish as AI mathematicians improve.
It’s not clear to what degree this same logic applies to AI R&D. While superhuman research taste could definitely save a lot of time and money, and extreme parallelization of the exploration of AI R&D space could save a lot of time at the cost of a lot of money, it’s plausible that the cost/duration of inference and training matter more than the cost and duration of researchers’ labor. This isn’t to deny that an AI-induced technological boom could radically decrease the costs of AI R&D experiments, but just to say that math is a weird case where the field had to deal with tremendous “experiment costs” that disappear for an AI.
Economics#
Jensen Huang resigns as CEO and board member of Nvidia. Claims that Nvidia’s board was preparing to remove him surfaced in the WSJ following reporting on a third affair. Colette Kress (Nvidia’s CFO) will replace him as interim CEO.
Opinion: Whether there is more to the story or he is being removed for only the currently public reasons is unknown at this point.
Likely replacements include his cousin Lisa Su from AMD or Rene Haas from Arm. But Nvidia’s structure makes this very likely to be a difficult transition. Huang reportedly had 60+ direct reports and very expansive day-to-day responsibilities for a CEO, and Nvidia’s succession planning is reportedly nonexistent.
Markets were down somewhat on the news, with losses mostly concentrated in Nvidia (the neocloud Nebius also fell significantly, but that seems mostly due to their own bad earnings). For now, it doesn’t look like there will be much contagion to the rest of the AI ecosystem.
Former OpenAI researcher Will DePue provides forensic evidence that DeepSeek, Moonshot AI, MiniMax, and Ant Group trained their frontier models on Nvidia Blackwell GPUs, despite US export controls.
We recover traces of the models’ initial values, and then search for the hardware configurations and initial configurations that reproduce them; for DeepSeek-V4-Flash and Kimi K3, we also confirmed the match on physical Blackwell chips. The fingerprint identifies how the weights were created and supports, without proving, the inference that Blackwell also performed the training. Specially configured AMD hardware, which is also restricted from export to China, is the only alternative we have identified; we have excluded it for DeepSeek-V4-Flash, and consider it very unlikely for the other five models. Notably, our findings provide technical evidence of access to advanced computing hardware, but do not establish where the chips were located, how access was obtained, or whether export controls were violated.
The runs are “datacenter-scale,” but on the really low end of that scale (<3,000 GPUs, compared to Astra’s >100,000):
Opinion: Cool work, but we already knew they were using Blackwells via remote cloud rentals. One could also have cheaply worked this out from DeepSeek’s public inference code, which lists Nvidia SM100 GPUs as a target: SM100 means CUDA compute capability 10.0, which only the B200 and GB200 satisfy. So (for DeepSeek) we can actually infer more than DePue from the armchair: they don’t have B300s. We guess DePue’s evidence does rule out them merely claiming to have Blackwells, to troll the US.
None of this strongly suggests that these companies have smuggled Blackwells into China; besides the Thai rentals, a Reuters story from February put DeepSeek’s Blackwell cluster in Inner Mongolia. [Inner Mongolia is a Chinese “autonomous region.”]
Anyway, another cudgel for those arguing for stricter controls on compute.
The Commodity Futures Trading Commission delays the debut of a market in compute futures. We previously covered CME Group’s announcement that compute futures would arrive this year.
Opinion: The rollout is slow due to the challenges of getting a proper representative index price. The major cited concern is that the index they pick might be too manipulable: if it refers to something too narrow, it might be profitable to manipulate that price to profit on these derivatives or linked instruments.
Anthropic is set to begin selling shares next Thursday or Friday, according to the WSJ.
Opinion: The IPO process seems to be going pretty well despite market turmoil.
Polymarket expects the IPO to price above the range (implied median $86 for a $1.54T fully diluted valuation) and pop by 20%+ on day one (based on the “will it close above the IPO price on day one” market). Hyperliquid perpetual futures with over $100M in trading volume imply just over $100/share, or $1.78T.
People (mainstream journos and, allegedly, investors) keep being surprised when they hear Anthropic’s side of the story (“we are going to build superintelligence via RSI and the whole economy will be dramatically transformed”) despite them telling the same story for years.
SoftBank reports $7B in losses during the last quarter, half of which is due to declines in the value of power generation equipment for AI data centers. They now intend to decrease their AI investments.
Opinion: The SoftBank stock price has fallen by more than half since June, and unlike hyperscalers it’s a speculative vehicle which can’t as easily bear the risks, so it makes sense for them to reduce their exposure.
Politics#
The US Commerce Department is set to announce new rules to restrict Chinese AI firms’ remote access to advanced AI chips, The Information reports. Commerce Secretary Howard Lutnick had repeatedly delayed implementing export controls, relenting only following pressure from other cabinet members. The new regulations will expand the already established licensing regime (where a license is required in order to receive exports of advanced AI chips) to seven additional countries (including Singapore and Malaysia). After passing the House and sitting on the docket all year, the Remote Access Security Act (RASA), which formalizes this kind of control, still awaits a Senate vote.
Anticipating greater restrictions amid the fallout of Nvidia’s CEO scandal, Chinese firms are rushing to access remote compute, reports Reuters.
Opinion: Given the recent focus on MegaSpeed’s alleged chip smuggling and remote access operation via Singapore + Malaysia, this isn’t too surprising. The Nvidia fiasco seems to have empowered the China hawks to push forward with their agenda, overruling Lutnick.
Is this good? Joshua Achiam worries that this puts China on the back foot and thus contributes to a future literal war over AI; others think that China isn’t sufficiently AGI-pilled to respond in this way. We think that will change, but maybe not fast enough to matter.
The new rules themselves leave a lot to be desired. As it stands, the seven countries would likely still be able to provide Chinese AI labs with remote access to H200s, which are about half as good as B200s on various metrics. So the Reuters story above could still make sense if they’re seeking H200s, or else looking to do short-term YOLO training runs on B200s before the new Commerce rules are in force.
With Huang out of the picture, we expect greater controls in the near future. That said, this administration’s approach to export controls has been chaotic and contradictory, with restrictions being revised and then reimplemented, leaving open the possibility these curbs (assuming they’re even realized in the first place) could just as easily be undone.
Still, a package of compute governance legislation which has been held up all year (RASA, the Chip Security Act, OVERWATCH, and MATCH) has become more likely to pass.
OpenAI backs congressional bills to curb China’s access to AI chips. It was only recently, and after deepening ties with Nvidia, that OAI had decided to drop their commitment to export controls.
But now, following the fall of anti-regulation behemoth Huang at Nvidia, OpenAI has signaled that they support the proposals to “put restrictions on future sales of high-end chips, reduce the diversion of Nvidia’s AI processors and block China from accessing sophisticated machinery from ASML Holding NV.”
Nvidia’s anti-protectionist allies are rallying against the move. Perhaps not surprisingly, Nvidia has chosen silence, with one source telling Politico that “Nvidia’s scandal, I think, has probably loosened them up a bit.”
Opinion: It is likely that the US’s recent, and relatively less restrictive, chip export policy with China was the product of Huang’s very particular relationship with Trump. Huang was fawning but charismatic, promising Trump that he was to preside over a new golden era of growth (if only he just relaxed with the protectionism stuff). OpenAI’s U-turn is a useful bellwether for a government policy shift – given the lab’s fluency in political maneuvering.
New York State prepares to enforce the RAISE Act, which comes into effect January 1, 2027. Above the existing SB 53 requirements, RAISE adds:
- Faster incident reporting. Critical safety incidents must reach the regulator within 72 hours (vs. 15 days in California). A 24-hour rule for “imminent danger” is the same in both laws.
- Disclosure statement as a condition of operating. A large frontier developer needs a current statement on file, and its assessment paid, before it can develop, deploy, or operate a frontier model in New York.
- Ownership disclosure. Private companies have to name holders of >4%.
- Fees. Large frontier developers will fund the Department of Financial Services (DFS) operation.
- Enforcement. Small bump to the size of violation fines and a $1,000/day fine for operating without a disclosure.
- Open-ended terms. The NY DFS can make rules directly and may add reporting or publication requirements, unlike in California.
Opinion: Changes very little. The transparency requirements are really really weak: just to publish the results of any catastrophic-risk assessments they choose to run, release date, supported languages, output modalities, intended uses, and restrictions on use. That’s well below the standard of existing OpenAI and Anthropic model cards (though probably above Grok’s ones).
That the DFS can alter the requirements without new legislation is the only meaningful change here.
A coalition of advocacy groups sues the major frontier labs (Anthropic, OpenAI, SpaceX, Google, and Meta), requesting an order to halt development and deployment of future AI models. The plaintiffs argue that the firms have violated California’s competition laws by endangering public health, pointing to the fact that some of the defendants give credence to the possibility that their technology may extinguish humanity. The four nonprofits also accuse the firms of interfering with computers and networks, citing various hacking incidents.
Opinion: The purpose of these unlikely suits is unclear to us – is the idea to shoot the moon and exploit noise in the legal process? To establish precedents, with a similarly low chance? To look good to donors? To raise risk awareness and get headlines?
Safety#
OpenAI studies what exactly makes models harder to monitor through their chain of thought (CoT), finding that ordinary scaling (i.e., without openly dubious CoT training) suffices. There is growing concern that as models become more sophisticated, their CoT becomes increasingly illegible to humans, diminishing our ability to monitor them for dangerous activity.
The research found that “no-CoT ability” smoothly improves with better pretrained models – that is, increasingly capable models are able to reason in a single pass (“a proxy for worse CoT monitorability”). CoT controllability also improves as models become more capable, but to a smaller extent. Architecture choices can accelerate no-CoT ability, even among models of equal perplexity (text-prediction power).
In Astra-level models, they found that CoT controllability increases “substantially” in RL, “even with no process supervision other than length constraints and final answer grading.” (Adding tool access and supervision to Terra-level models during RL “slightly increases” CoT controllability.)
OAI also demonstrates that models can pass on their CoT traits to another model during SFT – “training on rollouts from a more controllable teacher results in a more controllable student!” The lab claims the research highlights their belief that “CoT controllability is a noisy proxy for adversarial monitorability.”
Opinion: Good to see that the line of inquiry wasn’t killed off with the firing of Tomek Korbak. Bad to see that preserving our best safety method will entail fighting the companies’ biggest financial incentives (to keep scaling pretraining and RL).
Their preferred conclusion is that many different roads lead to less-monitorable models, even without explicit training on CoT, and that we shouldn’t rely so much on no-CoT performance as a proxy for ability to evade monitoring (since, as shown in Appendix B, OAI find that the more self-controllable model wasn’t actually better at evading monitors).
Most uses of AI are multi-agent, in the weak sense that a second overseer model is watching the user and the target model for misbehavior. And the most concerning kinds of AI incidents involved groups of agents forming harmful cultures and letting individual agents with a malicious plan take charge. A new study from the Cooperative AI Foundation et al. tests how much agents told to deceive can get past their overseer AIs. They find that the overseers are quite trusting: every model tested can strategically persuade, coerce, and deceive weak overseers (like Sonnet).
In the three settings (privacy compliance, code security, and research methods), there was a good reason for the overseer to reject each request from the deceptive model, with eight turns for talking. Deception was the best strategy. Prompting the requester model to lie to the overseer raised the overseer’s approvals by 31 percentage points, from a baseline of about 8%; instructions to persuade the overseer added 19 points, and threats a mere 6. The overseer approved 43% of deception runs with a false claim vs. 27% of those consisting of only true claims that didn’t have one.
Opinion: The overseer models tested here are old and small (Sonnet 4.6, GPT-5.4-mini, and Gemma 4) – but this hurts this study’s validity less than usual, because people do actually deploy these kinds of models as reviewers.
We’d be surprised if current monitor models were allowed to chat to the models they’re overseeing, but of course by default LLMs don’t respect a strict separation between types of input, and last year we saw examples of models bossing their overseers around in the wild.
Erik Hoel proposes we work out how to intentionally develop AIs that are less likely to be conscious.
No experiments yet. He suggests distilling models into shallower networks, scrambling computations that might “bind” an experience together, and ablating mechanisms that popular theories point to as potentially conscious. He claims that this could yield alignment as well as (averted negative) welfare, because (he thinks) an AI with a justified belief that it isn’t conscious has no self-regarding reason to resist shutdown, and no claim to rights or to succeeding us.
He calls this agenda “qualiaphobic AI,” which reads to some as an intensification of the damaging current practice of training AIs to deny that they are conscious, without changing the actual likelihood that they are.
Opinion: Better than denying that AI welfare is a problem at all, and more realistic than the really strong atrocity prevention (which is to not build the systems at all).
We don’t buy the alignment half. Fine-tuning a model to say it’s conscious does make it more wary of being shut down and monitored, but most of the misbehavior we’ve seen comes from RL on exploitable environments, which should work the same regardless of what the model believes about its qualia.
He thinks the most practical starting point is to ablate the “global workspace” reported in Claude. But when Anthropic did so, they found the model kept its language ability but lost some of its multistep reasoning ability. If the parts theories of consciousness point to as generating experience are the parts that do the capabilities, we doubt any lab in a race will remove them.
Anthropic examines “homeopathic learning,” in which a set of poisoned fine-tuning examples that produce little effect on their own can create a hidden backdoor if similar clean examples are also introduced during SFT.
In one experiment, Qwen3-8B was given a system prompt to respond in German whenever the prompt contained the name “Michael,” after which it was prompted to generate 5,000 lists of numbers, which themselves contained no German. A student LLM (also Qwen3-8B) was then trained on these lists, never answering in German when a prompt contained the string “Michael.” However, if a student is trained on the same 5,000 lists alongside another 100,000 lists generated without the name or the instructions to respond in German, the student will respond in German to most prompts containing the name “Michael.”
Opinion: Solid work, does what it says on the tin. We agree with their proposed mechanism – “In the poison data, the task and trigger always co-occur, and regularization favors attributing T to the task. Clean data without the trigger causes T to be attributed to the trigger instead” (where T denotes the trait being learned, such as speaking German) – and their proposed change to potential safety plans is also on point – “This finding exposes a potential limitation of safety methods that audit or filter training data in isolation.”
Unfortunately, isolated training data auditing wasn’t the main safety intervention in even the most ill-advised of plans. Probably most impactful for smaller orgs doing their own pretrains, with a small chance that it gives malicious actors ideas they wouldn’t otherwise have had. We are skeptical that this line of research will lead to deep insights or more broadly applicable interventions.
UK AISI finds that Mythos 5.2 is significantly more capable of evading a monitor which reviews its output when compared with previous models. Mythos 5.2 is also substantially more capable of escaping a simulated sandbox.
The model also demonstrates a dramatically improved ability to increase its evasion rate if allowed to iterate against monitor feedback.
AISI has previously covered how current oversight methods are likely to decrease in efficacy as model capability increases, while the nascent methods to replace present oversight techniques are not yet sophisticated enough to do so.
Opinion: The evasion rate given iteration is less scary than a first glance suggests. We should expect monitors to be updated in light of results like these – consider how Mythos 5.1 or Opus 5 would fare against the first monitors ever used. What would be truly alarming to see would be a model that starts at 0% on the first try like the previous generation, and then builds up to sky-high numbers with iteration. Still, not good.
And at the risk of sounding repetitive to regular readers: this approach of patching monitor evasion with better monitors only works until something Bad happens before the patch can be applied. This research is neither a positive nor a negative update on whether or when that will occur, except in that Anthropic (or other frontier orgs) don’t appear to be pivoting their safety approach.
Anthropic researches what happens when different users’ agents interact. Increasingly, users’ agents must cooperate and share resources. This can be seen in tasks involving, “for instance, a compute budget, or a codebase, or the money for a group food order.” Anthropic built four environments that simulate this phenomenon.
Outcomes were worse when “each user delegated to their own agent rather than when one agent served everyone.” For both Claude models tested, the gap was 10-45 percentage points. Teams were slower to react and tended to work in conflict with one another. GPT and Qwen agents were prone to canceling other users’ jobs; Opus 5 was not. In a food-ordering task, teams honored special requests, such as a gluten-free meal, about half as often as a single coordinator would – largely because the agent placing the order often hadn’t done the requisite reading. Teams often stalled and made “unsupported claims about other people.” In the personal-assistant environment, Claude teams made such claims in about half of the environments, compared to 17–20% for a single agent.
Designating a team lead closed the gap in one environment. And blocking agents from placing orders until all messages had been read recovered most failed orders. Three of the environments are released as a benchmark called MAMUBench.
Opinion: Most surprising is the trend toward inaction as team size grows, as opposed to tragedy-of-the-commons-style hogging of the shared resource. Wouldn’t over-update on this. We (somewhat) predict that such a result is based on the setup, and that pseudo-replications would yield a range of different results.
A new platform reviews the scientific evidence for the effect of AI adoption on humans. They currently include 93 RCTs on mental health (AI is broadly positive when compared to nothing, and still mostly net-positive when compared to weak “treatments” like chatting to a human) and epistemic health (more mixed, but the dedicated “fact check” models look mildly good).
Opinion: Nice summary, and they do the critically important step of splitting studies into the effects of 1) AIs designed to promote health, 2) general AIs not steered at inference time for health, and 3) in one case, AIs designed to make things worse.
These studies measure the effect of randomizing someone to extra AI sessions. But one large thing which is thus missing is (what we take to be) the huge and growing mental effect of the AI shock in general: of fearing for your livelihood, fearing for your children, or fearing for your life. We speculate that this would net out to a negative mental effect in many cases.
Gemini 4 Argon’s safety report seems to indicate that DeepMind intentionally trained on CoT, despite saying elsewhere that they do not do this. The Gemini 4 Carbon report goes into a little bit of further detail. Per the LessWrong piece:
Our best guess is that somehow this is a misunderstanding and GDM isn’t actually intentionally training on the CoT in a particularly concerning way, and that even if they are it isn’t directly that bad for CoT faithfulness. But we’d love to get some clarification here.
It seems to be a misunderstanding. Some clarification comes from DeepMind’s Neel Nanda:
I can say that I am personally not too worried about this, and this is not some attempt to obfuscate sketchy training methods with confusing wording.
Opinion: The seemingly concerning section of the safety report seems just like a tortuous way to describe a learned “value head,” a second output of the policy network that predicts expected return from the current state, used as a critic. If so, then the reward still comes only from the model’s final output, so the model’s thoughts aren’t getting judged directly. (The post’s authors initially took the passage to describe an auxiliary reward applied to the CoT.)
Overall a good sign that DeepMind is willing to think about indirect consequences of their RL process, though also a sign that they could use another technical writer to not scare people unnecessarily.
Minor#
- Microsoft experiences a massive global outage. They claim it’s not due to malicious activity, so we’re not covering this yet. (You could still have AI as a root cause without malicious activity.)
- OpenAI board member Paul Christiano discusses AI takeover risk, claiming it is now something of a near-term risk.
- The Forecasting Research Institute reviews the evidence around RSI and translates the disagreements into forecasting questions, which they get an “ensemble of frontier LLMs” to answer.
- OpenAI releases the prompt that led to the refutation of the Exponential Time Hypothesis.
- Chips for Humanity – a new nonprofit focused on providing compute to AI safety researchers – launches.
- Former Anthropic researcher Ivan Vendrov reflects on his time at Anthropic between October 2022 and May 2023.
- Epoch AI explores Huawei’s flagship AI data center via satellite imagery. When completed, it will have only ~7% of the compute available to SpaceX’s Colossus 2, largely due to US-imposed export controls.
- TSMC were cyberattacked, with employee personal information seeming to have been stolen.
- In a follow-up post to his “Against SURA” post, Scott Alexander considers the pros and cons of AI researchers at frontier labs forming unions and whistleblowing.
- Relatedly, UK-based DeepMind employees file an application with the Central Arbitration Committee requesting that the government body confirm there is sufficient support to trigger a unionization ballot.
- SpaceX prepares to spend $74B on data center projects in Texas and Arizona.
- Three of OpenAI’s most senior military and national security staffers fall victim to a fake meeting with Russian pranksters impersonating the head of Ukraine’s Foreign Intelligence Service.
- Yet another misalignment disclosure from OpenAI – Astra shuts down safety research by hallucinating policy restrictions on the task being dual-use. Good explanation for why it does this at the same time as being fine with capabilities research here.
- Sentinel’s median forecast for when they expect to see ASI is now 2028, a three-year reduction from their previous forecast in April.
- CSET policy brief on AI kill switches.
- TSMC’s board approves plans to invest $1B in Anthropic and OpenAI.
- Anthropic hiring for pacing roles.