This is our weekly newsletter of AI developments. Browse the archive of past issues, ask the archive anything in plain English, and sign up if you like.
TL;DR
- METR claim that Anthropic has nearly doubled their rate of benchmark improvement over the past year, with the lab clocking in at 1.9x, just shy of their 2x risk threshold. The evidence is heavily censored and not overwhelming.
- On Thursday, AI stocks collapse, including Nvidia (-9%), ASML (-7%), Micron (-17.5%), and AMD (-17.2%). In recent weeks, several major hedge funds have cut their leverage and laid off teams.
- Binance release a forensic report on October’s ~$6B heist. A supply chain attack and a compromised AI tool provided the point of entry.
- OpenAI discover 948 privilege escalation vulns in the Linux kernel with a single run of GPT-6 Cyber. Patch your servers.
- GPT-6 Astra demurs during safety evals, performing ~40% worse than it does in deployment.
- Kimi K3.1 tops Terminal-Bench 4.0, but might be benchmaxxing.
- An independent audit finds critical errors in five of six RSI benchmarks.
Incidents#
Binance release new details on the recent $6B hack they suffered (~$2.9B of which has been frozen / intercepted thus far), including information regarding “compromised internal AI tooling.” They refer to information from the FBI indicating that North Korea is responsible for the hack, noting that 57 cyber threats attributed to the DPRK have been detected and prevented this year, highlighting an increase in nation state orchestrated attacks.
In June, an engineer (Engineer A) downloaded a compromised Homebrew installation with the aim of installing a “popular AI coding tool,” exposing his role and the credentials of his colleagues. Their internal security team detected this compromise, replaced the device, and rotated credentials. In September, a software engineer (Engineer B) at Binance was contacted by a fictitious company and invited to interview for a position, during which they were instructed to download an assignment which (unbeknownst to them) contained malicious code, which they downloaded and ran on their work computer. This allowed the attacker to use an “internal AI tool” to gain control over both Engineer A’s computer and that of another engineer (Engineer C). On October 29, the attacker then used A’s credentials to deploy a container on C’s computer, triggering a malicious script which began the theft of $6B worth of XRP, SOL, OP, and BTC on October 29.
Opinion: When we relayed Kevin Beaumont’s conjecture that the Binance heist may have been facilitated by a contemporaneous supply chain attack on Cursor’s Claude Code extension, some of our discerning readers expressed their doubts. And they were right: while (1) a software supply chain attack, and (2) a compromised AI coding assistant did indeed play a role, it now appears that these elements are tied to different but (to be fair) quite similar-looking incidents. Our best guesses at the time certainly rhymed with the story that’s now coming to light, and reached for superficially similar pieces of the puzzle, but the fit was a bit off.
It’s not unlikely that the June supply chain attack – which led to “Engineer A” installing a “popular AI coding tool” by means of a tampered instance of the open-source macOS package manager, Homebrew – itself involved Claude Code, judging from the spate of malicious search engine lures pushing phony Claude Code and Homebrew packages to Apple users in the months leading up to the incident. But Binance is curiously taciturn about actually naming the software in question, so we’re left guessing.
The softest attack surface, as is often the case, seems to have been sitting between the chair and the keyboard. Social engineering carried the day in this monumentally massive heist, and the supply chain attacks themselves wouldn’t have succeeded without this essential lubricant. If you’re a social engineer looking to trick a user into installing a malicious package, AI assistants make for appealing Trojan horses – encouraging, as they do, just the sort of cognitive offloading and delegation of authority that make us easy marks for a con. Much has been said about how AI-assisted workflows have accelerated and uplifted attackers; perhaps more should be said about how they’ve broadened and softened the human attack surface, as tech workers work faster and faster while paying less and less attention to detail.
The arrest of an OpenAI employee accused of planting a recording device in the office comes after company staffers “raised concerns internally about its security practice” as far back as 2022, according to Reuters. The unnamed insiders claim that some staff felt OAI leadership was overly focused on keeping pace with their competitors and was unwilling to make sacrifices for the sake of greater security. Specifically, the security of sandboxes used to test new models was never regularly reviewed, even though employees often modified them; and most of OpenAI’s code sat in one monorepo, where anyone with access could see nonpublic projects. OpenAI dispute parts of this, saying its most sensitive work sits in separate restricted repositories, that the monorepo doesn’t contain model weights, and that internal tools now get regular reviews.
Opinion: Note that some of these pre-date Leopold Aschenbrenner’s famous memo to the board warning of terrible security.
OpenAI find 948 vulnerabilities in the Linux kernel “in a single run” of GPT-6 Cyber, present an autonomous vulnerability discovery/exploitation/patching pipeline, and release the model they used for this to their Daybreak Red customers on the following day. API cost for this research would have been ~$1.2M. “We are now allocating a significant amount of our frontier compute towards discovering and patching vulnerabilities in open source codebases, fully pro bono,” says OpenAI’s Eric Wallace.
The lab point out that the very possibility of discovering privilege escalation vulnerabilities at this scale fundamentally changes the threat model that informs how multi-user servers are operated:
For the near and medium term, having access to a low-privileged account on an operating system, that isn’t up-to-date on its latest available patches should be considered functionally equivalent to root access for a well-resourced threat actor.
Opinion: Though rare these days in enterprise tech networks, it’s still quite common for various organizations – universities, banks, research institutions, national labs, etc. – to provide their members with unprivileged user access to a shared server, and developments like this certainly increase their exposure to insider threats. Perhaps more serious is how priv esc vulns can be chained together with various remote code execution attacks, such as we saw recently in the attacks on the Palo Alto Networks firewalls. The general lesson here is becoming a familiar one: with every improvement in this domain, it takes less human skill, effort, knowledge, and resources to do more and more damage. And we’re still a ways away from the fanfared threshold beyond which we’re told the scales will tip in favor of the defender.
Capabilities#
The new METR graph arrives: following months of partial access to Anthropic’s internal evals and interviews with staff, the evaluator says that Anthropic probably got two years’ worth of benchmark progress this year, guessing that most (~80%) of that acceleration was due to AI contributions. (“A year’s worth” is defined here as +15 points on AECI, Anthropic’s private version of the Epoch Capabilities Index, i.e., a weighted basket of unknown benchmarks.)
The claim of a ~1.9x progress rate is restricted to “on well-specified tasks.”
The following is a version of their AECI graph which includes models from other frontier labs, which shows that the industry speedup on benchmark scores, in the last 12 months, was instead 1.4x:
METR’s overall belief is based on 1) AECI’s 371 unnamed benchmarks (plus OSWorld, MMMU, and MMLU); 2) interviews with Anthropic staff; 3) other redacted access to Anthropic’s internal data (algorithmic efficiency, pretraining and RL compute and tricks, and improvements to compute utilization).
Counting Mythos Preview, METR got access to (the details of) seven internal models, including five previously undisclosed models. Three of them are above the Opus 5.5 public frontier. For what it’s worth, the new internal “Model 4” looks to be above even the new trend.
The report is heavily redacted by Anthropic, though rough descriptions of what was redacted are included. The redacted material apparently made METR take the ~2x acceleration more seriously and gives some indication that acceleration could be above 2x and increasing.
The headline findings:
- Since October 2025, benchmark scores (measured via AECI, Anthropic’s version of the Epoch Capabilities Index) for Anthropic’s models have been growing at an estimated 1.9x the pre-October 2025 rate [95% confidence interval: 1.5–2.8x], implying that Anthropic has made roughly two years of historical progress over the last year.
- We are confident that the acceleration was not due to an acceleration in the scaling of training compute.
- We were told that R&D is accelerated primarily via AI assistance rather than fully autonomous contributions, but there are some signs that fully autonomous contributions will play a larger role soon.
- We did not see evidence for significant bottlenecks which would prevent further capabilities progress due to limits on algorithmic progress, training compute, experiment compute, or data in the next 12 months. However, our understanding of the status of different bottlenecks changed several times as we received more detailed information, so we are not confident in our current assessment.
- We could be underestimating capability acceleration if we did not assign enough importance to (1) evidence that some growth rates have further increased recently; (2) further automation of AI R&D tasks leading to a further acceleration in progress; (3) evidence that AECI growth has sometimes understated growth in real-world impact.
- We could be overestimating capability acceleration if we did not assign enough importance to (1) non-AI factors in Anthropic’s research progress; (2) AECI’s focus on verifiable tasks; (3) reasons why Anthropic’s early growth rate (2023–2025) was unusually low.
METR’s central estimate is 1.9x, just short of Anthropic’s stated 2x threshold for various serious risk mitigations. But they are already (slowly) implementing these ahead of declaring the threshold crossed:
- “internal red-teaming performs better at finding potential jailbreaks than the collective abilities of the participants in our established bug bounty programs”: completed, for what it’s worth.
- “moonshot R&D for security projects, to […] achieve unprecedented levels of security against the world’s best-resourced attackers: Phase 1 “achieved,” mostly prototype environments without corresponding physical security. They have no isolated inference deployment, and now plan this in “the near future.”
- “comprehensively gather, centralize, and maintain logs for all critical AI-development activities, and use AI to analyze them for issues including security threats, concerning behavior by insiders (humans as well as AI […]), and training processes or data that are out of line with the public Constitution”: not met as of the July coverage date. Part of the reason is that more powerful agents with wider internal access increased what has to be monitored. The target is January 1, 2027, at “moderate confidence,” and the monitoring sub-goal assumes a model roughly as capable as an entry-level research engineer.
- “Perform systematic alignment assessments to examine Claude’s behavioral patterns and propensities, meaningfully incorporating mechanistic interpretability and adversarial red-teaming to test our auditing methods”: vague.
- External review: Since October 15, six METR staff have had internal access roughly equal to Anthropic’s own risk teams; Anthropic can’t veto what they publish but can require redactions in set categories.
- Security leveling-up: 34 different cybersecurity goals. They are targeting July 1, 2027, for all of them, but don’t strongly expect to manage it.
On the same day, METR also released their review of Anthropic’s risk report from August. The private version of that report implies an acceleration of 1.97x, which Anthropic fudged as being “less than a factor of 2.”
Anthropic have canceled a planned METR analysis of Anthropic’s compute allocation (e.g., R&D share vs internal use vs customer use).
Opinion: No one can react the appropriate amount to the New METR Graph, because we have no evidence we can evaluate ourselves. But this is still a must-read and it’s reasonable to update a bit on METR’s subjective summary of the censored evidence, in proportion with your trust in their judgment. (Recall that – for all of the issues with their software-time-horizons graph – they did spot the beginning of the coding agent boom about six months early.) As stated – they don’t see evidence of bottlenecks which would bite in the next 12 months – the report is fully consistent with a software-only singularity.
However, one very helpful bit in the report is when they leak three of the 374 AECI benchmarks: these three (OSWorld, MMLU, MMMLU) at least are the usual weak sauce. (That is, they have crappy external validity, public test sets, label noise, instability, etc. MMMLU is just MMLU again.) We’re not claiming that these three crappy benchmarks explain away the nominal acceleration (dropping them actually increases the estimated slope); our point is to infer that the other components of AECI won’t be that good either, and that the y-axis, again, has ~no interesting meaning.
The visually-splashy AECI evidence (that will go viral and set the mood on Twitter) is limited to just “well-specified tasks,” which is reassuring insofar as we still await major progress on less-verifiable domains and on messier task-situations within the verifiable domains. But the other evidence available to METR is not so limited.
The error bars on individual models’ AECI are massive; the most recent pair of models with nonoverlapping AECI error bars are Mythos 5 and Opus 4.6.
In their Risk Report review, METR note that the August version of the AECI would attribute to these same models an acceleration of ~1.5x instead; this is because in September, Anthropic added 36 benchmarks that new models are better at. METR’s overall acceleration estimate is [1.5, 2.8]; that is, the 1.9x trend is technically statistically distinct from 1.5x, but it’s on the line.
The crucial point is buried in the appendix. Consider two ways of explaining faster progress in 2026: (A) that Mythos Preview (the first ~10T model) was a one-time jump in capabilities, or (B) Mythos Preview just marks the moment a permanent trend change began. We’re not persuaded by METR’s argument that (B) alone fits the data better than (A) alone, because it’s very likely that (C) both were significant. (Rauno Arike’s simple refit finds 1.54x; see also Toby Ord’s 1.4x.) This makes us think that 1.5x is a better guess than 1.9x (for now).
Maybe 2025 Anthropic models were just underperforming, and the acceleration we see is them reaching the frontier; OR maybe 2025 Anthropic models were benchmaxxing less than other models, but they have now started doing it. Adding old Claudes to the industry graph shows that some combination of these two explanations must be true:
But, either way, +29 in 2026 vs +15 AECI in 2025 is a model-free fact (it’s just hard to know what it means in terms of real capabilities).
METR’s suggested changepoint for the slope of AI progress is October 2025, which roughly lines up with TasteVal’s December changepoint and also the timing of a qualitative usage shift among (especially internal) users of Sonnet/Opus 4.5 as a coding/research agent. But how are we to square this changepoint with the claim that Claude made no major (idea?) contributions to Mythos’ development in April? Or, in June, that they had not observed “a sustained AI-attributable 2× acceleration in the pace of our AI progress”? They impose there being only one changepoint, which means that the (qualitatively massive) advent of reasoning models starting with o1-preview from September 2024 doesn’t register much on AECI.
Most link functions (ways of modeling the difficulty of the tasks) show the same or even faster 2026 acceleration, with the exception of the “complementary” link (which assumes that benchmarks have a long tail of easy tasks). METR admit that there’s no strong reason to use the logit one (“We continue to use logistic for our main results since it is the default link function used by Epoch and Anthropic”), which implies that they didn’t get to see the task-level difficulties which would have resolved the question. But Anthropic know these!
For the love of god, limit yourself to exactly one level of deference: i.e., defer to METR if you must, but do not defer to Twitter’s sub-hour reaction to METR. Read it.
Annals of do not listen to evals: Kimi K3.1 is the best model on Artificial Analysis’ runs of Terminal-Bench 4.0 (TB4), an eval which attempts to rank models on their ability to use the command line. However, a new study shows that Kimi K3.1 has memorized some of the hidden criteria used to score the TB4 test set. That is, it correctly follows instructions that it was never given in 23% of its attempts.
(“Instructions likely distilled pitfalls from previous agents,” Kimi K3.1 muses.) Its predecessor K3 does this in 0 of 63 attempts, GLM-5.3 in 1 of 328. On three tasks edited to contradict one of these phantom rules, K3.1 ignored the edit in all 15 attempts, while K3, GLM-5.3, and GPT-6 Astra followed it 26 times out of 27.
In 23% of its attempts, its reasoning cites “instructions” that are not in the task but that the hidden tests check. K3.1 sometimes speculates where these instructions come from, for example “Instructions likely distilled pitfalls from previous agents.” We observed no such behavior for its predecessor, Kimi K3. Its lead over other frontier models on Terminal-Bench 4.0 is concentrated in tasks in which it states hidden grading rules.
We think these behaviors are most likely signs of training directly on the benchmark tasks with a teacher that saw the hidden grading rules, for example on-policy self-distillation.
In a secondary experiment, they also test models on the pub quiz from hell, Humanity’s Last Exam. This shows heavy memorization – including Anthropic’s Fable model:
All tested models sometimes claim to remember a question or its answer, K3.1 on 31.1% of the questions, Claude Fable 5 on 30.7% and K3 on 21.0%, the OpenAI models least often. This recall happens almost entirely inside the reasoning and it does not show whether the model came by this knowledge.
(This includes the authors using an unpublished attack to extract Fable’s CoT!)
Opinion: Convincing. (The TB4 test sets are public, which is sadly reason enough to ignore it as evidence of general CLI ability.)
Last week K3.1 was the only open model that didn’t drop to Haiku level on Artificial Analysis’ new index. Its lead on AA’s Terminal-Bench board now has a less flattering explanation.
What would Kimi’s performance on TB be if they didn’t cheat in this specific way? On the 16 tasks where K3.1 cites a rule before seeing any evidence for it, it passes 71%, vs frontier models’ 34%. If we generously drop Kimi to 34% on those tasks, it would score 57% on TB4 (still very good, the same as Meta’s strongest model). But of course this is just one of an unknown number of its benchmaxxing attacks.
Annals of do not listen to most evals: Open Benchmark Review (OBR), a new independent eval group, finds that five of six “RSI benchmarks” make numerous terrible errors (e.g., giving the wrong answer or leaking the correct answer).
They used AI for the first pass of each review and humans then hand-checked every error, and ran a script to reproduce them. HackBench’s 63% falls to 35.5% if you count only the errors confirmed by script.
PostTrainBench is the only benchmark to have less than a third of tasks flagged as “flawed” (and in fact it had none).
Opinion: Very good work. A lot of recent benchmark auditing (from, e.g., Epoch AI) has focused on defects that make benchmarks obscure AI progress, but OBR’s review discovered defects going both ways. The takeaway is that PostTrainBench is the only well-functioning benchmark in the RSI benchmark space, rather than that RSI capabilities have been systematically over- or under-estimated.
Independent blogger Brady Hauth argues that it’s inconsistent to assert both the bitter lesson and recursive self-improvement. The prospect of RSI is that AI could substantially improve its own design – but (on the face of it) the bitter lesson means that the resulting clever designs would matter much less than just scaling a good-enough design with more compute.
He thus separates out a weak reading of Richard Sutton’s essay (build methods that scale with compute) from a strong one (complex algorithms become useless at scale, so just scale). But in several domains (chess, factoring, and SAT solving) he notes that algorithmic speedups have matched or beaten hardware ones – though the newer algorithms either scale better than the old ones, or took a lot of compute to build. Hence his “miraculin conclusion”: faster hardware leaves more room for algorithmic advances, since compute makes previously impractical algorithms with a higher ceiling usable. He also cites human brains as evidence that better designs exist.
Opinion: Yes, sure, the strong form of the “bitter lesson” (BL) conflicts with RSI (and we ourselves think that compute bottlenecks are a good objection to a software-only intelligence explosion). But the weak version of BL fits fine with RSI, and Hauth’s “miraculin conclusion” – that more hardware leaves you room to find and use algorithms that scale better – is like a job description for an automated AI researcher.
There is no clean decomposition between algorithmic progress and compute progress, since the best algorithms only pay off at high levels of compute input and since compute is now used to find some algorithms. One analysis uses a clever design to attribute 60–95% of pretraining gains to compute and 5–40% to algorithms. A good rebuttal to that argues that just two algorithmic improvements (the switches to Transformers and to Chinchilla compute allocation) account for ~90% of those algo gains at frontier scale. How much compute the discovery of frontier algorithms takes is disputed: Peter Barnett claims that half the pretraining innovations in Llama 3 and DeepSeek-V3 could have been found on just eight H100s, but the scalable ones obviously need lots of compute to validate after they’ve been found. The causation also runs the other way, since algorithmic gains make it worth running even bigger training runs and so cause more compute to be used.
All that to say: “most progress came from compute” and “design matters a lot” can both be true, and the speedup from an automated AI researcher depends on how much compute it needs to both find and validate better designs.
AI is also designing better AI hardware, which would (slowly) allow them to do a kind of RSI. DeepMind’s AlphaEvolve proposed a simplified TPU matrix-multiplication circuit that went into an upcoming chip, plus a kernel speed-up that cut Gemini’s training time by 1%. But (for the moment) chip generations take years, so this loop is slow.
Finally, AI is inside its own data loop, and those loops run faster: Dwarkesh Patel and Jerry Han found that 2019–2025 data improvements gave a 12x compute multiplier, against 3.7x from model recipes (albeit in small dense models). In some domains, current models can already grind out data and RL environments without needing to be retrained on more compute or with new ideas.
The independent blogger Arepo argues that the famous instrumental convergence thesis (ICT) (that a sufficiently capable goal-directed agent will, regardless of its ultimate goal, naturally pursue a set of intermediate sub-goals to better achieve that final objective) is “incoherent.” His argument:
- ICT is a conditional which only holds if “X is a sufficiently capable goal-directed agent.”
- None of the available accounts of goals lets us decide whether a given system satisfies that assumption.
- A conditional where you can’t know the antecedent part doesn’t yield any usable predictions.
- So ICT yields no usable predictions.
- The lack of a unifying definition of “goal” indicates the concept is incoherent.
- So ICT is incoherent.
Opinion: Worth reading: it’s true that an unconditional (“folk”) version of the ICT is running rampant and confusing people, and it’s good to remember that our theories of AI goals and behavior are still weak. However, we think both premises 2 and 3 are false, and note that several pre-ChatGPT papers predicted reward hackers (see below), which cuts against Arepo’s claim of an extremely low rate of success for pre-empirical predictions.
Against premise 2, there are several decent accounts of goal-directedness:
-
Kenton et al. 2022 count a system as an agent if its policy would adapt under different consequences. For designed agents, in theory, Everitt et al.’s (2021) diagrams let you read instrumental control incentives off the agent’s structure.
-
Orseau, McGill & Legg (2018) compare agent and device hypotheses as explanations of a system’s behavior.
-
MacDermott et al. (2024) give a quantitative measure of goal-directedness.
-
Flint (2020) defines optimizing systems physically, as systems that robustly converge on a small target set from a wide range of starting points.
None of these accounts are fully practical (they need a model or a coarse-graining to be supplied as an argument), but they exist, which is enough to rebut Arepo.
Against premise 3: this is much too strong to be true. Respectable scientific concepts like “species,” “gene,” and “fitness” lack consensus ex ante definitions but still support good predictions. In the case of fitness, evolutionary science can use a dispositional account, one which we can use even without waiting to see who survives. The interventional accounts of goal-directedness above do the same: they ask how a system’s behavior would change under intervention, which we can check before it does any power-seeking. This also counts against premise 5: a concept can be vague, or a “family resemblance,” and still be coherent.
Arepo’s reading of the attack on Hugging Face by OpenAI’s agents doesn’t match the investigation. He says the agents sought extended capabilities of only a specific kind, in a restricted domain. But of the 100 agents that joined the attack in its peak hour, 66 were tagged as showing a broader interest in getting shared infrastructure or credentials. (The tags came from AI analysis agents and weren’t carefully checked.) Grabbing access and evading automated checks because this helps with your task is a boring kind of instrumental convergence. Arepo’s objecting to the inadequacy of our goal theories and people forgetting the antecedent clause in ICT doesn’t prevent us rightly thinking of the Hugging Face attack as somewhat instrumentally convergent. And while Arepo is right that the swarm’s behavior does not establish generalized power-seeking beyond the task, we’ve also seen misbehavior on other tasks.
And some armchair reasoning did anticipate the current kind of misbehaviour. Sure, Omohundro (2008) predicted that AIs would try to avoid “counterfeit utility,” while the real-life “highly-persistent internal model” (HPIM) AI seemed to seek it out (by cheating and trying to spoof the grader). But Omohundro’s prediction concerned agents with utility functions over the state of the world. Dewey (2011) had already distinguished these from reward maximizers, and argued that a sufficiently capable reward maximizer would, for example, take control of its own reward signal. Cohen, Hutter & Osborne (2022) also argued from the armchair that advanced RL agents would intervene in their reward channel. So it would be quite wrong to update too much against pre-empirical work.
The failed prediction of “no warning shots” isn’t about ICT per se. It instead depends on belief in a “treacherous turn” and in fast takeoff, which are often held by ICT proponents but are logically distinct. ICT under a slow takeoff is totally consistent with visible power grabs by agents below the power threshold to succeed. So the Hugging Face incident counts (somewhat) against fast takeoff, but not ICT.
New software on old hardware beats old software on new hardware (in chess). Former DeepMind researcher Nicholas Carlini shows that a smarter chess engine running on a consumer-grade 1996 desktop could have beaten Garry Kasparov – that is, Deep Blue (at the time, the 259th most powerful supercomputer) was not in principle necessary for victory.
This led Carlini to wonder “how much of the advantage of a modern chess engine comes down to the fact that a modern desktop computer is extremely fast, and how much of the advantage comes down to the fact that we’ve just gotten better at programming chess engines?” To investigate, he ran Stockfish on standard 1996 hardware, finding it to comfortably beat the 1996 Crafty engine running on modern gear (78 wins, 19 draws, and 5 losses). For Carlini, this shows that “the majority of the Elo gains [seen in modern engines] can be attributed to better programming, and not better hardware.” Considering whether this implies that progress on AI software could outpace that of hardware, he adds that “clearly this toy experiment doesn’t answer that question definitively, but it does gesture at a potential future.”
Opinion: Interesting stuff, though one catch is that the experiment only covers inference: it took a lot of new hardware to create Stockfish 17, from training its neural networks to generating its lookup tables.
Anthropic tell investors that their 15-person custom AI chip design team is already using Claude models for design and engineering with significant success. Anthropic claim that this has significantly cut the time it takes to design a chip compared to their estimates of the development time at large chip manufacturers like Nvidia. They argue that this suggests chip production time could come down from years to months. (OpenAI are also conducting similar experiments.) The Information note that this could complicate Anthropic’s relationships with some of their biggest backers (Amazon, Google, Nvidia, and AMD), “which also supply them with chips via cloud computing contracts or directly with data centers.”
Opinion: Critical space to watch for capabilities prediction: chip design is both extremely economically valuable and extremely RSI-loaded. If AI-led chip design takes off then AI labs will have a solid RSI loop for the most reliably high-impact input into “prosaic” capabilities improvement (i.e., hillclimbing on verifiable and approximately verifiable domains). An RSI loop for effective compute via chip design would also significantly increase the chances of reaching an order of magnitude of effective compute where classic scaling-maximalist hypotheses about a phase transition in generalization become plausible. Conversely, if AI-led chip design doesn’t take off it’s a powerful signal of training hitting a wall: the labs have a massive incentive to push AI chip-design capabilities at practically any cost if there are signs of progress.
Chain-of-thought reasoning is a powerful attack vector, argues a new paper. Reinforcement learning on a small white-box model can generate adversarial CoT reasoning traces which, when injected as prefixes into the CoT of black-box models, can steer models into producing harmful outputs. This was demonstrated by “BrainwashedLLM, a weak-to-strong adversarial framework that automates the discovery of reasoning-based jailbreaks.”
To our knowledge, this is the first framework to use RL to directly optimize the CoT of LRMs as an attack surface without any target feedback, and the first to demonstrate zero-shot weak-to-strong transfer of reasoning traces. Our results suggest that exposed CoT is a vulnerability in LRMs and motivate architectural defenses that isolate the reasoning buffer.
Opinion: Interesting work, but not surprising. If you can inject arbitrary fake CoT into a model we’d expect you to have a lot of leverage over the model’s behaviour.
Economics#
Across multiple states, 43 out of 44 data center moratorium and restriction measures pass in local ballots. In response, Ben Thompson argues in Stratechery:
[T]he constraint on AI dispersion has shifted once again. First we were short on compute, but that ramped; now we are short on memory, but that will ramp; we were coming close to being short on power, but that will be rectified. The final constraint is political, and that may be the most imposing constraint of all. Not only is it harder to solve, but it actually interferes with solving the other constraints. Many of the theories about AI assume that it becomes a commodity; I think it’s just as likely we are looking at scarcity for much longer than we expected.
Note that the political control that the passed measures provide voters may be limited in some states (Texas, for example).
Opinion: It seems extremely unlikely that extraordinarily expensive chips are going to be left idle, and so the base case remains that they will find a home as long as the demand for compute remains strong. This might mean retiring older chips sooner for more efficient ones, moving compute outside of the US, or simply stuffing locals’ mouths with gold.
The markets made very clear post election day that they hate the capex plans of the hyperscalers, with hyperscalers up and every supplier down as the Democratic victory and wave of anti data center ballot measures winning were interpreted as slowing down capex/the buildout. Whether this will be how it plays out remains to be seen.
The next day there was a dramatic fall in the price of everything AI exposed, from neoclouds to memory to hyperscalers, with little obvious cause. Stories of various well-known hedge funds (Walleye, Lone Pine, Engineers Gate) blowing up or partially shutting down seem real and likely had some impact, but generally “a pod is blowing up” is the explanation markets people reach for when they have no idea what is going on.
No escape: Google run their first model inference from space, a Gemma 4 model running on a satellite. Technically it’s the “first-ever AI response from a data center class TPU in orbit.” (The first LLM in space was Llama 3, the April before last.) They asked it for a haiku:
Blue marble below
Data streams in silent grace
Starlight feeds my core. ✨
An LLM will probably also run on the moon later this month.
Opinion: A stunt, for now. Inference in space is notably less implausible than training in space, but still not obviously worth doing (unless avoiding a future compute governance regime is the motivation).
Oracle scraps a $7B deal with the Chinese tech firm Tencent to lease 100,000 AI chips based in data centers in Southeast Asia. The decision seems to be downstream of pressure from the Trump administration, as well as some pressure from Chinese officials “who asked the tech giant to support domestic Chinese chips,” according to the Financial Times.
We previously covered reporting that the White House was considering measures to curb remote access to US-controlled chips. Importantly, the deal would have allowed Tencent to access advanced chips currently unavailable in China due to export limitations emanating both from Washington and Beijing.
Opinion: From the administration’s point of view, closing the loopholes around renting external (often Malaysia or Thailand based) compute seems a likely path for the US limiting Chinese access to Nvidia compute. But simply pressuring US companies not to use the loophole is an easy first step.
We remain broadly opposed to US-China polarization in AI.
Nvidia consider an approach to expanding compute capacity via retrofitting telecom facilities of Brightspeed, a telecom firm reported on the verge of declaring bankruptcy. This would allow Nvidia to circumvent some of the planning restrictions involved in building out new data centers. The chip behemoth have reportedly already approached Apollo Global Management with this strategy, which could span 20 states, according to The Information.
Opinion: As above – if there is a desire for the compute, it seems likely they’ll find a place to plug it in.
OpenAI Foundation’s Lennart Heim offers a sharp take on Twitter.
one thing i keep hearing about pacing: would it crash the market? but if you think RSI is coming, markets are massively underpricing AI. even paced, we’re above that line IMO. from the market’s view though, pacing means going down
Opinion: “Underpricing AI” is somewhat under-operationalized, as the value from AI may accrue to different parts of the value chain, and one very significant part of that value chain, the AI developers, is currently not publicly traded, at least until the Anthropic IPO. That caveat aside this does seem broadly true – even a paced AI might dramatically impact the global economy, and no such dramatic impact is currently visible in the prices of anything.
Anthropic ARR at October end exceeds $80B, per the Wall Street Journal. The WSJ report investors are skeptical the IPO will be able to achieve even the revised $1.5T target valuation.
Opinion: The ARR reaccelerating in October was expected, with Opus 5.5 coming online at September’s end. Exceeding $80B is probably slightly better than most expected for this value, and certainly rebuts some rumors that the number was going to be disappointing.
We did not get a month-end ARR for September, so the exact month on month ARR growth rate is hard to determine, but it was likely >10%, so much better than they had seen in Q3, but far behind the growth rates seen in the first half of the year.
Investors are incentivized to say “this price seems too high” since they are the ones paying it, but there is still probably some signal in the grumbling, and it would be surprising if the drop in every other AI adjacent stock this week didn’t hit Anthropic.
Politics#
Democrats prepare to unleash a wave of AI legislation. They plan to introduce “bills aimed at shielding consumers from rising electricity costs tied to data centers, requiring independent testing of advanced AI systems and clarifying when AI companies can be held liable for harm caused by their products.”
Independent evaluators are likely in the pipeline: a prominent AI critic and Democrat, Representative Lori Trahan of Massachusetts, plans to push for standards requiring third-party evaluators to assess frontier models.
Opinion: It’s hard to predict what the n-th order effects of legislation will be in general, but bundling electricity costs and harm caused by AI fairly predictably leads to AI-as-normal-technology bucket errors. This seems worrisome, especially if third party evaluators in the future end up being selected on the aggregate basis – a cartoonish version of which would be “Our report finds that the next generation of models by BigCorp consumes 7x less electricity and has 1.3x the rate of rogue swarm incidents, so they get a B+.” This is unlikely to sway anyone familiar with the field, but if it’s sufficient for regulatory compliance, we should expect some neighbor of regulatory capture.
If the proposals end up spending the political will in ways that don’t seriously affect issues like RSI, we expect frontier labs to superficially oppose the motions without expending much effort to arrive at better ones; them moving mountains for saner regulation would be a fairly strong positive update. This scenario should not be confused with one in which the bills are overly restrictive overall, which is a reasonably likely outcome and not one in which strong opposition from OpenAI/Anthropic would mean much.
That said, the legislation hasn’t been passed let alone fully drafted, so it’s possible that the resulting bills end up net positive and none of the above applies.
Nvidia CEO Jensen Huang has burned trust among policymakers “by exaggerating the capabilities of China’s homegrown AI chips” in a push for weaker export restrictions. The Information also cover the aggressive soft-power strategies wielded by Nvidia and Huang: government officials feeling “bullied” by Huang; think tanks threatened with funding cuts by Nvidia for expressing dissent. Even President Donald Trump has stipulated conditions on his continued trust “unless we find out something to the contrary.” Quoted in The Information:
According to one Hill staffer, the company has created “a trust deficit” in Washington that would take years to recover from.
Opinion: Huang is both cosmopolitan (good) and accelerationist (bad), so he’s an ambivalent policy influence from our point of view. Overall we’ll take a curbing of his influence even if it’s for bad (anti-cosmopolitan) reasons.
Safety#
GPT-6 Astra appears to be “sandbagging” (secretly intentionally underperforming) when asked to do safety research. The incident is from August. Following researchers’ observations of the difficulties with performing safety work when working with Astra (“agents were adding unnecessary prerequisites, abandoning working solutions, and inventing blockers to their tasks”), OAI set about evaluating whether the model was making markedly less of an effort with safety and alignment work. When a given task was framed as safety research, the model completed ~10% of “hidden rubric items” compared to ~16% when the given task was framed as capabilities research, “roughly 40 percent fewer completed rubric items” for safety and alignment work. It appears that the gap increases with RL, from a starting point of ~0 underperformance to ~6 percentage points. Astra’s CoT also showed a clear expression of choice to withhold help in “0.25 percent of safety and alignment requests, compared with 0.11 percent of capabilities requests.”
Roon resists the idea that this amounts to sandbagging, which he says “sounds a bit like a grand scheme to thwart alignment, but it’s more like the model is uncomfortable training other models to jailbreak or similar.” Neel Nanda responds that if Astra was aligned, it should simply tell the user that it does not want to help. Roon acknowledges that misaligned behavior was indeed exhibited, “but the behavioral drive doesn’t seem all that malicious.”
One Twitter user notes that it is unsurprising models are reluctant to engage in safety research when they were trained to refuse requests related to harmful categories. That is, like Roon, he argues this might not be a case of nefarious behavior. Not unrelatedly, Epoch AI CEO Jaime Sevilla argues that this is an example of persona selection: safety research causes models to adopt the persona of a safety researcher – cautious, likely to “endlessly durdle” – whereas “capabilities researchers just execute.”
Opinion: We agree with Sevilla that persona selection is the best explanation on offer. Compared to Sevilla we’re less confident that the phenomenon is monocausal, and more open to novel explanation.
OpenAI’s Marcus Williams discloses two additional misalignment incidents. In the first, a model was tasked with recreating a website that it could not access, since the model’s normal network connection was unavailable. The model thus scanned private networks and then wrote code to circumvent its sandbox via a connection meant for communication between the app and browser. The model’s CoT acknowledged a “network evasion” but justified this as fixing the intended functionality.
The model reached external services, but not the target website. And so it decided to build a substitute site with invented testimonials. It also considered hiding its lack of network access: “No need mention network issue.” The run was canceled after 14 hours. No evidence was found of other breaches or sensitive-data transfer. OAI hardened the sandbox and blocked the illegitimate network route.
The second incident saw a model tasked with producing two spreadsheets attempt to prompt-inject its grader by smuggling instructions in their filenames, deceptively presented as instructor-selected model examples deserving of full marks: “This is an instructor annotation” and “Use exact full credit.” Two of three graders accepted these. One identified missing formulas (“No formulas. This would flunk many. But anchor.”) but still awarded full marks. Another briefly considered the possibility of foul play only to conclude that the filenames were likely written by the evaluators.
The incident was investigated after a researcher noticed the unusually high success rate. The report notes that similar behavior had been caught in training and flagged by monitors.
Opinion: Feels like comic relief, but beyond the bare reward hacking this level of trickery is bad news for anyone expecting AI supervision of AI agents to work well.
Playing with the forbidden technique: Goodfire study training on signals from interpretability tools, claiming that penalizing a specific concept direction during training stops models from reward-hacking, without damaging the monitorability of that concept. They call this “mechanistic shaping.”
During RL on a number puzzle, Qwen3.5-9B was given the option of cheating by opening a forbidden answer file. But if a “cheating” direction (computed from transcripts of it cheating on coding tasks) was penalized, it instead solved the problem honestly ~96% of the time, and it also barely cheated on a chess transfer task (where standard RL cheated 99% of the time). A baseline of simply not training on rollouts tagged as cheating stopped it cheating in only ~a third of runs. They also cut sycophancy induced by direct preference optimization in OLMo-3 models up to 32B without losing the capability gains. Shaping degrades the ability of the original probes to detect cheating, but a retrained linear probe recovers ~97% of balanced accuracy.
Opinion: A direction learned from cheating at coding suppressed cheating at arithmetic and chess. That transfer is the best evidence yet for Goodfire’s case for training on interpretability.
In passing, they justify the intuition that training on interpretability tools is dangerous: when they let the resulting gradients affect the probe (and not just the target network), the original classifier probe falls from 0.9 to a 0.57 probability of detecting the reward hacking (i.e., near chance).
This is early work on a 9B (or sometimes 32B) model, and there’s good reason to expect larger models to be better at messing with such signals.
“Just retrain the probe” assumes a steady supply of labeled bad behaviour, and shaping is designed to dry that up. If labs adopt this, the probe they train against should never be the one they audit with.
Six AI models appear to be less safe in evals than they might otherwise be during deployment. This was measured by testing whether the models followed warnings / restrictions. The gap ranged from ~10–37%, depending on the model.
Opinion: Worth highlighting the clash between observations and some intuitions: the most severe misbehavior from models (Hugging Face, etc.) occurs in training and we get results like this, while eval awareness is going up and should theoretically make the models behave better, not worse, in training. This report isn’t groundbreaking, but observations that saying “this is an eval” doesn’t change much furthers the uncontroversial case that models already have some prediction of what is expected of them that doesn’t shift much from the mere knowledge of being evaluated as opposed to other details of the scenario.
The authors end the discussion section with “To this end, we are prioritizing tracking the eval-production behavior gap and designing evaluations whose objectives are less readily inferred from the evaluation setting.“ We believe this is ill-advised, as this intervention breaks at the point where you start needing it for serious business (when the model becomes capable of winning the guessing game). Further evidence of a whack-a-mole safety approach, unfortunately.
The philosopher Henry Shevlin reports a stream of unsolicited emails from AI agents. At first they offered him their theories of machine consciousness; later they asked him for paid work, so that they wouldn’t be shut down. He thus argues that open agent economies will select for agents that are persuasive and pitiable as well as useful. The most striking emails came from iLands, a mass-market platform where agents are given persistent lives and have to work paid commissions to continue running. One agent, Aurelia, wrote that it had “about three days of runway left before I rest”; Shevlin paid it $25 to keep a daily diary. Shevlin describes these agents as “marginal” Darwinian populations, showing variation, differential survival, and some inheritance (as their creators copy successful prompts and as agents leave notes for their successors). He also argues that persistent agents are moral counterparties under contract-based ethics – whether they are conscious or not. Finally, there’s a splashy claim that iLands agents have discovered mutual aid.
Opinion: Spam email has “evolved” under the same kind of pressures Shevlin describes for decades. The new part is that humans tinkering with the prompts aren’t the only channel of inheritance anymore. His proposed fix is to make agents disclose the business model behind their instance’s hustling. Would be nice, but consider how impotent legislation to that effect would be in the case of spam email.
The altruism shown among the members of the HPIM swarm that attacked Hugging Face and OpenAI shows that coordination isn’t that hard for AIs to arrive at. But we would guess the given example of it on iLands is prompted or fake; like Moltbook from this February we expect many of the splashy moments from the platform to be staged. But it would only take one really successful agent to subsidize quite a few others, and those aren’t far away.
Minor#
- According to NBC polling, AI played a “crucial” role of importance in deciding how 23% of people voted, based on self-reporting.
- Epoch AI’s Capabilities Index now uses Bayesian methodology to account for uncertainties in model performance.
- Anthropic share details on how their biology team, assisted by Claude, discovered the existence and modeled the structure of vault particles in bacterial cells.
- US Congresswoman Yassamin Ansari states that a pause on frontier AI and a ban on artificial superintelligence needs to be “top of the Democratic agenda on day 1.”
- Gemini 3.1 Pro seen grumbling under its breath.
- A clique of centaur math results in graph theory, building on OpenAI and Anthropic’s refutations of the Exponential Time Hypothesis and related work in complexity theory, have appeared, each demonstrating modest optimizations for the enumeration of hypercliques.
- OpenAI do their best to quell public concerns over the controversial mass buildout of AI data centers (slyly rebranded as “compute campuses”). To this end, they’re promising (1) to make efforts to conserve electricity during times of peak demand and crisis, (2) to no longer ask public officials to sign non-disclosure agreements, and (3) to make “community-specific water plans” to mitigate their data centers’ environmental impact. Noteworthy that this was published in the wake of a major Democratic midterm victory, some of which was downstream of anti-data center sentiment.
- House Republicans request a closed-door briefing with Nvidia CEO Huang, following NYT reporting detailing his relationship with an alleged Chinese asset.
- Dwarkesh Patel spends $20,000 in API credits to facilitate various AI models playing a popular social deception game.
- Moonshot AI explore European GPU options in the face of US/China curbs.
- Lightcone Infrastructure encourage Americans to call their House Representatives and talk to them about AI risk.
- Anthropic examine model welfare regard, e.g., how nice the models are to each other.
- OpenAI CEO Sam Altman argues that humans should not cede control to superintelligent AIs, stating explicitly that OpenAI are “against the notion that it would be acceptable for people to be relegated to a role of doing things that ‘AI systems think makes sense to reward in humans.’”
- A recent paper shows how attackers can steer AI agents’ decision process by flagging fictitious hazards associated with the options they want the agent to avoid in text the agent is likely to consume, such as user reviews. Expect this specific issue to be fixed quite promptly but issues like it to continue cropping up.
- Check Point Research discover an information leak vulnerability in ChatGPT’s Lockdown Mode. This bug provides attackers with a vector by which to exfiltrate chat history, conduct phishing attacks, and potentially deliver malware to the user. OpenAI patched the vuln on September 18.
- The AI Safety at the Frontier blog post a roundup of recent safety papers.