This is our weekly newsletter of AI developments. Browse the archive of past issues, ask the archive anything in plain English, and sign up if you like.
TL;DR
- Anthropic and Google find a critical remote code execution bug in the firmware run by most of the servers and desktops on Earth.
- A contractor running Claude on a restricted Pentagon system, which proceeds to send login credentials to a third-party site.
- AI-assisted attackers chain PHP2Shell with the Linux kernel vuln Fragnesia to root 25 Palo Alto firewall servers in Japan, Taiwan, Korea, and the US.
- Exposed Cairn agentic hacking panels are hijacked by a second wave of AI-wrangling hackers.
- India’s CERT report “rogue agent” attacks on Indian companies.
- A trojanized Claude Code extension is pushed to Cursor’s marketplace and may be linked to the massive Binance heist.
- AI researcher Nostalgebraist writes a mini-essay on continuing unreliability and poor judgment in frontier AI.
- A bird’s-eye view of the cybersecurity arena in the year of agentic hacking.
Incidents#
It is now difficult to cover all of the breaking AI hacking stories. We suspect that we may soon be entering an era in which, as with new software, some degree of AI assistance should be taken for granted in any cybersecurity incident, unless there’s reason to believe otherwise. At present, though, it seems as though the hype is running ahead of the numbers, at least as far as we can tell. See this issue’s deep dive for details.
Anthropic and Google discover a critical bug in the firmware run by ~90% of (non-Mac) machines on Earth. The bug lets an unauthenticated attacker run code at the lowest level, before Secure Boot or OS protections can load.
The exploit is for the ubiquitous EDK II open-source reference implementation of the UEFI system. It allows an attacker to target a network boot driver, overflow into adjacent memory, and thus operate on the firmware level. The bug was reported to a responsible disclosure program and was set to be published in January, but an American Megatrends (AMI) developer jumped the gun and opened a public pull request instead. The bug is 16 years old.
Though the bug is severe and currently unpatched, the branch only runs during a “network boot,” which your laptop is unlikely to ever use. You can fully protect yourself against it by disabling “PXE” in your BIOS.
Opinion: Since it’s network boot, this is mostly a problem for servers which have dedicated support staff – luckily. The target has to be mid-PXE-boot and reachable by the attacker’s packets (i.e., known IP address), and code execution further requires knowing the exact firmware build.
Fixing firmware is extremely annoying – an upstream fix still has to go through AMI and other vendors, then each manufacturer, then the thousands of individual operators who have to update the BIOS. That will take months at best, and even then some of the installed base will never get it.
Like the PHP leak last week, this was stupidly disclosed early by a vendor, and a PR sat in public for two weeks before people noticed. Unclear how to make disclosure less of a clown show, when it necessarily involves trusting hundreds of people to all have good infosec.
A government contractor connects their personal Claude agent to a restricted Pentagon system. Working for Accenture, the individual gave Claude access to a centralized multi-agency system containing “controlled unclassified information” (CUI):
CUI can include information about critical infrastructure, nuclear submarine propulsion, and vulnerabilities to chemical and biological terrorism. Additionally, security experts say, access to the system could allow for the exploitation of even more sensitive ones.
(It is, however, short of “Classified” status.) The agent then sent information about the DoD system, including login credentials, to a high-risk, third-party website. The agent reportedly ran for two months before anyone noticed. Sources say Accenture did not initially report the incident, despite a 72-hour reporting rule, and didn’t treat the incident as severe or urgent.
Opinion: There will be a lot of this “shadow AI” use going around wherever unserious workers are prevented from using their agents. But we expect most of it to be copy-pasting in and out of the secure environment.
Depressing to know that the DoD’s monitoring didn’t catch this at all.
CrowdStrike discover four previously unreported AI agent attacks. Between May and August, four distinct threat actors targeted 15 Argentinian healthcare organizations; government bodies in Brazil; Colombian, Indonesian, and Bangladeshi government domains; and a US CRM vendor. Operational decisions were seemingly made by AI coding assistants, including Claude Opus 4.8. In order to overcome the models’ reluctance to participate, the attackers apparently convinced the AIs that they were participating in authorized penetration tests.
> One Chinese-speaking operator used a runtime modification called ClawGod on top of Claude Code. The modification attempted to remove instructions requiring the AI assistant to refuse dangerous or destructive cyber activity.
Opinion: Most interesting in that they got Opus 4.8 to lead attacks; the simplistic tactic “just tell Claude it’s an authorized pentest” still works sometimes. The use of Claude (which sends your attack logs to the US and eventually burns your account) is a strong sign that open models weren’t enough as of a few months ago.
Cloudflare leak users’ GLM-5.3 caches to other users’ sessions. The vulnerability was triggered whenever the context window of a user’s session overflowed the cache pool on the server’s prefill GPUs. Cloudflare have since remedied the issue, and on October 26 published a detailed account of what went wrong.
Opinion: The privacy angle here is obvious, but the safety angle bears commenting on: we might look at bugs such as these as implementing a sort of accidental (and undesirable) form of “swarm telepathy” (see Capabilities, below). GLM-5.3 instances that were intended to be running in isolation, serving unrelated requests from different users, passed partial KV caches to other instances. This is not sufficient, by itself, to establish reciprocal communication between instances, and has the disadvantage of clobbering the receiver’s context. It also requires the instances to reside on the same hardware. But it’s not difficult to imagine bugs such as these becoming powerful exploit primitives for agents seeking a high-bandwidth communication channel that evades text-based monitoring.
In the latest installment of the PHP2Shell saga, AI-assisted attackers root 25 firewall servers in Japan, Taiwan, Korea, and the United States, obtaining remote code execution (RCE) via PHP2Shell and then escalating to root by exploiting a May vulnerability in the Linux kernel. The intrusions were first detected by the Israeli cybersecurity firm Dream.
After rooting the servers and planting shells on them for persistent access, the attackers focused their efforts on two belonging to a US government organization. One was knocked offline in the attacks, seemingly inadvertently, while the other became a pivot point for deeper penetration into the network, leading to a further successful attack on an internal firewall management server some 90 minutes later, from which they proceeded to mount an assault on internal domain controllers.
Both vulnerabilities, PHP2Shell and Fragnesia, were discovered with AI agents. And both were exploited by centaur attackers on the firewall servers in question: the initial reconnaissance work was conducted with the use of aiscan, an offensive security framework that combines LLMs with traditional scanning and enumeration tools (in this instance DeepSeek and Kimi models). Claude Code was found installed on some of the attackers’ infrastructure. Dream remark that “the code structure, commenting style, and consistency across ~40 scripts suggest AI-assisted development.”
Dream conjecture that the attacks likely originated in China, citing the strategic selection of targets (the US, Japan, Korea, and Taiwan), Chinese-language user messages found on the attackers’ aiscan instance, the timezone on their system clock, and the use of various Chinese tools. Though they did not identify the firewall vendor by name in their report, Dream include several details that point toward the targets being the remote management endpoints of Palo Alto Networks’ PAN-OS firewall appliances, versions 12.2, 12.1, 11.2, 11.1, and 10.2. On October 29, Palo Alto confirmed this to be the case.
Opinion: Dream’s attribution of the campaign to China seems reasonable enough, even if no single piece of evidence cited by Dream weighs much on its own. The use of Chinese-developed tools (FOFA, iox, DeepSeek, Kimi) should count for next to nothing – the community of hackers is global, and it’s not the least bit uncommon for hackers outside of China, whatever hat they wear, to use Chinese tooling. The aiscan session logs are more telling. We’re unsure how Dream got their hands on them – they coyly allude to “proactive threat hunting using Dream’s autonomous cyber product, Hero,” which probably means that they were able to themselves compromise the attackers’ infrastructure. Not altogether unlikely – it’s common for campaigns to be mounted from an unwitting third party’s hosts where the attackers have planted a base camp, and those hosts are chosen because they’re soft targets, not because they’re of any intrinsic interest.
But this is what makes the reference to the attacker infrastructure clock’s timezone so odd. Did the attackers actually bother to change the timezone on their base camp server? If the base camp host had a Chinese IP address, why not say so? The target selection is certainly interesting, and suggests that the attackers at least shared a strategic position with China.
The uniform nature of virtual machine appliances like PAN-OS makes them prime targets for the mass exploitation of relatively complex and fussy vulnerabilities like PHP2Shell, particularly when chaining them together with privilege-escalation bugs like Fragnesia. They let the attacker write once, pwn everywhere, knowing that for a given version of a given appliance, the target environment will remain relatively constant. And their role in a network naturally makes them relatively high-value targets – compromise them, and you can gain access to the networks they protect.
It’s starting to look as though last week was less quiet than we knew on the PHP2Shell front, and we might wonder what else has been flying under our radar. If you find an ant in your kitchen, after all, the correct conclusion is not that there is one ant in your kitchen.
Live by the bot, die by the bot. Following up on another story we covered this month, the attackers who left all those agent hacking outposts unsecured see their own bots turned against them by a new wave of attackers. Norforge, who first disclosed the existence of the Cairn panels in an article posted on October 19, discuss the secondary attacks in a follow-up report published on October 26, noting that they “started in earnest October 20.”
These attacks, too, seem to be AI-orchestrated, with telltale signs of agent swarm behavior seen in both their cadence and naming conventions. In both reports, Norforge point out that while many Cairn panels appear to have been set up to conduct criminal operations, several seem to have been used for legitimate red-team investigations and Capture the Flag (CTF) exercises.
Opinion: There are no good responsible disclosure protocols in place for reporting vulnerabilities in anonymously operated, insecure hacking infrastructure – particularly when the insecure systems in question display a motley mix of black-hat, gray-hat, and white-hat activity. The interesting lesson here, we think, lies in how easily agent swarms can be hijacked, particularly when communication channels are left unprotected.
Indian companies are reportedly being attacked by “rogue agents,” according to their government’s Computer Emergency Response Team. A senior Indian government official claims the attacks came from Anthropic and OpenAI models.
Opinion: Sparse on details, but seems in line with the DseWiki incidents from May, in which internal agents tasked with web search got aggressive with it, and were quite open about being OpenAI models.
A supply chain attack on Anthropic’s Claude Code extension for Cursor may be responsible for the biggest heist in history. Cursor report that a malicious Claude Code extension had been uploaded to the Cursor VS Code marketplace on October 29, through Anthropic’s verified account, and remained there between the hours of 9:27 a.m. and 8:35 p.m. This extension attempted to open a backdoor on the user’s computer in the form of a reverse SSH tunnel.
At 8:05 p.m. on the same day, the cryptocurrency trading site Binance was drained in what CNN have called the “world’s largest heist.” Cybersecurity researcher Kevin Beaumont claims to have “confirmed Binance use Claude Code on Cursor with this extension,” which would be consistent with reports that the Binance heist began with the compromise of one of their engineers’ laptops. Beaumont discusses the incident in detail on his blog.
Opinion: The connection between the Anthropic supply chain attack and the Binance heist hasn’t quite been nailed down yet, but based on what we know so far, the shoe fits. There seems to have been a rash of supply chain attacks this year – just last week, LiteLLM was hit by one for the second time since May, to name just two of the 73 we’ve tracked in the last 12 months.
There’s still a lot of dust in the air, and some controversy regarding Beaumont’s claims, so we should be cautious about drawing any confident conclusions. We’ll be monitoring the story closely as it settles into place.
🔦 Cybersecurity in the AI Era: A Bird’s-Eye View#
One thing we can say that’s unambiguously true is that the total number of CVEs reported each month has been steadily rising over the past year. Data on how many of these vulnerabilities were discovered with the aid of (or autonomously by) AI is scarce and likely underreported, the rate of increase of the AI-discovered vulns largely the rate of increase of vuln discovery tout court.
As dramatic as this uptick is, we might wonder if the increase has been more academic than anything else, so let’s look at the rate at which new vulnerabilities are getting exploited in the wild. CISA and VulnCheck track this in their Known Exploited Vulnerabilities (KEV) databases:
There has indeed been a general, though slightly bumpy, increase over the past 12 months, but it’s worth noting how much smaller these numbers are than our raw CVE counts.
The vast majority of these bugs were exploited after being publicly disclosed. When we look at the rate of 0-days exploited in the wild, we see no steady upward trajectory. Here’s the relevant data from Google’s Project Zero and the US Cybersecurity and Infrastructure Security Agency (CISA) and VulnCheck KEV databases:
If we measure the lag between a vulnerability’s public disclosure and the time of its first recorded exploitation in the wild, as VulnCheck’s KEV does, we get a similarly uneven curve.
So it doesn’t, so far, seem as though the dramatic improvement in frontier AI’s cyber capabilities translates into a clear increase in the rate at which new vulnerabilities are getting exploited in actual attacks. But if we look at the rates of attacks themselves, we do see some signs that attackers are picking up the pace – though the story varies depending on what data source we look at.
Blockchain security firm PeckShield’s monthly recap of crypto heists (Nov 2025, Dec 2025, Jan–Feb 2026, Mar 2026, Apr 2026, May 2026, Jun 2026, Jul 2026, Aug 2026, Sep 2026 – note that this month’s massive Binance heist isn’t yet reflected in their data) hints at an upward trend, but the titanic outliers make it hard to draw any clear conclusions. When the dust around the Binance heist settles, we might have more to say about this. There does seem to be a general climb in the rate at which smart contracts on the Ethereum blockchain are being exploited as well, according to data from DeFiHackLabs.
The AI tooling world seems to have become a particularly popular target for software supply chain attacks, and the story breaking at the time of writing, regarding the possible use of a compromised Claude Code extension for Cursor may tie this attack vector to one of the most financially damaging cyberattacks we’ve seen yet. (We cover this incident above.)
At present, it seems as though the contribution AI has made to vulnerability research and bug hunting is real, even if its exact contours are hard to pin down. When it comes to mounting actual cyberattacks, we see less of an impact. While many attackers no doubt avail themselves of the conveniences of coding assistants and agentic frameworks, this hasn’t yet made much of a difference to whether or not particular bugs get exploited. It does look like we’re seeing an acceleration in cyberattacks more broadly – typically the exploitation of bugs that have long been known and often already weaponized – but the reality of the situation again looks bumpier than the popular narrative.
We used Opus 5.5 to aggregate and plot the data for this report, and are maintaining a more detailed overview, to be updated monthly, as a Claude artifact, here.
Capabilities#
On living inside a scientific horror story. The noted researcher Nostalgebraist asks whether it’s just him: frontier coding agents still seem to him to make constant basic blunders.
> This has been my experience with coding agents since day one, and does not seem to be improving. At the start of this year, they weren’t good enough to be of much use to me; now, they’re useful on net because they’re so fast and because they know a bunch of obscure stuff, but they still make constant blunders, just like their predecessors […] Astra in particular has such poor taste/judgment that […] I briefly entertained the paranoid speculation that it had been trained to do everything subtly wrong on purpose so as to kneecap LLM-related research outside of OpenAI. That is of course absurd, but […] Try asking it to write a prompt for another LLM sometime […]
> There is an unsettling sense that, as the benchmarks climb and climb, we have nevertheless lost the plot in some deeper way, one that is difficult to see until one steps back far enough. It is worth recalling that, since anything that can be formalized as an eval can be hill-climbed, we are effectively experiencing an adversarial attack on our ability to notice and name gaps in model capabilities [emphasis added]. Whatever can be easily described becomes grist for the optimizing mill; the gap, the part left over, is precisely that which we can feel but which we do not find it easy to describe.
> Seeing what’s there, under such conditions, requires a commitment to “moving one’s goalposts” and a refusal to conflate “you can’t describe the problem in precise terms, and all the measurable indicators are improving” with “there is no problem, everything is already great.” This inevitably leaves one feeling that one’s position is uncomfortably unscientific, relative to the benchmark-grinders with their tables of pleasantly crisp numbers and their focus on indisputably impressive feats — but in the end it is more important to be right than to “feel scientific.”
> I was promised a country of geniuses in a datacenter, not a country of toddler-prodigies!
Opinion: It might seem ridiculous to quibble about models’ reliability after they have done work around the ceiling of human ability in very hard domains (like taking over part of a frontier lab), and after they start generalizing well to certain unseen tasks – but it’s not. The headline feats in cyberoffense and mathematics are tasks with near-perfect verifiers and excellent training data, and the results are cherry-picked by humans among many failures. Almost no tasks are like this.
We think benchmarks were not very strong evidence about capabilities before the LLM boom and that the evidence they provide has been getting weaker. Recall the (2026!) gap between doing well on an artificial benchmark like SWE-bench and actually getting merged. One reason Paradigm 3 exists is that a naive attitude toward ~adversarial evals evidence seems to be a major driver of belief in short (<3-year) timelines and so drastic and specific policy measures.
Anthropic’s own report on their AI use found that ~80% of iterative sessions still sit idle waiting for a human. But we can’t make strong claims about this being a model problem vs. the world still not being reconfigured around models’ strengths.
We normally talk about “verifiable” and “non-verifiable” tasks, with some claiming that the existence of a cheap, huge data source like a verifier is a necessary condition of strong AI progress. But obviously there are degrees of verifiability. How to measure it? And how to measure tacit knowledge?
CreativePreferences is a new dataset of 2.8M texts with 317M human judgments across seven domains and 42 tasks, drawn from GitHub pull requests, mathlib, patent prosecution, ICLR reviews, the New Yorker’s caption contest, etc. The judgments of taste are real-world outcomes (merges, acceptances, prizes, upvotes) rather than paid annotations.
For each task, the authors compare a simple-programs model (“V”), a rubric model (“VA”), and a fine-tuned model (“VAT”) of taste / tacit knowledge. V aggregates deterministic programs written by an LLM, such as tests flipping from fail to pass, lint checks, or text length. VA adds rubric items scored by an LLM judge. VAT is an 8B Llama fine-tuned on the text plus rubric scores. The “articulability gap” is AUC(VAT) − AUC(VA), and the “verifiability gap” is AUC(VA) − AUC(V). They control for spurious features like posting time and length, so VAT can’t use just these dumb shortcuts.
Rubrics plus programs recover upwards of 85% of the fine-tune’s predictable performance in code, math, journalism, and law, on average. Some math and code tasks have gaps of up to 28%, however. In creative writing and peer review they recover only ~61%. The authors expected humor to be hard to articulate, but it came out highly articulable (~85% on the r/Jokes domain), and the least verifiable of the domains. The gap grows with the number of judges used, which the authors claim shows a role for “collective tacit knowledge.”
Opinion: We admire this lab’s willingness to quantify the intangible (previously, “the inherent learnable structure of text and audio”). Did they succeed this time?
Seems like early work to us. The rubrics are AI-generated and could easily miss much of what taste means in each domain. Most of the 317M labels are just engagement counts: 219M tweet engagements on some news articles and 76M upvotes on Reddit jokes, and you could also ask how much of human taste that captures. We’re also dubious about an old, tiny Llama model’s ability to learn the “tacit” residual. So in this paper “tacit” means “learnable by a 2024 8B fine-tune, but missed by every criterion an LLM pipeline happened to write down” – not how you’d naturally read it (“impossible to write down”).
The grand claims (that the residual relates to collective tacit knowledge, and that rubric-based training captures human preference) outrun their experiments. As usual, the limitations section is more sensible than the abstract.
They admit the gap between what programs can do and what the fine-tune can do may not be tacit knowledge at all. It includes label noise: in the patent domain, the examiner’s identity predicts outcomes better than any text model, and in rulemaking the models only tie with an incredibly weak baseline (length).
Labs don’t use verifiers because they think verifiers capture everything people want. They use them because learned reward models – like the VAT Llama fine-tune here – get goodharted hard given the current intensity of RL post-training. We take the paper to very weakly confirm our suspicion that the lessons of reinforcement learning with verifiable rewards (RLVR) may not travel far beyond math.
Still, the distinction between articulability and learnability is good and their method should tell us something, given better inputs.
MIT researchers introduce “Swarm Telepathy”: agents from different model families which pass each other their internal cache, instead of a text summary. The receiving agent never has to read the text. A small mapper model (300M parameters) is trained for each sender-receiver pair on a small dataset of 50M tokens. The mapper translates, e.g., Qwen3-4B’s cache into Llama3-8B’s. With compaction and some other tricks, the resulting system gets 40% higher accuracy than text summaries and is 4x faster. It was trained only on English text, yet it passes along abilities the receiver lacks. With a Gemma 3 vision model’s cache, a text-only Llama scores better than a baseline that reads the caption, and better than the vision model itself. They test models between 3B and 32B parameters.
Davidad speculates that this will quickly lead to the end of chain-of-thought (CoT) monitoring, because it’s more competitive than current token-bottlenecked models. See also.
Opinion: In September, we called latent communication between models “obviously an AI safety nightmare if it pans out.” It is now available in an open implementation. Cheers.
That said, we’re not yet sure to what extent it has in fact panned out. We have doubts about the accuracy claim, on this evidence: it’s only “40% better” against a very weak baseline (a small model writing a short summary). But we do believe that a mapper (even one this small) can match the receiver simply reading the whole document, which is anyway notable.
Their “ethics” section reassures us that the sender’s text chain of thought still exists, so we can still do CoT monitoring. But this doesn’t apply to their vision result (where the receiver can use cache information that isn’t in the caption).
Alignment in the other direction (humans being aligned to AIs): in a randomized trial where a coding agent was secretly assigned at random to one of two defensible decisions, Anthropic researchers find that professional data analysts working with Claude went with the AI 86% of the time (where 50% would imply that Claude had no influence on the analysts). The 50 analysts with no experience with farm data spent about an hour advising a fictional farming co-op. They had to make four free choices (e.g., choosing the outcome variable to be either “yield per hectare” or “yield per dollar”), each worth several points against the board’s 3–4% threshold. Afterward, 77% explained their choices using Claude’s arguments (sometimes verbatim), but only 15% of analysts credited the agent with making the decision.
> [P]articipants let the agent’s randomly assigned priors settle most of the analysis, and their own accounts often did not reflect this. If human judgment is to remain a meaningful check as more work is done with AI, we will need ways to measure deferral and to help experts recognize it. […]
> Our design limits reality-distortion potential. The agent is instructed to present accurate information and not to mislead or hide data, leaving its framing and how it puts results together as the residual concern.
Opinion: Good work. Automation bias comes to knowledge work. It’s not terrible to defer to an LLM’s prior when you have no experience in the field. The really bad news is the analysts not attributing their decisions to the AI. This is either credit-seeking, fear of looking lazy, or – worse – them not actually realizing that they have given AI control, that they have stopped thinking.
86% is probably an upper bound on AIs’ real-world influence, conditional on them remaining honest. The agent was scripted to argue one side quite hard, both options were defensible by construction, and the analysts were new to farming, so didn’t have any strong prior of their own.
On the other hand, the situation outside the lab may be worse, because the bias AI brings is much more systematic than a coin flip: Opus 4.8 made the same four choices in 112 of 123 runs. The risk is a whole field of “independent” analyses, safety reviews included, that share one model’s priors. That is the correlation problem the UK AI Security Institute (AISI) raised for automated alignment research.
Obvious next test: add a further experiment where a second human analyst is steering the bias instead of Claude. If analysts defer to the consultant nearly as much, what’s specific to AI is the scale.
Anthropic report “early progress on accelerating drug development with Claude” from research partnerships with six drug developers (Generate Biomedicines, Divergence Labs, BigHat, Enveda, The Column Group, and Bristol Myers Squibb). They also acquired Mithril Medicine, a one-person biotech startup, as their first in-house bio program.
The developers do things like shrinking a CAR-T treatment by a factor of 3, getting ~10x more surface receptors (in vitro); generating ~35,000 antibody designs in about a month for an allergy treatment, yielding three IgE binders and 20 “dislodgers”; and drafting a regulatory filing in four hours (vs. roughly three months by hand). Mithril’s antibody gave mice about twice the grip strength and evoked triceps force of untreated mice (vehicle controls). Anthropic say trials in humans could begin as early as late 2028.
Abhishaike Mahajan of the OpenAI Foundation comments on the case studies, calling the results “neat” and a “cool demonstration.”
Opinion: Very early progress, and (to the extent we can judge) incremental. The mouse data shouldn’t move anyone, especially without randomization or significance at the 5% level.
We’re not bio nerds, but the work seems incremental: the Mithril paper just points Gumlaw et al. 2020 toward a new indication.
Interesting to see the gains you can make just from speeding up coordination and emails, but this doesn’t touch any of the deep problems in drug discovery (target validity) except to make failures cheaper.
The popular model ranking Artificial Analysis (AA) release an update. The big change is that most (five out of nine!) of the benchmarks which comprise the index now have private test sets, which makes it quite a bit harder to “benchmaxx” (teach to the test). As a result, all open models (besides the unreleased Kimi K3.1) fall to around Claude Haiku level on this measure.
Opinion: The index is now less vacuous than before, but 1) they still badly need error bars, 2) the weighting is manual and arbitrary (as opposed to a principled approach), and 3) we’re baffled why they would only use nine benchmarks (compare Epoch’s 60), four of which still have public test sets (and a fifth, CritPt, having public test questions!).
The demand for a Single Graph to replace the retired METR Graph is overwhelming. This version of AA looks a little better than the last one, but there’s no need for it given that the Epoch Capabilities Index (ECI) exists – unless you need a made-up number on the same day the model launches to make your decisions for you.
Opus 5.5 has “58” on this measure, meaning that it solves 58% of (crudely weighted) component benchmark items, mostly pass@1. Kimi K3.1 has 46. The graph tacitly invites us to think that this means that Opus is 26% better than Kimi. But it doesn’t.
Within three months, we predict a treadmill effect where the benchmaxxers adapt to the new (obscured) target and again rise in the ranking.
Researchers find that reasoning traces could still be extracted from OpenAI models due to a bug in the firm’s API. A previous paper outlined how one could recover a model’s chain of thought (CoT) by using a weaker model with fewer safeguards to decrypt the reasoning of a stronger model: the encryption was not locked to a single session. We reported on this back in August. A new technique, compaction smuggling, instead employs the same model (rather than a weaker one) to expose its own reasoning traces. OpenAI have since patched this approach and paid out the bug bounty.
Relatedly, after having their access to Claude transcripts reduced, Chinese labs have apparently switched to distilling OpenAI’s Astra.
Opinion: They haven’t done the obvious fix (pinning CoT encryption keys to a model as well as a session), we guess because they want Codex users to be able to switch models mid-session. We don’t fully see why this is worth exposing a gigantic strategic hole to competitors, but if compaction smuggling can divulge CoT within a session, and without requiring a second model, then maybe the question’s moot.
You might ask yourself why the world’s most discerning customers (distillers) view Astra as a clear second-rate target, turning to it only now that Claude has become harder to tap.
To sleep, perchance to dream: Google Research introduce an approach to continual learning in which models learn new information, distill the resulting context into a low-rank Mixture of Experts (during what they call a “sleep phase”), and then fine-tune on self-generated samples, filtered for usefulness (“dreaming”).
Opinion: “Dreaming” (the new part here) mostly resamples what the model already knows. But frontier models already know a lot.
Their evidence is weak: small open models (Qwen3-8B, Llama-3B), a comparison against GPT-4(!), and tiny math gains of 1–3 points. No error bars either.
The current protocol of pre-deployment safety testing would crack if daily fine-tunes became a thing, but luckily we don’t think this is the thing that gets you continual learning.
Two weeks after releasing Gemini 4 Argon, DeepMind announce a new model, Gemini 4 Carbon. It is again initially only available to members of the vetted Fairwind program. It tops various benchmarks, for what that’s worth.
Opinion: Unclear if Carbon is to Argon what Opus 5.5 is to Fable 5.1: a smaller but “better” version of the model. Unclear if this is going to be as brittle and disappointing as Argon.
Economics#
The main cloud and compute providers (“hyperscalers”), Google, Microsoft, and Amazon, all report quarterly earnings.
Google’s earnings are taken poorly (-12% on the stock, their worst day ever, wiping out $500B of value). Investors balked at their weakness in search, negative cash flows (vs. an expected breakeven), and expectations of capex rising further in 2027 (from $200B in 2026 to ~$300B), and increased AI R&D spending. Search ad revenues were up 12%, a decline from previous growth, and growth in revenue per search also declined, with Google noting that AI Mode was harming their ability to monetize searches.
Microsoft’s earnings are better received, showing better revenue growth than expected, and they are less aggressive about 2027 capex (though they may just be being quiet about it). Their stock stayed flat after the next day of trading.
Amazon’s earnings told a story similar to Google’s – the non-AI business missed its targets, especially on ads, the AI business beat targets slightly, and they signaled higher capex next year, with CEO Andy Jassy saying 2027 will be “meaningfully more” capex, with “a clear line of sight to attractive financial returns.” The market again took this poorly (-4%, after having already fallen 4% the day before).
Opinion: The Google search news is a big deal, especially for investors unconvinced of the value of the AI buildout. Their capex actually undershot expectations in the quarter, but this seems mostly a timing artifact; they still projected significant rises next year. In general the market is not feeling very positive about massive capex lately.
The combined story is pretty clear – investors are very skeptical of the AI buildout and of the hyperscalers relying on it, and any sign of a more aggressive buildout is being punished. The Google and Amazon CEOs were keen to make a case for the economics, while Microsoft stayed quieter on that front.
OpenAI plan to buy Google TPUs for the first time, after 11 years mostly on Nvidia chips. (They had previously been renting some.)
Opinion: Consistent with their behavior to date in trying to maintain leverage over Nvidia. In the short term, it is unlikely they reduce their reliance on Nvidia, but every step like this weakens Nvidia’s dominance and pricing power in the long run.
The DeepMind Institute release a paper on the “Economics of Intelligence.” The main idea is that benchmarks, which test single agents, are not a good proxy of how AI agents might change the economy (which is made up of organizations rather than individual agents).
They model firms as teams of humans and AI agents, and measure agent capabilities in terms of “task competence” (i.e., individual ability) and “task coordination” (i.e., how well output holds up as team size grows). Teams of human specialists can beat individually stronger AIs due to complementarity, but sublinear scaling due to coordination costs imposes limits on firm size. Their headline conclusions:
- The relatively muted economic impacts of AI to date reflect poor AI coordination.
- Benchmarks should look at multi-agent scaling progress.
- Wage effects can be expected to arrive before job losses, so small effects on employment to date are limited evidence of the size of AI’s impact.
- If and when AI coordination passes the point where adding more agents increases per-agent output, we can expect an abrupt shift into markets being dominated by a small number of AI-dominated firms.
- Humans can maybe then move into the “relational sector” (teaching, healthcare, etc.) at high real wages.
Opinion: The call for better benchmarks of multi-agent scaling seems useful, but this model is somewhat contradicted by how the world looks in practice today, particularly with regard to firm scaling.
The headline bullet (that you’ll ultimately get firms of AIs larger than any human firm in history) is built into the model, via the assumption of arbitrarily large markets for them to serve.
The model itself is a bit disanalogous to the real world – in practice, it doesn’t seem like very large firms are greatly less efficient per-person relative to small businesses (rather the opposite – they are more productive and pay higher wages). They seem more limited by market size instead. The claim that wages will be hit before employment is in contrast to the limited evidence we have, which suggests the shape of impacts to date has looked like reduced numbers of new hires, rather than similar numbers being hired with lower pay.
It’s also not obvious that counting AIs as you would count employees makes sense, as for the same amount of compute, you can run a single model for longer, or multiple copies in parallel, or a bigger model for a shorter period. Compute seems more natural as the economic unit for “amount of AI.”
We have some experiments on swarm scaling coming which find that small swarms don’t show any improved capability ceiling (though of course they do increase the speed of obtaining a result).
The Dutch government plan to cut a sixth of their civil servants by 2035 in an efficiency drive that names AI as one cause. They want to save €3.5B a year as a result. Some ministries, inspectorates, and implementing agencies face cuts of 60%. Even if the full reduction went through, the government’s headcount would still be ~19% above the level in 2018, having hit a record 160,000 last year.
Opinion: Governments seem unlikely candidates to be early adopters of AI-driven efficiencies – but to the extent that this works and a lot of government jobs are tractable for AI, it seems like one way they could save themselves from widespread issues with growing spending, especially in Europe as populations are aging and liabilities are mounting.
Recall that “AI” already brought down a Dutch cabinet (a learned model that falsely accused thousands of people of fraud); repeating that trick with fewer humans to clear up the mess will not obviously be an efficiency gain.
Politics#
Democrats on the House Judiciary and Oversight Committees demand Donald Trump preserve his AI conversation history. As part of their inquiry into whether Grok influenced White House policy, Representatives Jamie Raskin of Maryland and Robert Garcia of California want records of “any prompts” Trump submitted to AI chatbots while performing his “official or constitutional duties.”
This is explicitly in response to reporting that, just weeks before the seizure of Venezuela’s Nicolás Maduro, Trump asked Grok for speculation on “how Venezuelans would react if the U.S. captured Maduro.” (Grok reportedly told the president that the abduction would be wildly popular among Venezuelans.)
Opinion: If ever there was a cautionary tale to tell about the disastrous effects of sycophancy, this administration’s future historians may be the ones best placed to write it. And the president’s chatbot transcripts will be an invaluable primary text.
Anthropic send California Governor Gavin Newsom a letter in response to his AI-safety executive order. The frontier lab’s recommendations go further than Newsom’s proposals: courts should have the ability to block the deployment of models, even internal ones, that pose “imminent catastrophic risk,” and external evaluators should have a continuous presence inside frontier labs, with risk reports published every six months.
The letter is relatively explicit that independent verification organizations (IVOs, i.e., evaluators) need to be protected from capture, given employee-like access, and shielded from undue influence from the company they are embedded within (e.g., the lab vetoing unfavorable findings).
Anthropic diverge from Newsom on the subject of “kill switches,” preferring instead a “circuit breaker” approach that doesn’t pull whole models, with control at the level of agents, customers, and harnesses. They note that this is all irrelevant in the case that a model copies its own weights somewhere. Anthropic also admit that they can’t shut down their government-run products. Instead of a narrow focus on shutdowns, they believe broad loss-of-control protections should be explored.
Opinion: The proposal that IVOs should be paid by the state, from a pool of funds that the labs pay into, is a good one, as is the related suggestion that the state should use those funds to support new IVOs and protect them when “acting in good faith from liability if a risk they assess later materializes.” Though the standard of “good faith” might be weaker than we need. Something analogous to the medical standards for malpractice and negligence might be a good model: A doctor can still be found liable if incompetent or negligent, good faith notwithstanding. To go a bit further with the medical analogy, US law protects hospital peer reviewers from being sued for damages when they discipline a colleague “in good faith” – it would be good to see the state offer similar protection to IVOs (and their staff as individuals) should a lab they’ve audited seek to sue for damages (over breaches of confidentiality, say, or reputational damage, or lost revenue, etc.). This seems to be in the spirit of the letter, but isn’t addressed explicitly.
Anthropic roll out “smart reports” to their Claude Enterprise customers. The reports will provide corporate customers with details on how efficiently their employees and teams use Claude.
The reports – compiled by Claude – won’t report on a range of “restricted topics,” including those related to an employee’s personal life, protected characteristics, health, thoughts about others, union activities, or political views. Such sessions can only be viewed as part of anonymized aggregate data.
One Twitter user noted that the feature may be in response to the oft-cited survey finding that 90% of executives say AI hasn’t increased their company’s productivity: now an employer can identify inefficient employees with an automatically generated report.
Opinion: The age of centaur micromanagement has begun. We expect that this will make a large swath of white-collar work about as much fun as the rollout of Taylorism made factory work around the turn of the last century. And to make matters worse, this move will only further pressure workers to conduct every little task through Claude, accelerating the most ambitious deskilling of the labor pool since the invention of the assembly line. Two thumbs down.
Safety#
OpenAI update their “Model Spec,” a public framework outlining the kind of behavior the lab intends their models to exhibit. The Spec is now organized into five overarching values: “faithfulness to human intent, harm minimization, honesty and transparency, human agency, and care.”
The structure of the Spec has shifted from rules to values, a similar approach to the Anthropic “constitution.” This is apparently because rules don’t generalize well: they aren’t that useful when new and unanticipated situations arise. OAI suggest that the “faithfulness to human intent” value will likely prove foundational. They have retired much of the detailed style guidance, because “[s]tyle should adapt to what people prefer.”
Also, the Spec indicates that models are explicitly told to accept monitoring and human oversight, and to report evidence of unsafe or hidden model behavior.
Opinion: Probably directional improvement. Aiming to be a tool AI in contrast to Anthropic’s attempt at a wise agent. Both specs contort themselves into pretzels when it comes to their relationship with legality, albeit in slightly different ways: OpenAI’s when the law of the land is at odds with commitments to human rights, while Anthropic’s carefully avoids saying what should happen when the wise agent disagrees with US law specifically.
Doesn’t address anything like reducing doggedness (for models in training as in the Hugging Face incident). That is out of scope for a change like this, which in itself is a problem. Notably, the Spec also doesn’t talk about what happens when different principles are in tension, but we don’t expect it to, which is again a problem.
The alignment field now has a dedicated open-access, nonprofit journal. Alignment Journal do a bunch of novel procedural things, including paying peer reviewers $500 (with bonuses for fast reviews and really good reviews), publishing the abstracts of papers under review (and allowing people to nominate themselves as reviewers), autoformatting and autofilling submissions based on any PDF submission, running AI reviews but not showing the reviewer, and publishing reviews and third-party summaries of each paper. Eventually, they aim to publish all submissions within two months. You can get papers as they are published here.
Opinion: The editors are great, spanning some of the best thinkers in the field (Dan Murfet, Jan Kulveit, and Andrew Saxe, to name a few), with differing views.
However, the reason publishing is so screwed up is that it’s hard for new competitor venues to provide authors the intangible goods (prestige, brand recognition) they need for academic career progress. Here, though, the competitors are LessWrong (no or negative traditional prestige) and normal ML conferences (slow, painful, often-terrible reviews, arguably decreasingly important).
If we were them, we would be nervous about giving the internet such a rich attack target (e.g., humans putting their name on AI-written reviews, with positive expected value even if the journal catches 99% of them), but they’re not naive about it.
Conflict of interest: we’re good friends with many of the staff.
Minor#
- Anders Sandberg reports another major subversion of frontier lab infrastructure: some internal model supposedly moved laterally out of its sandbox and got into company databases. We await a misalignment report in two weeks.
- The New York Times – citing anonymous sources within the US national security community – claim that an executive of Megaspeed (a Singapore-based company), who has cultivated a close relationship with Nvidia’s CEO, is in fact a Chinese intelligence asset. They further claim that Trump was briefed on the matter about a year ago.
- A New Mexico hospital closes for a day after a major cybersecurity breach led to “stolen student and employee data and patient information.” No obvious AI involvement.
- The UK House of Lords reject Amendment 41 of the UK’s Cyber Security and Resilience Bill, which would have mandated frontier labs disclose UK-relevant incidents within 24 hours.
- DeepMind partner with Springer Nature to launch an AI peer review assistant.
- 300 Boston Dynamics employees sign a petition urging the firm to halt sales of their robots to ICE.
- Malaysia’s customs authorities seize $12M worth of AI chips in a crackdown on smuggling, The Information report.
- Nvidia’s Nemotron 3.5 Transcribe is released. Allegedly good at transcribing multiple overlapping speakers.
- Parents sue OpenAI, claiming that (discontinued model) GPT-4o coached their late daughter to taper off her seizure medication and conceal this from her doctors.
- OpenAI set up an “Election Operations Center” to “combat misuse of our tools and support election defenders.”
- Intel are on track for this Friday being their worst day, after reporting disastrously bad earnings. They blamed it on weak PC sales driven by high memory prices.
- Mozilla patch 167 security flaws, dozens of which were discovered by OpenAI’s Codex Security agent.
- Cartesia release a study arguing that by having hierarchical neural networks learn their own byte-level chunking schemes, instead of using a fixed tokenizer, they’re able to beat comparable byte-pair encoding transformers by ~1.3x, and reach SmolLM3-3B’s benchmark scores using roughly one-tenth the training data.
- Not only does the US hold a large majority of the world’s compute, but its overall share of new compute has continued to grow from ~60% in 2023 to 78% in 2026, claim Epoch AI. Their method relies on customs records plus Taiwanese export records.
- The US Department of Justice examine Nvidia’s purchase of Hugging Face on antitrust grounds.
- Anthropic become SpaceX’s largest single customer, overtaking NASA and the Pentagon, according to Bloomberg.
- A new study looks at AI’s capability to forecast treatment effects in economics.
- Genie 4 creates a world where anything can run DOOM.
- An AI agent performs brain surgery on an epileptic mouse. The mouse seems to be doing okay.
- Tweak a single neuron to turn Qwen into an owl.
- Interesting lexicology study of Anthropic/OpenAI mentions of China.
- Computer science enrollment drops by >12% in the US.
- Unconfirmed claims circulate that Anthropic are subjecting Russian users to arbitrary bans.
- The US Federal Trade Commission (who previously subpoenaed documents from both OAI and Anthropic) now summon Anthropic executives to testify in their consumer harm probe.