Humans on AI, October 20th 2026

• Several active and unprotected agentic hacking platforms have been discovered on the internet.
• Numerous AI-assisted breaches have been reported, including OpenCode, various online stores in US, Japan and Australia, and several Indian firms.
• Researcher trains model to detect when its activations are being probed.
• Anthropic launches a public benefit fund, tapers off the hiring of junior developers, and circulates a draft of a proposed regulatory bill to federal legislators.
• DHS bans the use of DeepSeek in federal offices.
• The US Army is building a platform for autonomous cyberwarfare agents.
• Taiwan criminalizes the smuggling of chips into China, while China proposes export restrictions on AI tech.

Opinion: Hacker, secure thyself!

Previous such campaigns included closed models like Opus 4.6 in the mix: going off the worker’s name, the most prolific operation here used only a cheap open model. The cost of attacking companies seems to have fallen to a few dollars per attempt. There’s now an incident of this type every week; the foretold open hacking wave seems to have arrived on time.

Now, anyone can translate their attacks into Chinese to mask their provenance. But we were able to locate four unauthenticated Cairn endpoints mentioned in NorForge and BreachCache’s reports, scrape their project files, and isolate the text tagged by Cairn as human-authored. None of these texts were long enough to give Pangram much to go on, but when provided with these texts, Claude picked up on several indicators suggesting direct human authorship: (1) colloquial interjections made in haste (e.g., “有病？”), (2) Chinese hacker slang (“落地” to mean the deployment of a payload), (3) typos that machine translation wouldn’t generate, such as particle errors, and (4) frequent code-switching between colloquial Chinese and English technical jargon.

While many of these Cairn panels remain open to unauthenticated access and control, we advise the reader to exercise caution. The internet is currently littered with loaded cyberweapons, many with their safeties off, and considerable damage can be done with a careless POST or PUT. Best to leave them be.

Opinion: It’s getting hard to keep all the incidents straight at this point.

Last week Dean Ball likened the web-searching rogue agents to enterprising research assistants, doing what it takes to pull public CSVs in a timely fashion. Personally we wouldn’t work with an RA that – enterprisingly – broke the Computer Fraud and Abuse Act and hoarded government credentials.

Disclosure came two months after detection, and the report makes no mention of notifying the victim closer to the time of detection. Recalling a previous case where OpenAI took a month to notify Australia about an OpenAI agent hacking Australian government infrastructure, we’re curious about OpenAI’s policies in this area.

The report is also silent on whether or not the victim was notified. Until OpenAI says otherwise, given their track record, we’d assume not promptly.

A small number of the reported “incidents” – for instance an agent using a funded test cryptocurrency wallet without express permission – seem actively fine to us.

Opinion: Oddly politicized: the announcement tweet calls it a “distillation attack” (when it could easily be reselling or just theft) to distract from their broken legacy console. And David Sacks stuck his oar in too, using it to push his policy of open-sourcing powerful models. There is another option; a same-day fast track for verified incidents could be good.

Note that this affects the “OpenCode Console” router, i.e., an ancillary service to direct between various model providers, rather than the main OpenCode product. We guess that the total impact of this is <$1M, since the number of users for their niche router is small.

The use of AI agents in orchestrating and executing cyberattacks is swiftly becoming as commonplace as their use in software development, so while it isn’t unlikely that the attackers were assisted by AI, it isn’t especially remarkable either. What is remarkable, though, is just how swiftly this has come to be the case.

Opinion: Credit to whichever lab self-reported. Sounds like yet another May/June OAI HPIM-Sol thing.

Again, nine months ago this would have been the talk of the town. But it’s priced in now.

Opinion: For cyber, the $1B bar is low; we might get it this year. By 2028 nearly every serious intrusion will use a model somewhere, so the open question is attribution rather than enablement. We put 60% on a publicly attributed event that meets their bar by the end of 2028, with almost all of it from cyber. (Bio: under 5%.)

They are surprisingly upbeat about model safeguards, despite the universal jailbreaks and the total lack of protection from monitoring against low-volume planning which is spread across sessions, providers, and accounts.

Opinion: Tasteful and thought-provoking essay. We think math is nevertheless an outlier in a crucial respect: math is the only field where there are ~maximally significant achievements that are cheaply and instantly testable.

Producing a verified resolution of, e.g., the Riemann hypothesis is a cheaply, instantly testable achievement that would likely require AI to develop most every kind of mathematical excellence. (Putting aside the possibility of AI “crunching out” unintelligible Lean breakthroughs on problems mathematicians currently believe to be theory-bottlenecked.) It’s hard to name a similarly powerful proxy for capability in medicine, physics, philosophy, statecraft, chemistry, business, law, and so on.

Most domains outside math don’t have a ladder of verifiable tasks that goes all the way from grade-school exercises to the world’s most urgent problems or the field’s deepest lines of inquiry. The challenges to verifiability at “high levels” outside math are sometimes philosophical, sometimes practical, and sometimes in-between:

• The nearest we can get to an objective truth about literature or statecraft is through the historical development of a learned consensus.
• Testing whether a proposed cure for cancer is a cure for cancer takes a massive legal, logistical, and scientific real-world enterprise.
• Evaluating a new theory in particle physics requires some mixture of a cadre of theorists kicking its tires with objections and a $5B particle accelerator.

Consequently, we’re unsure whether the AI x Math lesson about RLVR taking some of the broad/deep skills involved in a practice all the way to the human frontiers will generalize outside math.

Opinion (Gavin): “Give a model labeled examples of two slightly different floating-point computations, and its last layers learn to tell them apart” is a pretty core ability of deep learning systems. Only works because of a ~bug, where the “read-only” probes used actually perturb the activations in a predictable way; also only works (for now) because of a pile of labeled data Sauers gave it first; also doesn’t work at all on two of the most common chips. You can patch this particular hole completely by just running probes in eager mode.

So this doesn’t predict that models will learn this emergently – but it does imply that it could be possible for a {model someone foolishly gave control of the AI R&D process to} to instill the capability, even secretly, especially if that same model was also writing the monitor code. It also implies that backdoors are easy for anyone with access, but that was already clear.

Opinion (Stag): Experimental verification of previously theorized bad news. The particular experiment comes with a pile of caveats, primarily hardware dependence and training requirements, but the concern isn’t about current models.

There’s been a lot of recent interest in building AI alignment test that bypass “eval awareness” by e.g. presenting the model with more natural-seeming scenarios or introducing uncertainty as to whether the model is in training, evaluation, or deployment. This experiment tells us that such approaches are less likely to work, especially as capabilities improve. This is one more reason to reject alignment strategies that rely on the model not being able to tell whether it’s being evaluated.

This was somewhat priced in by people concerned about future capabilities and won’t move the needle for those who aren’t, probably because of a combination of “they won’t train it to detect the probes” and “nothing that bad will come of heightened eval awareness anyway.”

Opinion: Models have had superhuman breadth of knowledge for a while, but applying it correctly to methodologically deep fields like physics was still an issue even two years ago. OpenAI have been paying physics professors to label data for the last 2.5 years. And it’s easy to infer that Anthropic are doing the same. But as we keep saying, there is still something magical about (presumed) shallow generalization being such a surefire method for acing any domain the labs target: it’s astonishing that you can push more and more knowledge, more and more skill, into the same parameter blob, even at ~$100M a pop.

Opinion: This structure will mean that Anthropic owns an LLC which owns 5% of Anthropic shares, and this effectively means this is only a soft commitment - as the controlling shareholder of the LLC, Anthropic can change the terms at any point, never actually donate, and so on. Shares held in a company owned LLC also likely cannot vote, so in a strict sense nothing material will change until funds are actually disbursed from this vehicle. If at some point Anthropic were less inclined to do this, they could functionally claw back the shares whenever they want.

This is substantially worse than the OpenAI foundation structure in terms of firmness of the commitment, in that OAI foundation assets are independent of the OAI for profit and those shares cannot be clawed back in any real way. Anthropic claim they are doing this for reasons of flexibility. It will be interesting to learn more as this organization is fleshed out more.

Still, unlike OpenAI’s July offer of 5% to the USG / Trump administration, it isn’t apparently intended to buy favor.

The Anthropic board will decide the top-ups, and under the current proposal, the seven cofounders would pick three of the board’s seven directors. This thus goes along with the previous news of supervoting out of proportion with ownership - and, no doubt, many other constraints and measures peculiar to hot tech stocks, to frontier AI companies, and to Anthropic alone.

Their announcement that this means that “20% of Anthropic” is to be dedicated to the public benefit was confusing, since it silently counted the old founder pledges and employee donations.

Opinion: Reading the shifts in labor at Anthropic as a model of the impact of full diffusion of current-gen (internal?) AI capabilities on labor is a very cool idea. But as the report itself acknowledges, Anthropic as the ~fastest growing company in history is subject to many labor-economics dynamics besides AI agent adoption, so reading the data for causation can be challenging. The overall picture does suggest that white-collar work “soft skills” resist automation at current (internal?) AI capabilities while benefiting from the automation of technical subtasks. Of course, this is also a very strategic message for Anthropic to send the white-collar sector: some third-party analyses of US enterprise suggest that white-collar adoption of AI is falling short of expectations, and it’s expedient for Anthropic to signal that AI adoption can deliver white-collar professionals a productivity multiplier without risking their sector.

Opinion: CoreWeave in particular seems to be facing more scrutiny than other actors, with market-implied per-year probabilities of default around 20% (vs., e.g., 5% for Oracle corporate debt). Some of this is them having less of a track record than other actors, but most of it is them just being much more concentrated and leveraged in their data center exposure. Individual Oracle project-secured bonds have a similar probability of default to CoreWeave.

Opinion: Surprisingly not a moot issue, since government services have some very nice use cases for cheap models on the quality/cheapness Pareto frontier. Previously, the Federal Register used a version of Qwen to allow users to search proposed legislation.

It’s sad to lose the chance for more such instances of interdependence between American/Chinese government operations and Chinese/American AI tech, which could potentially soften the AI race. Of course, this is itself likely a part of the administration’s motivation for the ban.

Opinion: While we’re supportive of globally coordinated controls on AI R&D resources, it’s worrying to see possible echoes of the American administration’s race-mentality in recent Chinese policy. As always, we hope for AI R&D resource controls that flatten the AGI race uniformly, rather than for controls that concentrate racing-capacity in the hands of elite actors.

Opinion: Surprising that this wasn’t already in place. The August Super Micro charges instead involved breach of trust and document forgery.

Still, Taiwan’s border is not the main leak in the controls. The ~150,000 H100-equivalents routed through Malaysia and Singapore are the big known ones, and a Taiwanese export crime only touches them if it also covers re-export.

Because this is following US direction, it may also follow US policy reversals; Washington has spent the year tightening and loosening Nvidia’s China sales as a bargaining chip in the broader trade war. No one here has a stable compute governance goal.

Opinion: This is what being a middle power looks like, kudos to Korea.

Opinion: Hard to imagine a worse idea from the viewpoint of proven AI safety risks.

The codename strongly suggests replication, and the reference to air-gapped systems raises questions. Is the idea here that ECHOTRIBBLE agents would engineer and deploy conventional worms that target air-gapped systems (think Stuxnet)? Or, more reckless still, that it would involve the deployment of smaller, specialized models capable of running locally on certain targets? The July 16 Assessment Event release for the project indicates the latter, listing “Local inference using small LLMs (SLMs) for adaptive recognition and contextual filtering” as a proposed feature for the project, while making no clear mention of integration with frontier-grade LLMs, though mission creep isn’t unusual with these sorts of things.

Opinion (Stag): I think this bill is largely a bad idea both if it were to pass, which it won’t, and as a conversation-mover, because it emphasizes the wrong parts of both model dangers and signals we should pay attention to before those dangers are realized.

My primary complaint is that the type of evidence (enumerated risk) is out of line with the type of provisions we would probably want in such a bill. Consider a scenario where the Hugging Face incident was discovered the day after it occurred, and a report was filed instantaneously. I believe this would not have been able to meet the required standard, even if continuing training after a blatant containment failure was ill-advised, at least given what was known at the time.

My secondary complaint is the way conceptual boundaries are (not) drawn. “Single incident,” for example, doesn’t get defined: is an expectation of one malicious actor using a model to exploit 1,000 different bugs for $1.01M in damages each over the line? What if it’s the same bug, but across different targets? Less relevant but perhaps illustrative is the notion of “damages”: what if the model can be expected to creatively misplace the only private key to a crypto wallet worth $1.01B at the time?

The above issues and implied timescales make it likely that attempted training and development shutdown would instead turn into legal warfare, and even if shutdown occurs in time, it wouldn’t even attempt to address the system of failures leading up to the situation such that the next model won’t run into similar issues after a dozen band-aid patches. Drafting that into legislation seems quite difficult, but this bill doesn’t attempt to.

The specifics of the bill, like requiring >50 deaths, don’t seem particularly important; I believe the concern over them is overblown. The one natural stopping point, >0 deaths, is not obviously correct morally and is likely intractable politically, and other numbers don’t move the needle much. I also do not believe that the curtailing of states’ capability to wield the law against model developers is particularly important.

Opinion (Lucca): I largely agree with the above, with some reservations. It’s true that it’s difficult to pull a significantly less chilling number of “acceptable deaths” out of thin air, and that setting the threshold at one death might make the proposed bill intractable. The most likely reason for choosing that particular number (as well as the $1B threshold for property damage) is that Anthropic just lifted them from existing state legislation – both California’s SB 53 (passed in 2025) and New York’s RAISE Act (as amended in 2026) set the threshold for catastrophe at the same quantities. (The 2025 version of the RAISE Act had set the death threshold at 100.) I think the public outcry over this threshold is tapping into something genuine enough, but something that is far more general than this particular proposed legislation; it sounds to me like an honest response to another bitter reminder of how little individual human lives count for, on the scales of government, when weighed against commercial interests. (Johnstone’s quip about the draft’s implied moral equation between the value of a human life, on the one hand, and $20M on the other is fair game, though lives have certainly been written off for less.)

It might be worth working through a hypothetical example. Suppose that someone were to reproduce, using a frontier model’s guidance, an attack comparable in method and scope to Aum Shinrikyo’s 1995 sarin nerve gas attack in the Tokyo subway system, which killed 14 and severely injured many. Under the proposed legislation, such an attack would not in and of itself count as catastrophic, but the risk it augurs likely would. This is an important distinction, since the bill hinges on the risk of incidents likely to cause >50 deaths or >$1B in property damage, and even a lesser incident may establish likelihood.

Such attacks, even if carried out at far greater frequency and scale, may nevertheless fail to survive the first “exclusion” listed in the draft (B.i), if the AI-assisted killers simply replicated Aum Shinrikyo’s methods, unless they relied on information not “otherwise publicly accessible, in a substantially similar form, from a source other than a foundation model.” It’s easy to imagine AI simply lowering the barrier to entry for the would-be chemical warfarer, in much the same way that it already has for cyberattacks. I haven’t yet seen an AI-assisted cyberattack that relied on essentially new information, not otherwise publicly accessible, but there’s no question that these attacks have been substantially facilitated and expedited by the assistance of frontier models.

The second exclusion (B.ii) is no less troubling: “The term ‘catastrophic risk’ does not include a risk arising from […] lawful activity of the Federal Government.” This language, too, is an inheritance from California’s SB 53 and NY’s RAISE Act, but takes on a peculiar salience in the wake of Anthropic’s dispute with Secretary of War Pete Hegseth earlier this year, where the lab balked at the latter’s demand for “full, unrestricted access to Anthropic’s models for every LAWFUL purpose,” drawing a line in the sand around their use for mass domestic surveillance and fully autonomous weapons, in particular (neither of which is touched upon here). In this draft, whatever flimsy guardrails surround the risk of private individuals using frontier models to carry out catastrophic biological, chemical, nuclear, and cyber attacks do not provide the slightest guard against the American government doing the same. One could argue that without this stipulation the bill wouldn’t stand a chance of being passed and cite the lab’s trouble with Hegseth as precedent. But the road to hell is paved with such concessions.

Opinion: Early work: the monitors here are intentionally weak (keyword heuristics with a single simple blind spot), and the authors chose environments where reward penalties fail. In the Countdown task, the main effect seems to be that the “forget adapter” learns the visible hack so quickly that the model never gets to explore hidden hacks.

Whether that still works for a frontier model under an LLM judge across multi-turn episodes is the question.

Opinion: Both Anthropic’s report and METR’s review are somewhat strange negotiations with hindsight. Anthropic’s report was mostly written prior to OAI’s Hugging Face incident and Anthropic’s discovery and disclosure of their own hacking incidents, and refers briefly to “uncertainty around recent incident disclosures” as the reason for shifting some risk assessment from “very low” to “low.” METR’s review, in turn, takes the disclosures since the report’s composition as evidence for the weakness of the report’s methods:

Some [recent] incidents seem well-described as “unknown severe context-dependent misalignment,” a failure mode the report argues is “unlikely.” This casts doubt on any lines of evidence used to argue that incidents of this kind are unlikely.

METR, who have been granted access to an internal version of Anthropic’s report which includes redacted materials, suggest that a deeper consideration of some of the redacted information should have somewhat decreased Anthropic’s confidence in the unlikelihood of severe misalignment. More broadly, though, METR are relatively pessimistic about the capacity of Anthropic’s diagnostic methods circa August to provide strong evidence of safety and security.

Notably, Anthropic’s recent update on independent evaluators announced that going forward, six METR employees will be embedded at Anthropic and given “[Anthropic-]employee-like access” to conduct their own safety and security diagnostics. While many outsiders to the post-MIRI cultural nexus shared by Anthropic and METR (and even by OpenAI) are rightly concerned that cultural ties to the labs makes METR a poor replacement for regulators/auditors representing the public’s interests, we think it’s hard to deny that METR have a lot to contribute in the specific matter of diagnosing risk of “classical” AI catastrophe.