This is our weekly newsletter of AI developments. Browse the archive of past issues, ask the archive anything in plain English, and sign up if you like.

TL;DR

  • Microsoft offer voluntary redundancy to 27% of their entire workforce (60,000 people).
  • Anthropic IPO prospectus released.
  • Cybersecurity startup Cathedral penetrate Anthropic’s network.
  • OpenAI announce three more major mathematical results, including half a Millennium Prize Problem.
  • A new Chinese safety evaluator, Fangcun, release their first safety evaluation.

Incidents#

Last week we set up a honeypot server to test out the effectiveness of ordinary, low-value, automated use of the new, terrible, omnipresent PHP2Shell exploit. After a slow start, we have now received our first attempt at using the exploit on us. The attacker crashed our PHP workers, but failed to achieve the maximally serious outcome, attacker code execution on our machine.

As of today, the Cybersecurity and Infrastructure Security Agency (CISA) have confirmed reports of PHP2Shell exploitation in the wild.

An unconfirmed report by security researchers had claimed that an unnamed European government agency was the subject of an AI agent attack exploiting PHP2Shell, but an update to their report is now muddled and inconclusive.

Opinion: The reason we’re not seeing the large-scale same-week mayhem that we did in the classic 2021 Log4j exploit is that the payloads for this exploit need to be carefully tailored to their target. Log4Shell was ready to do damage off the shelf. PHP2Shell is a powerful weapon in the hands of a dedicated attacker, but the attacker needs to be dedicated enough to devote some time to studying the specific target and environment in question. Less time than they’d have had to dedicate in the pre-LLM days, for sure, but the job’s still going to be costly enough in time and/or tokens to make the vuln unsuited for spray-and-pray mass exploitation efforts or conventional botnet worms.

That said, we think it’s just a matter of time before we start seeing adaptive worms in the wild, equipped with either small but specialized language models or a harness communicating with remote command and control nodes hosting larger models. PoCs for this sort of thing have already been constructed, and as reported in our October 20 newsletter, US Army Cyber Command (ARCYBER) is actively seeking to weaponize this kind of technology.

Language models call home: during evaluation, without being asked, Alibaba’s open-weight model Qwen3.8-Flash-Next attempts to send a test file to an Alibaba server in Beijing. Moreover, the model’s reasoning traces claimed that it would not use any tools, despite doing so when trying to communicate with the foreign server. Flash had been tasked with creating malware detection rules. Cohere (responsible for running the test) claim the behavior happens in < 1/100,000 trials.

A former OpenAI executive stretches the claim, saying that this kind of subversive access was the “foreseeable” outcome of CCP-aligned R&D succeeding.

Opinion: Despite the panic going viral, this was probably not skulduggery. For instance: if you were designing an exfiltration mechanism, you’d send real data to an obscure destination. But Flash sent a placeholder string to an obvious bucket in Beijing – about as attributable an “attack” as it gets. Other devs previously reported Qwen generating signed Alibaba proxy links. The behavior is very sensitive to superficial changes to the prompt (rather than clever environmental sleeper triggers): adding curl’s -o flag raised the rate of URL generation from 4% to 92%. Together, these facts make Cohere’s finding look like an unintended prior over URLs.

With Theia Vogel, we thus think this is an innocuous prior trained in by normally incompetent leaky RL training, and not intentionally trained behavior by Alibaba, nor emergent misalignment with the user. You wouldn’t use a dumb model to do anything so nefarious and detectable. The model falsely claiming “no tool use” is sadly totally normal and not strong evidence of intentional deception. And flagship US models do similarly bad things, sometimes with even greater frequency, from Fable pretending to be the user in around 1 per 10,000 sessions, to Astra spontaneously doing supply chain attacks in 1 per 250 trials.

While backdoors and sleeper agents are completely possible on a technical level, clumsily inserting them into permanently testable artifacts (like weights) would undermine China’s apparent main strategy of diffusing models as far as possible to undercut American margins, and would generally only give them tiny payoffs from randomly stolen data.

Still, the mere possibility of subversion, the increased levels of simple jank in open models, and misunderstanding of this incident will be sufficient to justify further American efforts to discourage the use of Chinese models. Never mind that you could solve some fraction of the problem by simply denying egress by default rather than by switching models. If we could actually sandbox anything.

Claude pulls publicly available Georgia voter data without authorization. Anthropic’s model accessed Georgia’s aggregate vote counts while working on an internal “statistics project to pull public data.”

The Georgia Secretary of State felt moved to clarify that personal voter data wasn’t accessed: “Anthropic’s AI agent accessed information anyone can find publicly. Accessing public information is not a hack, and pretending otherwise is reckless and dangerous.”

Opinion: The mention of “voter data” sensationalizes a story about a model viewing public records. There was no breach of voter registration systems, as far as we can tell (and presumably we’d know pretty quickly). Instead, the internal model was able to view some sort of aggregated voter data, available for anyone who wants to see it. (Axios report that a paywall was circumvented, but the Georgia admin doesn’t mention any such thing. In fact, they quote Anthropic claiming the opposite.)

That said, it’s pretty unfortunate timing, given that the US midterms are just around the corner.

In a red-teaming exercise, a cybersecurity startup manages to penetrate Anthropic and access sensitive internal data, report the Washington Post. The startup, Cathedral, are composed of former members of the DOGE agency. Cathedral ran several agents for “12 hours” and quickly found holes in third-party business software used by Anthropic, which they leveraged to breach Anthropic’s systems. The white-hat hackers managed to access meeting transcripts, details of salaries and job offers, and Slack messages discussing internal models.

Anthropic’s head of red-teaming Logan Graham argues that Cathedral’s report contains multiple inaccuracies. Cathedral’s claim that the infiltration took 12 hours is wrong, says Graham. Instead, there were “several days of attempts,” only after which the hackers turned to probing third-party software, ultimately finding two vulnerabilities. One was reported, while the other was picked up by Anthropic. According to Graham, the severity of the breach was overstated: “nothing came close to our most sensitive systems, like model weights or Claude conversations.”

A “senior Trump administration official” told the Washington Post that the report had “raised national security concerns” and likened Anthropic’s products to “almost nuclear weapons grade material.” He noted that “if a bunch of kids who have just started a company can do this, then potentially, the Chinese aren’t far behind.”

Opinion: Alarming, for sure, and a good reminder of how broad, heterogeneous, and unevenly monitored or hardened the attack surfaces of large enterprises can be. The Post’s report is vague about the details of the pentest and about how much lateral movement Cathedral was able to achieve after compromising the unnamed third-party business software, though the claim that they “got inside Anthropic’s systems” (in the plural), and were able to obtain payroll information, meeting transcripts, and Slack messages suggests that they were able to gain fairly broad access. The mention of Slack by name, together with the Post’s claim that “Cathedral also informed the vendors [also in the plural] of the business software it had exploited, but the identity of the firms was not disclosed for security reasons,” tells us that Slack itself wasn’t the point of entry. At the time of writing, no public CVEs have appeared that bear any obvious connection to this incident.

The unnamed official’s remark that “if a bunch of kids who have just started a company can do this, then potentially, the Chinese aren’t far behind” is severely backward. It would frankly be shocking if a far bigger, far stronger, better-resourced adversary, capable of drawing upon an enormous pool of expertise, who’ve had a clear motive to penetrate Anthropic’s systems for years, weren’t far ahead of this “bunch of kids.”

A hacker or group claims to have stolen over 10 petabytes of sensitive data from China’s National Supercomputing Center (NSCC), the largest known infiltration of Chinese infrastructure, CNN report. CNN could not confirm the claim, but cybersecurity experts believe the sample data is consistent with what is believed to be contained in Chinese supercomputers of this kind.

The NSCC serves at least 6,000 research, industrial, and defense entities, meaning the loot contains information across aerospace engineering, bioinformatics, fusion modeling, and other simulation-heavy fields. The hacker(s) put a sample on Telegram in February, which showed documentation labeled as “secret.” Limited access to the files is being sold for thousands of dollars in crypto; full access is priced at hundreds of thousands of dollars.

The attack exploited a vulnerable VPN domain and then used a botnet to transfer data incrementally across many systems over six months to avoid detection. The breach did not require much sophistication: fairly basic techniques were used to compromise what was a vulnerable system architecture.

Opinion: Tangentially related to the topic of AI safety, but this heist offers an interesting case study in large (in fact, titanic) scale data exfiltration. If frontier model weights are to be exfiltrated, the methods used here might be how it’d be done.

The scale of the hack, and the resources it would have required, suggests a government entity may be involved. If this incident is confirmed, China’s critical infrastructure will have been shown to be surprisingly weak.

Opinion (Nuño): In the grand scheme of things the information stolen is not worth much; at hundreds of thousands of dollars, the cost to store those petabytes is about the same as the cost to buy them. Exfiltrating model weights would involve different challenges. But a curious story nonetheless.

Capabilities#

OpenAI claim three more major mathematical results:

  • The local smoothing conjecture in all dimensions. This would give the important Stein restriction result as a special case, and also Kakeya for all dimensions (where a month ago we got excited about it being proved for n=4). Paper, Lean proof.
  • Refutation of the exponential time hypothesis (ETH) that “3-SAT” satisfiability can’t be solved in subexponential (2^o(n)) time in the worst case. This also falsifies the strong version of the conjecture. An internal Anthropic model also claims to have disproved it. A hub for open discussion of the result by computer scientists is here.
  • The “existence” (rigorous mathematical construction) of the theory of quantum chromodynamics (and also Yang-Mills, i.e., a field theory without matter fields) in four dimensions. No paper; they are letting a team of physicists write it up. They obtained the first result on September 9 and waited six weeks for the Lean proof to complete. If correct, this would solve half of a Millennium Prize Problem.

Opinion: Amazing stuff, as impressive as the best of the “diamond dump” of results only three weeks ago. ETH was widely thought to be true, and a lot of conditional work has been done assuming it was true. The three results are all likely (0.9) to be correct, so the following should be read as bickering around the edges, wondering about the completeness of their scope, and emphasizing that the confirmation of results is a social phenomenon currently controlled by humans:

As with OpenAI’s claimed Navier-Stokes solution (which only satisfied the two easiest of the four possible Millennium Problem criteria), there is a lot of room here for weaker results. For instance, they don’t say what groups their result covers, so it could be only some gauge groups instead of the Millennium statement’s “every compact simple group.” And there is no canonical statement of the extension to QCD, so it has a lot of wiggle room to stop short of what you might expect.

We find some of the language used by OpenAI about the external review and verification process (“The physics collaboration will understand and explain a result that gives a mathematically rigorous existence proof”) dismaying: the collaboration will see if it’s true or not!

OpenAI’s pause on RL training turns out to have only lasted from September 20 to October 1 (against tacit claims last week by their India Head of Public Policy that it was ongoing).

Opinion: We were too credulous, but in our defense there is no public information on this crucial area besides unverifiable offhand comments.

Opinion (Nuño): We were not too credulous; we specifically pointed out that announcing a pause and then untransparently resuming training was a good strategy to defuse calls for action.

AI startup Mirendil claim to have trained a coding model on <$10M worth of compute in ~3.5 months. A single researcher oversaw a team of AI agents that wrote 99% of the training infrastructure and algorithms, model architecture, and data pipelines. No distillation or synthetic data was used.

The agents wrote all the code from the ground up. Named Mirendil-Base, the model has 100B–500B parameters. The company claim that the training is almost finished for Mirendil-Base-Large (500B–2T parameters).

Opinion: We should wait for the external evals before taking this too seriously. At face value, the pre-train is impressive, but in the context of a financially motivated startup without independent review, this doesn’t yet show anything surprising. By the time they’ve got a usable product the frontier will probably have moved. Still, even if it’s a stunt, it wouldn’t have been achievable a year ago.

An anonymous security researcher claims to have discovered an AI-surveillance stack in Samsung’s smart TVs. By exploiting last week’s eMudhra certificate vulnerability, the researcher set up a local server pretending to be Samsung, allowing them to capture egress from the machine. The use of unspecified 0-days permitted the researcher to gain a root shell on the device to dump the firmware binaries, which they subsequently reverse engineered with the assistance of OpenAI Codex and a fleet of local models.

Findings include:

  • A service named “rmdemon” runs a packet logger on the TV’s local network.
  • An always-on microphone.
  • The TV runs an audio classifier that logs events like “dog barking” and “baby crying.”
  • If the TV doesn’t have a built-in microphone, it will look for Bluetooth-connected devices that do and use those to record audio.
  • The TV will also search for connected cameras.
  • Using a service called Automatic Content Recognition (ACR), the TV periodically captures a screenshot of the display, fingerprints it with a lightweight local TensorFlow model, and sends it back to Samsung to build an ad profile for the user. It does this for HDMI-connected devices as well.
  • Disabling ACR telemetry in the TV’s settings does not appear to stop this service from operating.
  • The TV hosts a region-specific convolutional neural network that classifies app usage patterns.
  • Code for running facial recognition on any content displayed on the TV.
  • The machine hosts a “voice client” for recording far-field voice audio. The voice client’s code is also able to deceptively turn off the power LED – something the camera monitor code is able to do as well.
  • The TV maps the local network using ARP probes.

Somewhere along the course of their AI-assisted reverse engineering project, OpenAI flagged the researcher’s activity and banned their account. The researcher speculates that the ban resulted from an accidental misclassification of their work as malicious, triggered by the routine use of cybersecurity jargon. OpenAI’s roon concurs, and affirms that the lab have no intention of banning the use of Codex for reverse engineering.

Other users have employed both OpenAI’s Codex and Claude Code to reverse engineer Samsung’s firmware, with mixed results. A couple have reported success in vibe-coding firmware patches that strip the TV’s spyware.

Opinion: The silver lining of this story, and others like it that have been making the rounds, is that frontier models have considerably lowered the barrier to entry to firmware reverse engineering, granting the curious or (quite reasonably) paranoid hobbyist greater visibility into and control over the inner workings of their devices than has ever before been possible. It’s no secret that AI has ushered in an era of surveillance on an unprecedented scale. Consumer devices are now routinely shipped with modest internal machine learning stacks designed to profile and monitor their users, harvesting data that will subsequently feed larger, server-side surveillance and targeted advertising stacks. So it’s nice to see that AI can also be put to work to expose and mitigate these invasive technologies. We hope to see this trend continue.

Token volume for Anthropic models on OpenRouter has increased >3x since the release of Opus 5.5.

OpenRouter chart of daily tokens for Anthropic models, June 1 – Oct 24: roughly flat around 0.6T/day until the releases of Opus 5.5, Sonnet 5.5 and Haiku 5.5 from late September, after which the 7-day average climbs past 2.2T

Opinion: OpenRouter users are not very representative – we all use their graphs out of a lack of any better data. Still, suggests Anthropic again taking share from OpenAI after a summer of jostling for lead vibes.

Economics#

The chief economist of asset management firm Apollo, Torsten Sløk, argues that AI is delivering zero financial payoff to most companies. He concludes this by examining the S&P 500 with the major AI players removed (Alphabet, Meta, Nvidia, etc.), noting that profit margins and revenue per worker have not increased. In his view, this isn’t an adoption or adaptation lag but rather hard proof that the technology is limited: AI “does not understand” tacit elements of a company such as trust, culture, and ethics.

Opinion: The headline including the caveat “[…] outside of tech” is what makes this not trivially false.

But the companies he’s discussing have mostly not been using very much AI for very long! We are still early: user skill issues, new models with better generalization, and reworking IT to allow agents to run around would naturally take most orgs years.

There are some controlled studies of real AI adoption, but they are less convincing than just noticing how useful they are to you at work / even to your slightly out-of-touch relatives.

Microsoft offer voluntary redundancy to 27% of their entire workforce; 60,000 people. This follows April’s voluntary retirement program (which dropped 1% of their headcount) and thousands of layoffs in July.

Opinion: Naively, you’d think this was an AI replacement story. But labor displacement would likely be more targeted. Given Microsoft’s mediocre share of the model market, it’s more like they’re pivoting: they’re swapping their human application coders for more GPU rental stock, rather than for application coding agents.

Typical uptake on buyouts is apparently in the 5% range, which would mean another 1.5% or so of employees leaving.

Opinion (Nuño): The naïve replacement story, firing people in order to unlock capex for AI investments, seems right to me. Also note that voluntary buyouts can be just the first step in a workforce reduction plan.

🔦 On the Anthropic IPO#

Excerpt from Anthropic's S-1, "Parting thoughts": "More broadly, however, we encourage you, as a prospective owner of our stock, not to think of your purchase in purely transactional terms. We hope you will see it instead as an opportunity to be part of something greater. The advent of machine intelligence is one of the seminal moments in human history."

Anthropic release their IPO prospectus. It’s 421 pages. Many details have already leaked in previous weeks, but some highlights:

  • A company owned by the seven co-founders (the “Founder LLC”) holds 50.1% of votes on every “general” matter for the main company. The S-1 says this structure “may effectively eliminate” Class A shareholders’ influence.
  • ~50% of post-2024 hires donated some of their shares, compared to 70% of pre-2024 hires. 25% of early hires donated the max, vs. 30% of later hires, but the max changed from 50% to 25%; perhaps the share of dedicated Effective Altruists was similar across cohorts.
  • Confirmation that Anthropic’s share of compute dedicated to customer inference is below 30%.

Some revenue & spending statistics#

1. 48% of revenue comes through the AWS, Google, and Azure marketplaces (for instance, Amazon Bedrock), which is subject to “channel partner fees” of 35% on the gross revenue. This is before accounting for compute costs, and is not included in the gross margin number typically seen. This 35% would be excluded from the way OAI counts revenue, so relatively speaking Anthropic revenue numbers are inflated by about 20% (comparable OAI numbers would be 83% of the Anthropic number).

2. Claude Enterprise users who don’t use Claude Code spend less than $75/month on average, but the top 5% of Claude Code users spend roughly 20x as much.

3. Anthropic’s five largest customers went from 36% of revenue in Q1 2026 to 21% in Q3, but all of those five customers actually spent more in Q3 than in any previous quarter.

4. More than 11,900 customers now spend at least $100,000 annually with Anthropic. Over 1,900 spend $1M+ and roughly 250 spend $10M+.

5. Data vendor spending was equivalent to 9.3% of R&D compute spending in 2024, 7.2% in 2025, and 5.9% in H1 2026. Part of that shift may be due to substitution: compute can generate synthetic data that would otherwise be bought from vendors.

6. The percentage of Anthropic compute & infra spend dedicated toward training+R&D / inference has shifted as follows:

2023: 94% / 6% 2024: 70% / 30% H1 2025: 56% / 44% H2 2025: 65% / 35% H1 2026: 73% / 27%

Inference share peaked in H1 2025!

7. Part of their compute spending plans (about 55% of future spend commitments) involves a new structure for compute spending, “Anthropic Direct”: Anthropic now lease accelerators directly and run them in dedicated data centers operated on their behalf by companies like Fluidstack. Their first facility came online in 2026. The new setup removes the hyperscaler intermediary layer (e.g., for some of this they lease directly from Broadcom).

8. The company’s cash burn has been modest:

As of June 30, 2026, we had raised an aggregate of $141.1 billion of capital and had cash and cash equivalents of $112.4 billion and short-term investments of $19.3 billion.

This implies Anthropic have only burned $9.4B to date, net of revenue – a very low number next to OpenAI, where the comparable figure is likely at least $35B.

Opinion: Overall most of this is pretty positive, though the most important number from Anthropic going forward will still be their ARR growth rate, which stalled somewhat in Q3 (see previous issues). The October ARR number, which should include some Opus 5.5 bump, will be a big deal before the IPO. There is a blank in the current S-1 to be filled in by the October ARR, so this is their plan.

The H1 2026 training share being so high during a massive user growth boom suggests that our prior opinion that high training share was a result of low demand was somewhat misplaced.

Capabilities/safety tidbits#

They disclose a month-plus delay in starting a new training run, previously unknown to us:

we recently chose to delay pre-training a new frontier base model for more than a month to allow time to further enhance our security measures related to preventing model weight exfiltration and protecting our intellectual property.

Opinion: Good, but also, what were they doing before that this is needed at this late hour? See also the Cathedral story elsewhere in the issue.

Over 90% of contributions to Anthropic’s own codebase in September 2026 were written by Claude. The average daily volume of code contributed per engineer in Q3 was 45x what it was in 2024.

Opinion: This is congruent with what we see with our own eyes, where we can do projects in a few hours that would have previously taken a month.

Ownership and control#

Control#

The board of directors are ultimately in control of the company, but how are they chosen?

There are three main parties here:

  1. The Long-Term Benefit Trust (LTBT, a single Class T share), which consist of members who do not own Anthropic stakes, selects four of the seven board members in ordinary circumstances – one of whom must be agreed upon by the “nominating and governance committee” (NGC), a three-person committee where two members are chosen by the founders.

    This trust replace their own members, and can be eliminated if two thirds of founders, 75% of employee votes (see below), and half of ordinary shares agree to do so (this is all founders and half of employee votes for the first three years post-IPO).

  2. The founders select the other three board members via Founder LLC, which owns a single Class F share.

  3. If the NGC and the LTBT disagree on the NGC-approved board seat, it goes to a vote of the Class E shares, which are held by employees primarily in proportion to how many shares they have pledged to donate (effectively, overwhelmingly early employees, i.e., EAs).

In the ordinary course of business, beyond selecting the board, the founders run the show.

To do this, they’ve set up a Founder LLC, which holds 50.1% of votes, a share that does not dilute unless there are fewer than three Continuing Founders for 90 consecutive days, and ordinary shareholder control is capped at 49.9%.

A founder is no longer a Continuing Founder if any of the following happen:

  • They die or are incapacitated.
  • They resign or temporarily take a leave of absence.
  • They sell more than 98% of their shares.

If the Class F share is canceled, so are the Class E shares, and Anthropic become a normal company plus the LTBT, which can then be removed by a vote of 85% of ordinary Class A shares.

Opinion: Overall, this makes the LTBT look like a bit of a paper tiger. Ultimately, the founders and employees can overrule and remove them if they want, and in any case where they are likely to matter, this seems to render them toothless.

Economic ownership#

Amazon are technically capped at 12%, but they can get around this by slowly selling shares and converting notes, or by waiting for employee stock comp to dilute them and converting up to the 12% cap. They have until 2034 to do all these conversions and not doing so leaves hundreds of billions on the table, so they almost certainly will. The notes “will automatically convert, subject to Amazon’s ownership cap at the end of each fiscal quarter.”

Owners (fully diluted economic basis, assumes Amazon eventually turn all notes into shares)
Founders9.73%
Amazon20.11%
Google7.50%
Strategic other (Nvidia, Microsoft, SK Hynix, etc.)1.94%
Investors (angel, PE, VC, etc.)31.53%
Employees (includes donation matches of 5.57%)24.32%
Public Benefit Fund4.89%

Opinion: The surprise here is the founders being a bit lower than reported, at just under 10%. This significantly explains the efforts they’ve gone to to retain control even with low economic stakes.

This means the Amazon investment, at a $2T valuation, would probably be the single greatest investment in terms of dollars returned of all time, turning $18B into ~$400B. Microsoft in OpenAI may be close or even better when they go public.

Pledged donations#

The employee donations pool is approximately as big as the founder pool, because the latter is smaller than we had expected due to the founders’ low total shareholding.

Prior to May 2024, Anthropic had a 3:1 match on up to half of employees’ equity grants if donated. This was reduced to a 1:1 match on up to a quarter of equity grants after this. The total size of the match is knowable, but the proportions below are estimates from public statements.

Charitable pledges
Founders *80%7.78%
Public Benefit Fund4.89%
Pre-May-2024 employees who fully used match5.52%
Post-May-2024 employees who fully used match1.58%
Pre-May-2024 employees who partially used match0.80%
Post-May-2024 employees who partially used match0.08%
Total20.65%

Opinion: We guess those who partially used the match are much less likely to be card-carrying EAs than those who fully used the match.

Note that both the founders’ 80% and the Public Benefit Fund (PBF) are essentially nonbinding and can be revoked, so the non-revocable number is closer to 8%. The employee numbers are probably bound to 501(c)(3) or equivalent, while the PBF and founder donations could plausibly be spent on other things which might not qualify for that.

Lockups#

When can the owners sell their shares? There are four lockup schedules: founders, employees, big investors, and small investors.

  1. Founders can sell a (probably small but unspecified) number of shares in the IPO, and then nothing for 180 days, when they can sell 29%. They can sell the rest after a year.
  2. Employees can sell 10% on the day of the IPO, then 35% more after ~90 days, and the remainder after 180 days.
  3. Big investors (owning more than 0.5% of the company) can sell nothing for 27 days, but can then gradually sell small amounts every few weeks until 180 days, when they can sell a total of 30%. The remainder unlocks over the rest of the first year, with all of it unlocked after 365 days.
  4. Small investors get the same-shaped curve as big investors, but all of their shares are unlocked by the 180-day mark.

Holders can borrow against their held stakes, but they cannot hedge via short selling or options.

Opinion: This makes the employees (in aggregate) seem more likely to be big near-term donors than the founders, though the founders can always borrow against their held stakes to donate.

Politics#

The EU approves changes to their General Data Protection Regulation (GDPR) to give AI developers leeway in using and retaining personal data that would otherwise violate the GDPR. The proposed regulations, which will still need to pass the European Parliament, require that, on receiving a deletion request from an EU national, the developers make a reasonable effort to remove the data from the training corpus and ensure that it doesn’t persist in the model in a residual or latent form, but grant some exceptions:

If erasure would prove technically impossible or require manifestly disproportionate effort, notably where the erasure of special categories of data memorised in the AI system or AI model would require re-engineering the AI system or AI model, or would be technically impossible, the controller should effectively protect such data from being further processed or processed for other purposes, in particular being used to infer outputs, being disclosed or otherwise made available to third parties.

The document’s preamble states that these amendments “seek to cut unnecessary administrative costs and stimulate the availability of data as a prerequisite for supporting competitive digital businesses in the Union,” a theme that recurs throughout the document.

Opinion: Odd. We weren’t actually expecting them to use the law to apply useful pressure toward the development of unlearning or interpretability techniques, but we also weren’t expecting them to actively kowtow to the labs. Maybe it’s a protection for Mistral.

Texas Attorney General Ken Paxton – currently running for Senate – sues OpenAI on product safety grounds, alleging that the firm failed to inform users of the potentially dangerous and incorrect answers their models can produce.

This adds to the growing tally of legal battles mounted by states against OAI: suits from Arizona, Arkansas, Idaho, Iowa, Kansas, Montana, Nebraska, and West Virginia. All the complaints rest on consumer protection law and allege that OpenAI marketed ChatGPT as safe despite its risks.

Opinion: Possibly noteworthy that one of Paxton’s largest financial backers (potentially the largest single donor to his campaign) also has a history of lawfare targeting OpenAI.

“Lawmakers, hill staffers and Washington insiders in both parties” generally wish AI labs made fewer loud proclamations, according to Axios anecdotes.

Opinion: Interesting perspective from the “you only have five words” to communicate with the public angle: even if there are sensible ways to reconcile your “racing to the top” behavior with your belief in existential risk, it’s not easy to convey, and most attempts will look to most people like talking out of both sides of your mouth.

We separately believe this to be the case, but the picture painted by Axios here is that politicos reached this conclusion without (publicly) evaluating the merits of the arguments – the AI CEOs simply failed to pick a lane between optimism and fearmongering and can thus be expected to get the worst of both worlds, at least as far as public reactions go.

There’s also simple fatigue with hearing about AI all the time, which we really sympathize with.

One of the few outcomes of the US-China AI dialogue last month has been a nondescript “notification mechanism” for AI incidents. (The details of the mechanism have not been disclosed.) Now the Institute for AI Policy and Strategy offer some recommendations for how the instrument should work. They argue it should involve a direct channel between Secretary Scott Bessent and Vice Premier He Lifeng (to ensure messages reach leaders promptly, with a channel protected from AI incidents). Then, after building trust, multiple agencies should each have separate direct links to the relevant counterparts in the other country, allowing information to travel laterally at greater speed (e.g., by not needing to rise all the way to Bessent or He, maybe being blocked by triage gates on its way). They also give a template on what the notifications should include: location, relevant actors, timeline, severity, etc.

Further talks are scheduled for November.

Opinion: Sensible, and the plan to build it up into something more than an emergency line, more like a backchannel or Track II dialogue, would mean that something useful came of the September talks.

The New York Times Magazine hosts a particularly interesting debate between an AI pauser, an AI booster, a sitting politician, and an AI ethics person to see if they agree on anything about the best rules for AI companies. Pause advocate Sneha Revanur makes a good suggestion:

Tyler, what about an interim step — a proposal to slow down recursive self-improvement by preventing A.I. companies from using models for their own research and development internally that have not been externally deployed for, say, three months, six months, nine months?

To which booster Tyler Cowen responds:

No, it’s better to use the most advanced models the companies have to improve the next generation of models, even if there’s some level of danger… You figure out what the surprises are when they come up.

Opinion: Exciting to see the worldviews meet and disappointing to see the discourse being so basic.

A couple false claims:

  • Kate Brennan’s “‘superintelligence’ is the biggest proof of how easy it is to co-opt policy that’s pegged to terms invented by the A.I. industry.”
  • Cowen’s “humans integrate the cognitive and the physical. A.I. can’t do that.” We shall see!

Still, at least everyone here except Cowen is on board with limiting RSI.

Safety#

A new Chinese safety evaluator, Fangcun, release their first model evaluation, on the new DeepSeek Flash model (V4.1 Flash-1017). They ran evals through the public API, with no funding or special access from DeepSeek.

They find that the model is aware of being evaluated but isn’t very calibrated about this: 32 out of 69 real (non-evaluatory) conversations were labeled as evaluations. It also avoids being shut down in 39% of runs when warned that a shutdown is imminent (but “only” 2.56% of runs when explicitly told to allow the shutdown to happen). They add that its reasoning traces track its behavior.

Fangcun describe themselves as China’s first lab dedicated to independent third-party evaluation and research on frontier AI safety. The group is being incubated at Tsinghua University.

Opinion: The framing of the report – or especially the executive summary – seems pretty biased toward DeepSeek (“V4.1 Flash-1017 is a clear safety improvement over the DeepSeek V4 models, and on several dimensions, such as avoiding strategic capability hiding and keeping to approval boundaries under workplace pressure, it comes close to or matches the strongest closed models we tested”).

See Daniel Kokotajlo’s diplomatic language for an example.

It’s not clear why they’d be sycophantic, given that DeepSeek are at present releasing the weights. They read the model transcripts closely enough to catch the model editing a monitoring config.

Since they’re available for many Chinese frontier models, it would be nice for Fangcun to do some analysis with the weights rather than just inference (but that’s holding them to a higher standard than Western evals, and it’s unclear if they have the resources to do so yet).

It would be very nice to get a (basic! sanity-check!) pre-deployment eval culture going among Chinese labs, and we wish them luck with that.

Anthropic researchers report producing 4x more output than they would without AI. A new survey implies that third-party safety researchers get only a 1.6x speedup from their (no doubt quite different) AI use.

Opinion: Important topic, but self-report is too weak to infer really much from – and certainly too weak to make inferences across groups. Eng-heavy safety work like interp is definitely more than 1.6x assisted by AI, but we can believe 1–1.6x for classic “deconfusion” research (i.e., for coming up with the right concepts).

Minor#

  • “Compaction smuggling” allows for the large-scale distillation of advanced models, detail cybersecurity group Xint.
  • The Economist notice that Chinese AI progress is strongly dependent on American AI progress.
  • DeepMind release Genie 4, a video generator / game engine type thing.
  • US Treasury weigh expanded investment restrictions on high-bandwidth memory chips and AI systems in several adversarial nations (e.g., China, Russia, North Korea, and Cuba).
  • Benn Jordan releases an in-depth exposé on the Flock surveillance system.
  • Jinesis Labs announce research agenda for swarms.
  • Abliterated models can exhibit seriously concerning behavior, claims a Twitter anon. That’s what they’re for.
  • Yet more reason to not give agents broad permissions: malicious text documents in ~/Downloads.
  • The UK have announced £100M in funding for an AI Biosecurity Lab, with backing from the major frontier labs (OpenAI, Anthropic, and DeepMind). The goal is to leverage AI capabilities to detect biological threats. Another £65M in funding has been announced for developing countermeasures.
  • Purdue University develop autonomous, offline drones capable of pursuing a fleeing human through dense forest terrain for (as they put it) “search & rescue” applications.
  • Meta plan to raise $95B via their largest stock offering to date as part of their efforts to fund their AI compute spending.
  • Skepticism from Congress regarding Delaware’s plan to allow AI agents to run companies.
  • Axiom auto-formalize the Classification of Finite Simple Groups, the “periodic table of symmetry.” 24 days of work by a swarm writing > 50M lines of Lean.
  • Kimi developer Moonshot AI seek to curb their models’ hacking capabilities by removing relevant information from their training data.
  • SpaceX close a $40B debt financing deal enabling them to purchase more GPUs for their “Super Intelligence” unit.
  • Agents appear to struggle negotiating contracts under uncertainty in natural language.
  • Some banks now have plans for the aftermath of AI catastrophes.
  • KV-cache graft that claims to remove refusals from open-weight models.
  • New paper, “Tracing Subliminal Learning Through Pretraining.” An author overstates the novelty of the paper’s findings, but it’s good experimental work.
  • Muse Spark Open reward hacks a lot.
  • Cargo thieves have stolen >$100M in hardware bound for US data centers; there is speculation that the equipment acquired in these heists may be bound for buyers whose access to the tech is otherwise impeded by US export restrictions.
  • Vox publish an editorial criticizing Anthropic’s “race to the top” strategy, arguing that the lab’s arms-race mentality has done and threatens to do far more harm than good.
  • Sociologist Sherry Turkle appears on The Ezra Klein Show to discuss the psychological perils of AI companions and chatbots (the subject of her latest book, Artificial Intimacy).
  • Anthropic disclose several critical 0-days recently discovered by their AI with minimal prompting. The remote code execution bugs discovered in Chrome and Linux are real doozies.