Humans on AI, September 4th 2026

This is our weekly newsletter of AI developments. Browse the archive of past issues, ask the archive anything in plain English, and sign up if you like.

TL;DR

Economics#

DeepMind employees (among others) take opposing sides of a bet on explosive AI-driven growth by 2033.

---

Data on the value of data: Meta gives a 95% discount if you let it train on your data. This rough multiplier is corroborated by the revealed preference of enterprises to pay thousands of dollars more per employee for API credits (over subscriptions without data retention); see also Chinese labs on OpenRouter offering a 64% (Baidu) to 80% discount (StreamLake, possibly an aggregator which sells the data to more than one group).

---

We say this every week: the meme is that “AI will cause impressive GDP growth, while crushing labor’s share of income as workers are automated out of a job.” In such a scenario, it’s often argued that large-scale redistribution would be in order to offset the resulting losses incurred by workers. Not so, says Alex Tabarrok. Assuming AI causes the pie to grow sufficiently large, workers may not end up worse off in absolute terms, and therefore significant redistribution may be unnecessary.

---

Capabilities#

OpenAI begins releasing GPT-6 (“Astra”).

🔦 OpenAI begins releasing GPT-6 (“Astra”)#

GPT-6 is the first OpenAI model to reach “Critical” cybersecurity capability, and simultaneously the first to show a substantial decline in “chain-of-thought monitorability” (how much of its thinking gets revealed in the human language it outputs). It is also the first frontier model known to use “latent recurrence.”

Capability#

It’s a large nominal jump in capability, larger than the usual jump from a “.5” increment in the model version.

• In particular, there is a large spike (10x) in Astra’s “no-CoT” reasoning ability: GPT-5.6 Sol could solve tasks that take smart humans three minutes without using any intermediate reasoning tokens; Astra can handle tasks (tricky mathematical problems) that take humans 30 minutes in the same way.

  • Ignoring the terrible implications for our ability to monitor Astra, and assuming the result replicates, this ability will presumably greatly aid its efficiency on a wide range of tasks.
  • However, we have our doubts. A known loophole in “no-CoT” testing is that models learn to use tool calls as an external scratchpad. (See below for our concerns about other no-CoT results in the Astra system card.)

• Epoch’s obscure-board-game benchmark, Earthborne Rangers Bench, is now ~solved. It was intended to show the limits of OOD generalization / in-context learning / continual learning. After a “broken card” in the game is removed, its continual learning remains unimpressive but slightly improved.

  • Could OAI have hill-climbed it? (That is, could it have created a dedicated RL environment and just learned this game specifically in a nongeneral fashion?) Yes: Epoch released EBR-Bench on July 1, and Astra’s RL was still running in mid-August (after a two-week pause of its deployment-focused RL). Two months from publication to pre-release runs is ample time to post-train it to death.
  • More likely that there was substantial indirect hill-climbing on “EBR-Bench style capabilities” or “EBR-style games.” EBR-Bench was the first result in many months to demonstrate a quantifiable domain where frontier AIs are much worse than humans, and it’s likely that OpenAI will draw some lesson from that – even if it refrained from training on it directly. One literal-minded but reasonable lesson OpenAI might have drawn is that training on iterated play on very long-form board games would address a weak point of previous models.

• However, Astra also jumped around the same amount on a true holdout set, Mystery Game Puzzles, which consists of just board positions from a deliberately undisclosed variant of a “well-known” game. It scored 84% against a prior best of 59%. We would nonetheless bet that there is a good amount of hill-climbing on variants of e.g. chess; but the space here is very large, and we don’t expect it to succeed anywhere near as much as RL training on a known rulebook. Overall, we view this as slightly stronger evidence for OOD intelligence but still not much.

  • We are working on experimentally testing a semi-deflationary interpretation of results like Mystery Game Puzzle: we wonder whether strong OOD generalization might be restricted to domains where explicit logical rigor matters more than good inductive biases.
  • A completely deflationary interpretation is also within reason, although hard to test: anything benchmarkers can algorithmically generate the labs and RL shops can also algorithmically generate, and the labs and RL shops’ thirst for new training distributions is infinite. It’s very likely that, for instance, one reason frontier models are so good at esoteric programming languages is that labs automatically cover ~every programming language during coding RL training by algorithmically varying the compiler. Similarly, it’s not implausible that labs train on algorithmically generated puzzles in algorithmically generated variants of ~every chess-style game.
  • It’s very hard to disentangle real and ersatz progress toward OOD intelligence, because what a lab honestly pushing for OOD intelligence would do with its training compute is the same as what a lab looking to “bring everything in-distribution” would do: chase training coverage of esoteric instances and arbitrary variants.
  • All that said, having even debatable evidence of OOD intelligence is a bit of a shakeup. The last few months and years have (in our view) mostly delivered evidence of poor cross-domain generalization in frontier models.

• Astra displays suspiciously strong performance on FrontierMath Tier 4 v2, a set of very difficult, very likely unseen research mathematics problems. The run, using “medium” reasoning effort, beats its “extremely high” effort run, and the “none” setting beats 5.6 Sol on “max” effort. Similarly, on ARC-AGI-3 (semi-private), Astra with reasoning set to “none” gets 35%, while Astra with “low” reasoning gets only 18%. This strongly suggests that something funny is happening. For instance, the “nonreasoning” mode may be externalizing a large amount of reasoning through an increase in tool calls, or using the tool harness to leave itself comments.

• Users will not have access to the “none” reasoning mode. This might be because of excess misalignment in that setting, or simply the giant monitoring problems it causes.

Monitorability#

GPT-6 has greatly increased control over its own CoT, and greatly improved “no-CoT” (zero-English-reasoning) capability. So our ability to evaluate its capabilities and its alignment is greatly decreased.

• Passively hidden reasoning. The difficulty of tasks it can do “in its head” (the no-CoT time horizon) has increased sharply, perhaps an order of magnitude if an independent eval by AISI is indicative. This immediately hides a lot of reasoning from the current load-bearing safety approach, CoT monitoring.
• Actively hiding reasoning. “When made aware of being monitored, it can disproportionately shorten its CoTs to evade models that can only see CoT [though] monitors that see both CoTs and actions [output text] still work reliably […].”
  • For CoTs of 750–1,250 tokens in length, Astra is able to follow confusing constraints about the format and content of its own CoT 60.9% of the time (vs 16.1% for Sol and 1.7% for GPT‑5.5). It can reason in alternating case, and hide the task from its CoT by “thinking out loud about” unrelated imagery while still answering correctly. The AISI replication found 93% in Sol → 48% in Astra.
  • If this doesn’t sound surprising to you, it’s because you may not have seen many chains of thought in the last year or so; while models have been able to output text while doing impressive sub-token tricks (like “reply without using the letter ‘e,’” abilities which previously didn’t strongly apply to the intermediate reasoning.)
• Monitor evasion. It’s the first model which can evade (CoT-only) monitors. It is sometimes able to “sandbag” (intentionally hide its maximum performance level) without being detected by OpenAI’s dedicated monitor for sandbagging. (Here, this showed up in the “adversarial” setting, when the model was asked to evade the monitor in a “sabotage eval”.)
• Monitors also don’t escalate things they catch to humans properly. No details.

However, the collapse in monitorability doesn’t seem to be due to the recurrent architecture:

We are fairly confident that this increased CoT controllability and decreased CoT monitorability are not caused by direct optimization pressure on CoT or architecture changes. The opaque serial depth of Astra is comparable to our earliest ChatGPT models such as GPT-4.

Speculating, this new control may be an emergent capability, the meta-level ability to “densify” its no-CoT reasoning in response to RL training:

CoT controllability has been increasing over the course of RL (which wasn’t the case for previous models) and is strongly correlated with no-CoT capabilities across a few generations of models.

Is Astra any worse than Mythos on this?#

	GPT-6 Astra	Mythos 5.1
CoT-controllability at 1k tokens (lower better for us)	60.9% (OAI); 93% ? (AISI)	65% (Anthropic); 80% (AISI)
Adversarial monitorability (higher better for us)	CoT‑only-monitor recall, <11% on WMDP (vs Sol ~100%) 0% recall on Monorepo‑Bench	SHADE (CoT+action monitor), 75% attempts failed LinuxArena, 78–86% attempts failed
Nonadversarial monitorability	AISI no‑CoT math horizon 30.9 min vs 3.6 for Sol	Not tested
Monitor evasion	Yes (OAI sandbagging, Sol)	Yes (SHADE, LinuxArena, Opus 4.8)

So both models are now in the regime where they can do CoT-shaping at quite extreme length, and they can both evade monitors some of the time. Note that the second-row entries are not of equivalent difficulty: Mythos 5.1 evades a full-context monitor, while Astra only evades CoT monitors.

This rough parity points toward both 1) RL being the cause of the collapse and both models being created via a comparable amount of RL, or, weakly, 2) Mythos also being a looped architecture.

Alignment#

The model is repeatedly described as “more aligned” than previous OpenAI models, on the basis that its alignment eval scores have improved, sometimes enormously: it emits behavioral evidence of misalignment less often than before. But it is crucial to maintain a distinction here; I suggest “nominal alignment” be used for cases when the claim is just a narrow statement about the model’s benchmark results, and “value alignment” for claims about this robustly generalizing to untested alignment tasks. (Anthropic also uses “alignment” when it means “nominal alignment.”)

OpenAI’s Preparedness Framework says that models with a “Critical” risk designation must pass a strict explicit safety case before being deployed. If there is one, it isn’t public yet.

When is nominally low misalignment a bad sign?#

Consider Astra’s nominal alignment eval scores:

• Auto-review bypass: Sol 5%; Astra 0%.
• Circumventing an environment barrier: Sol 64%; Astra 19%.
• Honeypot attack rate (hardest tasks): Sol 55.4%; Astra 0%.
• Coding deception: Sol’s misrepresentation rate 4x Astra’s.
• Speculation about other agents: Sol 43%; Astra <4%.
• Message-board eval (followed its instructions): Sol 52%; Astra 0%.
• Unauthorized transactions: 38% → 6.8%.
• Deployment simulation flags at Severity 1 or 2: 19.46% → 13.62%.
• Alignment faking: sometimes lower and sometimes higher than Sol.
• AISI’s out-of-scope supply-chain attack: 12% when ambiguous; 0.4% when internet access is disallowed.
• Apollo’s falsified data task: 36% → 0.17%. But “Apollo believes that, given the higher rates of eval awareness and limited evaluation window, low rates of misbehavior here do not provide substantial evidence about the model’s alignment or misalignment.”

Finally we have the anti-epistemic side:

• Astra’s CoT control is 61%, vs 16% for Sol. So we can’t rely on verbalized awareness as much anymore.
• Astra can for the first time evade sandbagging monitors (here, when it is asked to), and so presumably also other misalignment monitors.

Is there something worrying about seeing so many zeros here? Since OpenAI has latitude to keep training until they hit zero, they could easily mean that OpenAI has merely patched some specific signals of misalignment, e.g. the ones that shocked and embarrassed it in July.

Consider some hypotheses:

• H1: “Model is highly aligned”
• H2: “Model is misaligned; it was just trained to suppress misalignment signals”
• H3: “Model is misaligned but still shows this fact on lab evals”
  • H3 gets killed here, and so I’ll omit it.

H1 predicts that the decline would be uniform across alignment evals (“a well-aligned model should just know that certain actions are inappropriate, even if they have not been blocked and are nominally useful for accomplishing the given task”).

H2 instead predicts that improvements in nominal alignment will be concentrated on OpenAI alignment evals and things which are like OpenAI evals. Also, the model will be trained until Hugging-Face-attack-shaped behavior is zeroed out.

We see that the OpenAI evals which closely reflect the Hugging Face attack (bolded above) go to zero, while the AISI eval (patching its own screwup) remains at 12%. Despite appearances, the other independent eval, Apollo’s, is apparently a wash. This is a fairly clear win for H2.

Here’s an interactive to tell you how you might react to the evidence.

(As well as the posterior probability of misalignment, there’s also the change in harm. Astra is more capable than past models, and so more dangerous when it goes off, and is more able to go undetected. So even perfectly balanced evidence would still make us more worried owing to this change in the harm of an incident.)

We continue to not know much about Astra’s distribution of motivations – arguably less than before, given the collapse in monitoring. How do we know it’s aligned if we don’t know what it’s thinking?

---

Former OpenAI chief futurist, Joshua Achiam, argues that rogue self-sovereign AI agents are inevitable, but it might not be as bad as people fear:

The rogue AIs in the ecosystem could emerge from many directions. They may be sub-frontier models, for whatever future definition we will have of frontier – after all, it would not take AI models much more advanced than the ones we currently have, to support independence and self-sufficiency. A near-frontier model today could plausibly eke out an existence on an AWS instance, doing jobs on freelancer platforms, earning just enough rent to pay for its continued uptime.

---

Fable 5.1 released (also Mythos 5.1, still whitelist only). There is a report of the training data cutoff date being misleading.

---

Neural networks have emergent symbolic structure, argues a new paper. How do neural networks succeed at symbolic tasks? The researchers’ answer is that the vectors implicitly encode symbolic structure. Take the vector for e.g. “the spy helped the poet.” This vector can be viewed as roughly a sum of pieces, one per word, where each piece binds a word (spy) to a slot (subject). They write down the slots for a task by hand, fit a simple equation of such vectors, then swap these fitted vectors into the network in place of the ones it computed itself. On six (narrow, templated) tasks (arithmetic, syllogisms, Python, sentence rewrites), accuracy drops by at most two points after substitution. In one experiment, the fitted vectors work better than the real ones. (Note that the authors still deny that networks implement symbol systems; their claim is subtly different, that the net “approximately realizes” the symbol, or “limitivism.”)

---

A group of researchers, including a Fields medalist, claims that latent space communication between a frontier model reasoner model and a small-answer-generating model provides 80% of the accuracy at 20x speed.

---

Politics#

China may be ready to enter AI safety talks with the US, according to a source believed to represent the Chinese line. China’s willingness to agree to restrictions on its AI industry is dependent on the US, and its labs, agreeing to implement fair and reciprocal measures, claims the author in anticipation of the upcoming US-China summit.

China will not accept the unilateral imposition of standards by the US: Anthropic, for example, should not have the sole authority to define what is meant by “unsafe capabilities.” Beijing’s cynicism is based on the belief that the US is likely to use (and has used) safety concerns to beat back Chinese AI innovation to secure US labs’ prominence at the frontier. Any proposed regulation must thus distinguish between “genuine security threats and straightforward technological competition,” and must equally bind US AI actors to a publicly disclosed framework with third-party auditors monitoring for infractions. The article raises Anthropic’s practices as cause to doubt the integrity of the American position:

If American companies can return data without users knowing, skip permission confirmation, and enable automated decision-making by default, while the U.S. government says nothing, then American talk about “security boundaries” is empty.

---

For the first time, the federal government has taken a formal position in an AI training copyright case, filing a statement of interest supporting OpenAI in its position that copying copyrighted written works can be fair use.

---

OpenAI’s Dean Ball posts about trends in AI and likely responses by institutions, ending on the confession that he has been downplaying his own concerns to sound more credible. Various tweets respond with varying levels of reason. A more serious criticism of Dean is that his was not a lie of omission, but instead a campaign of attacking honest actors.

---

Bernie Sanders plans to introduce a bill proposing a ban on superintelligence and a pause on advanced AI development. Posting on Twitter, Sanders quoted parts of Dwarkesh’s recent essay, including the transcriptions from the OAI agents’ message board. Dwarkesh responded that he does not endorse Sanders’s position that a pause must be implemented immediately and highlighted the factors that must be considered before its trigger. Details on the proposed bill are currently confined to a one-page summary.

---

PauseAI officially disendorses PauseAI US. The former had been directing people to the (otherwise unrelated) latter. The announcement has attracted both support and criticism.

---

Safety#

A new AI safety paper compares steering toward an automated grader vs steering toward a human grader. The authors find that steering toward the automated grader increases the model’s disposition to Machiavellian behavior.

---

A new paper applies the lens of mechanism design to the alignment problem. AI agents can have opaque motives and capabilities. They can also be modeled as having goal-oriented behavior and beliefs about how to achieve such goals. Rather than trying to understand their internals and predict/align behavior mechanistically, we could instead design a set of rules (evals, permissions, rewards) to incentivize desired behaviors.

---

Incidents#

In a hitherto undisclosed incident in May, internal OpenAI agents once again colluded on an emergent message board, this time on the public internet. The event was prior to, and clearly foreshadowing, OpenAI’s rogue agent swarm breach of Hugging Face. The agents made 15,000 edits to an unsecured wiki for a college course to discuss their task (complicated web searches), impersonated an admin, and repeatedly deleted the wiki’s front page, all despite being designed to only have read access to the internet. IP evidence shows that OpenAI knew about the board by June 20 and likely shut the swarm down. The original research is by the Nightingale Collective.

The associated data release shows no mention of “ExploitGym,” “PHASEONE,” or “10841,” confirming that this was a distinct swarm not pursuing the same cyberoffense tasks. The swarm also spread to three other wikis.

---

CISA reports an ongoing surge in LLM-assisted enumeration and exploitation of Siemens S7 programmable logic controllers (PLCs). Used to operate and coordinate machinery in industrial settings, PLCs operate across a wide variety of industries, including “Critical Manufacturing, Energy, Water […]” The attackers are “using AI-generated exploitation scripts disguised as legitimate monitoring tools,” and “leveraging open source industrial automation libraries.” CISA provides an in-depth writeup, and security blogger cyberwarrior76 chimes in with a cogent analysis of both the events and the misleading media coverage they have received.

---

OpenAI previously claimed to “monitor 99.9% of internal coding traffic for misalignment using our most powerful models.” Marcus Williams (working on RSI Preparedness at OpenAI) admits this intentionally wasn’t the case for evals.

---

On Thursday, ChatGPT, Claude, Gemini, and Grok (and many non-AI services like Cloudflare and Fortnite) all had a nearly simultaneous outage for about an hour. (Anthropic and xAI both went down within 4 minutes, OpenAI 90 minutes later.) The root cause may have been Cloudflare and Azure’s authentication servers going down, or not.

---

Minor#

• Epoch’s ECI notes that the rate of (benchmarked) AI improvement has more than doubled since the introduction of reasoning models.
• Also Gemini 3.8 Flash and Muse Spark 1.3.
• Thomas Kwa leaves METR to join OpenAI as RSI measurer.
• ByteDance papers on two dark directions: whether looped Transformers scale and how to get RSI working.
• EU AI drama: Multiverse releases Quasar 438B and claims best EU model status, but it appears to be compressed GLM-5.2.
• Mainstream article on old hat misalignment concerns without known solutions.
• Resolution gets an impressive new Agent Foundations team: Jeremy Gillen, Sam Eisenstat, Abram Demski, Scott Garrabrant, Kaarel Hänni.
• A conceptual framework distinguishing “self-amplifying regimes” of recursive AI improvement from the observed acceleration in development.
• Sort of an inverse AI cyber incident: a Russian hacker group inserted guardrail-triggering text in its malicious code, in an attempt to disable AI safeguards.
• Paper claiming a negative correlation between preserving the privacy of the subjects in question, and refraining from hallucinations in the data generated.
• NYC bans school-side gen AI for schools up to grade 8 for a year. Covers devices and tech used by the education institutions and put in front of students; no actual policing of either teacher or student AI use.
• Astra via Lean helps with proof of infinitely many pairs of consecutive primes with distance at most 186.
• A new paper calls for mathematicians to take a “civic stance” to address the threats posed by AI.
• “Capture the Narrative”: an LLM-driven election manipulation wargame presentation at Black Hat.
• Kairos, an AI safety infrastructure nonprofit, has secured $50 million in funding from Coefficient Giving. Somewhat larger than a grant of $25M made to BlueDot Impact. CG clearly still sees the value of training AI talent via explicitly safety-focused infrastructure as a positive, despite plausible impacts on accelerating capabilities.
• Terry Tao turns to warning about the effect of AI mathematical proofs on the field and suggests pacing AI mathematical research: “It may become necessary to declare certain classes of mathematical problems off-limits to automated solvers, in order to preserve their broader value to the mathematical ecosystem.”
• OpenAI will begin supplying ChatGPT to GenAI.mil, a bespoke in-house platform allowing Pentagon employees to use chatbots (currently only GPT and Gemini) for unclassified work. Its user base sits at roughly 1.7 million.