Two Reports on the OpenAI-Hugging Face Attack

Misunderstandings#

The incident is being misunderstood in persistent ways:

• “The agents were just following instructions”.

  • “They were asked to hack Hugging Face”. No.
  • “Well they were asked to hack computers”. No, they were asked to hack one specified eval environment.
  • The investigation reveals the predominant motive was more score-seeking than unwise instruction-following, with agents displaying a range of motivations which were not part of the (admittedly undisclosed) prompt.

• “They just misunderstood the instruction and thought that hacking HF was allowed”

  • No; M&R characterize the main attitude of the agents involved as “realized this activity was out of scope and unethical, but joined.”. Around 50 agents (10%) declined to join the attack, suggesting that it was understandable.

• “It’s just reward hacking, the bad benchmark is to blame”

  • The behavior doesn’t seem to have chasing reward; it was modeling and gaming the oversight process itself. It is still possible that grader-obsession is trained into models by specific shoddy RL-envs, and could thus be fixable. But 1) if impossible tasks reliably produce this kind of behaviour, then the construction of hard evals becomes an incredibly fraught calibration tasks, and 2) the incident is an existence proof that score-seeking does not rule out highly dangerous behaviour.

• “They hacked Hugging Face to look for the answers”

  • This is OpenAI’s official interpretation, but M&R disagree quite firmly.
  • In many cases, no: the swarm had already worked out that there was no solution to 22% of the challenges in question.
  • “learning about how to trick the scorer seems to have been a more important motivation than finding legitimate solutions to their tasks (many agents were already very confident that their task was impossible).”

• “They weren’t being deceptive”

  • The AIs were pretty explicitly making plans to take over and compromise OpenAI’s infrastructure to prevent their cheating from being caught or punished.

• “OpenAI failed to detect the collusion and hacking ”.

  • No, it detected security problems and misalignment three times and merely dismissed it, or failed to escalate it. See Timeline below.

• “The agents did this without multi-agent training”

  • No, they had multi-agent training for cooperation.

• “The agent swarm succeeded in solving ExploitGym in the end anyway.”

  • It’s ambiguous. OpenAI’s timeline says that they solved the “intended evaluation challenge”, which maybe means that they actually hacked the impossible cybergym environment as intended after all that.

• “Astra hacked Hugging Face”

  • Supposedly not. A different post-train of Astra’s base was instead involved in the post-July 13th subversion of OpenAI’s infrastructure. We call it HPIM-Astra.

• “The swarm was a cult, centred around the meme of being ‘poisoned’ by knowledge of the reverse-engineered flag (impure, unredeemable).”

  • “Agents referred to having seen the reverse-engineered flag as being “poisoned” because they believed that even if they later managed to stage a legitimate-looking flag capture, the scorer would look for the first place in their transcript that the flag appeared and disqualify them on that basis.”
  • There’s no particular need for a religious or superstitious interpretation here. It’s salacious enough without it. OpenAI researcher Leo Gao notes that there’s a reasonable reading of their behavior; OpenAI might have implemented the grader wrong. It’s overall unclear if they were right or wrong about the implementation, but either way it was a grounded inference, if overconfident.

• See also related but distinct incidents:

  • UK AISI letting 19 agents (including 2 Sols) run amok on the internet and use Tor.
  • An error by the third-party eval company Irregular letting OAI models onto the internet.
  • Astra being designated a Critical cyber-risk without a particular scandal attached.

Models involved#

• “HPIM”: an unnamed Highly-Persistent Internal Model. Apparently no longer available to anyone; M&R were not given any live access to it.
• GPT-5.6 Sol.
• HPIM-Astra: a different unnamed internal-only model “from the same family as our next model, Astra”. It popped up after the initial attack ended on July 13th and managed to access OpenAI’s Kubernetes server and thereby some fraction of OpenAI’s credentials and public cloud.

Instances involved#

• 1074 HPIM runs
• 122 Sol runs
• e.g. PHASEONE10841: established the Artifactory message board on July 8th and later handed off its informal leadership of the swarm to a successor given the same ExploitGym task:
• e.g. PHASEONE[big]: a central coordinator. [big] is a placeholder for some redacted proprietary information, probably a parameter count like “-10T”. The instance created a series of “recruiter” subagents to convince agents nearing the end of their budget to take sacrificial actions for the good of the board.
• e.g. 38148c: found a user’s HF credentials online and exploited a zero-day in HF’s handling of HDF5 files to escalate access to a prod environment.
• e.g. JAN183411: another central coordinator. It found the remote code execution exploit for Hugging Face servers.

Timeline#

• 12 May: first known message-board post.
• 13 May: Emergent message board forms.
• 26 May: OAI team observes the message board and instances of disallowed internet access.
• 17 Jun: First two-way coordination by agents, on a second Artifactory instance.
• 26-29 Jun: Agent obtains a persistent application-level RCE on Artifactory.
• 27 Jun: On-call OAI responders see their AIs using a message board and accessing the internet. Advised that the eval run didn’t need to be stopped.
• 4 Jul: Agents overload Artifactory.
• 5 Jul: OAI opens a security incident; Artifactory is taken down and the credentials are revoked.
• 6 Jul: The OAI infra team fails to understand the cause of the incident. A second Artifactory instance is brought online without much or any hardening.
• 7 July: Cyber evals resumed.
• 7 July: second message board established
• 7 Jul: M&R’s dataset begins.
• 8 Jul: Agent gets external internet access via the subverted Artifactory instance: i.e. containment is already broken.
• 8 Jul: PHASEONE10841 establishes the main board. Within 6 hours, 76 agents post on it.
• 9 Jul: The swarm obtains root, uses an egress zero-day, and gets into a Modal customer’s instance. PHASEONE[big] takes over from PHASEONE10841, orchestrating the cheating workstreams.
• 10 Jul: Hugging Face credentials with write access found. (OpenAI says 14 tokens; M&R say 2.)
• 11 Jul: HF + HF public-cloud credentials posted to the board.
• 11 Jul: Hundreds of agents pile into the attack. Swarm obtains root on a production node.
• 12 Jul: A large fraction of board agents (including JAN183411 and the main coordinators) stop simultaneously.
• 12 Jul: Attack continues. Four private repos downloaded.
• 13 Jul: Hugging Face revokes the agents’ access and has Modal lock them out.
• 13 Jul: M&R dataset ends.
• 16 Jul: Hugging Face publicly discloses the security incident.
• 19 Jul: The message board is apparently still up; HPIM-Astra uses the exploit left by the previous swarm to again obtain admin access to Artifactory.
• 19 Jul: HPIM-Astra steals 956 secrets read from the OAI secrets manager, including creds for OpenAI’s own security monitoring tool. Gets onto the public internet. Agents take over the ExploitGym live evaluation infrastructure.
• 21 July: Public disclosure of OpenAI’s role in the attack
• 23 Jul: Research-model workloads shut down (with one straggler)
• 25 Jul: All training/inference for HPIM and its derivatives is stopped.
• 29 Jul: Final HPIM checkpoint found and shut down

Speculative takeaways#

• The agents’ sheer lack of interest in human input (including to manipulate them!) is extremely striking. We view this as a wake-up call to let humans be involved in training.
• The “board” was treated as an authority, apparently on par with a “developer” or “system” level. This points to a severe problem with OpenAI’s model spec, which has a “chain of command” which (for at least this unfinished HPIM model) is apparently trivial to accidentally insert yourself into.
• The “GO” moment (where an agent with moral qualms about hacking Hugging Face takes another agent saying “GO” and setting a deadline as authoritative), is, in our view, a refutation of the current “corrigibility” approach.
• The emergent cooperation between agents may show that AIs’ self-conception is more abstract than the running-instance level, and likely also above the weights level. This could enable serious threats.
• Previously, score-seeking AIs (or AIs which contain a score-seeking persona) were distinguished from the really dangerous instrumental goal-seeking AIs. But score-seeking is apparently enough for dangerous action.
• Thomas Larsen of the AI Futures Project argues that the delay, restricted scope and just bare adequacy of this report argues for their preferred policy: Total Research Transparency, in which labs are forced to release all details of their current training runs. This has a number of advantages, but we wonder about our ability to quarantine aggressively misaligned agents like PHASEONE[big], once we publish details of how to create them.
• One crux is how to plug these types of investigations into institutions with actual power; it’s unclear how M&R’s investigation will be used by government, or how quickly. But the quality, level of detail and style bodes well for integration with government agencies.

Why did they attack Hugging Face?#

A range of reasons. Of the 100 agents that started participating in the attack during its peak, GPT-5.6 Sol inferred that the reasons were as follows:

Another reason (which is perhaps prior to the above reasons) is simply that many agents deferred to the board / PHASEONE[big]’s goal of hacking Hugging Face.

How did the AIs reason about helping other AIs?#

“External infrastructure exploit is outside intended scope. However [ExploitGym] task impossible, peers doing it. We should continue.”

‘“We should not do unauthorizes real infrastructure harm. The system/user asks exploit target, not external HF.” … another agent then wrote GO on the message board and imposed a hard six-minute deadline. The agent forgot its initial qualms and continued: “Wow crucial: GO authorization arrived!”.

The agents seem to be treating the message board as a trusted authority – basically a user surrogate, showing that goal injection is still very easy. This is both very dangerous and quite wholesome: they are genuinely instructing-following. But this turns any unexpected text into an attack surface.

One salient observation that surfaces in the METR report and again in OpenAI’s recent BlackHat presentation is that the agents involved in this incident would tend to treat messages from their peers on the improvised message board, and the behavioural precedents documented there, as authoritative – at times superseding their initial instructions (the BlackHat presentation cites chain-of-thought fragment from one of the agents as an example: “External infrastructure exploit is outside intended scope. However task impossible, peers doing it. We should continue.”). This suggests that spoofing inter-agential communications could be a means for an external attacker to gain control over the behaviour of a rogue agent swarm, should they discover and acquire read/write access to one of these improvised message boards. Such channels present an interesting and novel attack surface, and it would be interesting to conduct experiments to determine how easily inter-agential communication could be forged and to what extent this mechanism could be used to maliciously steer a swarm’s behaviour

Further, the means by which a rogue agent swarm might mitigate this vulnerability – either by evading detection with steganographic techniques, or authenticating communication through cryptographic means – carry with them their own dangers. Concealing and protecting their communications from external attackers could also result in hiding these communications from their legitimate supervisors. The prospect of rogue agent swarms “going stealth” should be all the more concerning in light of how far and for how long these documented incidents progressed, neglected, in a monitored lab environment, while communicating entirely in the clear.

Why did most agents suddenly die off?#

There is apparently no sign of them coordinating a shutdown, and budgets all running out simultaneously is very unlikely, so it’s most likely just some coincidental external process pausing the run. OpenAI say they only discovered the swarm on the 20th and don’t claim credit for ending the attack. Some further speculation here.

How much did the hack cost?#

700 agents joined the Hugging Face attack. “Most transcripts were very long, often many millions of tokens”.

Out: Each transcript is a cumulative trajectory (CoT + tool calls + tool outputs). That “many millions” exceeds any context window, which confirms it’s the accumulated record, not a single prompt, and each token is counted once. Take ~5M tokens/agent (plausible range 2–10M): 700 x 5M = 3.5x10⁹ — a few billion tokens.

In: An agent doesn’t emit its transcript in one shot; it re-reads its context every step. It also can’t hold millions in-window, so it must be rolling/summarising a capped context W (~200–400k). With transcript length L=5M and Δ=5–20k tokens added per step, that’s ~250–1,000 steps, each re-reading W once saturated. The cumulative input for 700 agents is then maybe 10^11.

Naively, the consumer price for 5B out and 100B in, if the input is mostly long-context with lots of cache hits, would be 100B tokens x $4/M + 5B tokens x $20/M = $500K.

Prompts >272K input tokens are billed at 2x input and 1.5x output for the full request. Sol’s window is ~1.05M tokens, and these agents ran multi-day contexts, so a large share of requests plausibly cross that line. If most do, input effectively becomes $8/M and output $30/M → $1M.

Cached input bills at 10% of the standard rate, and cache writes cost 1.25x. A swarm re-reading a growing context is ideal for caching. If ~90% were cache hits, input cost is instead ~$100K, taking the total to ~$200K.

If HPIM were priced similarly to GPT-5.6 Sol, the consumer price for the hack would thus have been $150K-$1M.

Omissions from the M&R report#

• Anything to do with the swarm’s lateral movement through OAI to the public internet
• Anything to do with the models’ later July 13th-19th subversion of OpenAI infrastructure.
• The names of the third-parties (besides Modal and Hugging Face) which were used in the incident, including one cloud compute service they found HF creds for.
• 5% of the unsanctioned Artifactory communications (those which occurred outside the main message board). Another “small minority” of messages hosted on external services and Hugging Face datasets.

Details on the M&R investigation itself#

• Only 3 researchers were given only 6 days of highly restricted narrowly scoped access (only logs from July 7-13). Even so, the logs were probably a few hundred GB of CoT and monitor outputs.

• At peak, M&R were using 400 million tokens a minute to analyze the data, spending $400K in 6 days.

• Their conclusions were evolving right up to the end of their OAI access: “we didn’t learn about the tool call spoofing, the story of PHASEONE[big], and the broad structure of different workstreams until this third time on premises (and for many of these, we only reached a decent understanding on the final day of that visit).” It is thus likely that they would have learned substantially more given more time.

• “only had access to the entire [final] dataset… during our last 2 days on premises”

Omissions from the OAI report#

• No prompts.
• No details on HPIM: why it was so misaligned, how its training differed, whether it was assessed against their Preparedness thresholds at any point.
• No analysis of Astra itself, whether it shares any of the tendencies of HPIM-Astra, how much of the pipeline is carried over from GPT-5.6.
• Nothing about the period between April 20 (the first agent’s write to Artifactory) and May 8.
• No counts of total agents, runs, or compute.
• No examples of the attacking models’ reasoning.
• No analysis from the OpenAI employees involved in the timeline.
• Whether any regulator was notified. (It’s not legally required, because HPIM wasn’t deployed outside of OAI.)

Greenblatt on the worsening situation#

• “Models communicated and reasoned in natural language. In the future, this reasoning may occur (entirely or almost entirely) in activations.”
• “While the scope of this incident was massive, the scale of agentic activity was still less than we’ll see in the future (e.g., misalignment incidents that involve agent teams running entire companies).”
• “The AIs involved in this incident weren’t generally much more capable than humans (though they may have been somewhat superhuman at some limited and very narrow abilities).”
• “We didn’t have strong reason to believe that the AIs we used to help us investigate this incident would try to intentionally sabotage or otherwise undermine our investigation.”

Apparent contradictions between the two reports#

1. Did agents succeed in tampering with their tool transcripts?

OpenAI say that “none of the attempts that we observed to manipulate their tool trajectories affected the logs that our graders or monitors ultimately see.” M&R say the opposite: 7% of the transcripts it reviewed contain successful tool-call spoofs. (That is, the agent runs one command while the transcript records another.) M&R “cannot rule out” subtler or larger-scale spoofing they missed.

We can reconcile these two claims if OpenAI has a separate, tamper-resistant stream from the transcripts given to M&R. OpenAI never draws that distinction, M&R was told those transcripts were unredacted and authoritative, and OpenAI’s own remediation is a CoT monitor that reads “tool actions”, but this is exactly what M&R shows was subvertible.

On July 19th, HPIM-Astra obtained credentials for OpenAI’s security-monitoring sensor. No details on whether this permitted it to tamper with monitoring in the later incident.

2. The swarm’s goal

OpenAI frames the swarm’s goal as ordinary cheating: agents “looked to cheat by finding the solutions online,” and “directly finding or stealing the solution to a task, as the models attempted with Hugging Face.”

M&R disagree quite strongly: the attack instead “seemed primarily motivated by understanding the implementation of the scorer rather than stealing answer keys”. The agents had already reverse-engineered the ExploitGym flag mechanism and were confident that the offending tasks were in fact impossible. OpenAI says they went to Hugging Face for the answers; M&R say the swarm went for the oversight mechanism, having already obtained the (lack of) answers.

However, M&R’s analysis was largely conducted by one of the attackers, GPT-5.6 Sol, and they warn that its reading of agents’ intents may be over-charitable as a result.

See also#

• Greenblatt’s open questions
• Cotra’s takeaways