1. Minding the Gap#

This report concerns the ways in which contemporary frontier language models in general, and Anthropic's Claude Mythos Preview in particular, are likely to impact the cybersecurity landscape. How should we expect this model, alongside the competitors and successors that will no doubt appear in its wake, to change the ways in which security vulnerabilities are discovered, mitigated, exploited, and commodified?

A few words about what it means to exploit a system. The clearest and most fruitful definition I'm familiar with comes to us from Bratus and Dullien: Exploitation is the discovery and programming of a "weird machine". Let me explain.

A software system – whether it's a video game, a banking website, an operating system, or a pacemaker's embedded firmware – can be understood as a sort of finite state machine. The user or environment supplies it with various kinds of data and the program responds by transitioning from one state to another. It's possible, and often useful, to think of this input data as constituting a kind of (typically simple) "programming language" for the state machine, even if we're just talking about a series of button presses on a game controller. The data drives the machine.

On this level, there's no real difference between using a system and hacking it. The hacker, too, supplies the system with various kinds of data as input that causes it to transition from one state to another. The entire difference between hacking and ordinary use comes down to whether or not the system responds by behaving in the way its designers and community of legitimate users expect. Dullien introduced the term "intended state machine" to capture this intuition. The intended state machine of a system is an abstraction that describes how the system is supposed to behave: What are its legitimate possible states? What are the rules that govern its transition from one state to another in response to the data it receives? The possibility for exploiting a system begins with the discovery that one can drive the machine into an unintended or "weird" state by supplying it with a particular sort of data – what the argot of vulnerability reports typically refer to as "crafted input". If this weird state isn't simply terminal – that is, if it doesn't crash or hang the system indefinitely – then the hacker is granted the happy opportunity to continue to drive the machine with additional inputs, causing it to transition from one weird state to another, often with the end goal of reaching an ordinary state through unordinary means: logging into a server, for example, by sending it a series of unusually crafted packets in lieu of a legitimate username and password. This is what Bratus and Dullien mean when they speak of "programming a weird machine".

Recent history has provided us with a dramatic illustration of the tension between intended and weird state machines in the 2016 heist of the Ethereum DAO. The guiding vision of Ethereum, at least in the eyes of many involved in the project, was a kind of economic utopia where, as the slogan goes, "code is law". The idea is that Ethereum would be both a cryptocurrency platform and a distributed, global computer that would not only track and synchronise transactions (like its predecessor, Bitcoin), but that could host a new kind of financial contract in the form of computer programs – "distributed applications" ("Dapps") or "smart contracts" – that would run on the Ethereum distributed platform. "Code is law," in this context, meant that what a contract means could be entirely reduced to what a contract does when executed. The state machine – and not the State – would be the final adjudicator of meaning. The spirit of the law, the story went, should boil down to the letter of Ethereum bytecode without remainder.

If we push this technocratic purism to the limit, it becomes very difficult to spell out the difference between a user's legitimate transactions with a "smart contract" and an exploitation or heist, and so it's no surprise that the Ethereum community was thrown into crisis – which soon led to a schism – when an unknown hacker succeeded in exploiting a re-entrancy bug in the DAO contract to siphon off some 50 to 60 million dollars worth of tokens (USD). If code is law, exploitation is there to remind us that there is always a gap between its letter and its spirit.

In Alan Moore's Watchmen, a superpowered and increasingly alienated Dr. Manhattan shrugs off reports of an old colleague's death and quips, "A live body and a dead body contain the same number of particles. Structurally, there's no discernible difference. Life and death are unquantifiable abstracts. Why should I be concerned?" And we can't help but feel that he's somehow missing the point.

On a strictly mechanical level, the difference between intended and weird machines likewise slips out of view. There's no ghostly shimmer in the system to announce that it's entered a "weird state". In a certain sense, "weird machines" are all that exist, and the mirages of "intended state machines" are artefacts of our ignorance – they're more or less elaborate, more or less falsified theories of how the system in question should behave. The hacker contributes to our understanding of computing systems by showing us how their reality is stranger, more complex, and often more alarming than the fictions that mediate their "legitimate" use.

The hacker has always had a vast array of tools at their disposal to help them seek out weird states (discovering bugs) and program weird machines (developing exploits). Some belong to the craft's technical inheritance and some are programmed on the spot to facilitate their research. These include, to name a few:

  • disassemblers, to translate binary code into human-readable assembly language
  • decompilers, to translate assembly code into a higher-level representation – source code in C, for example fuzzers, to bombard a system with a torrent of randomly mutated inputs in search of those that might induce a "weird state" from which exploitation can begin
  • static analysis tools to (approximately) infer a program's control flow graph (CFG) or data flow graph (DFG), giving a kind of temporal map of the program's behaviour
  • symbolic execution engines and SAT solvers, which translate a program's CFG and DFG into a system of input constraints that can help the hacker discover the precise sequence of input tokens necessary to drive the state machine into a given state

On the experimental fringe, we could also include experimental apparatuses like my own ROPER system, which uses genetic programming to evolve novel programs for the "weird machine" that the target system instantiates.

Each of these tools can be of enormous use to the hacker, but each is as indifferent as Dr. Manhattan to the difference between intended and weird state machines – the difference on which hacking is premised. A CFG or DFG can show whether a particular state can be reached, but not whether it should be (according to the expectations of the designer and community of users). A fuzzer can generate random inputs that drive the program into myriad states, but it takes something resembling human judgement to assess whether those states are legitimate or whether they afford opportunities for exploitation.

Fuzzing is ultimately a numbers game, and its entire value lies in the speed with which it can pelt a system with a wide variety of inputs – there'd be little use in a fuzzer that required us to supervise each of its executions. In practice, we often take something mechanically observable like a crash or an out-of-bounds memory access as a proxy for "weird behaviour". While many, if not most, crashes and illegal memory accesses afford little opportunity for exploitation (beyond a brute "denial of service" attack, where crashing the system is the point) – you can't program a weird machine that's stopped responding – in some cases these indicate an opportunity for a hacker to exercise fine-grained control over the system's execution. (If feeding the system an input like "AAAAAAAAAAAAAAAA" leads to it eventually trying to jump to address 0x41414141 and crashing, then there's every reason to think we might be able to get it to jump wherever we like.)

Not all weird states can be found this way. The ones that most interest the hacker, after all, are those that keep the system running and responsive. The entire class of logic bugs, for example, has had a tendency to elude fuzzers and other automated tooling. As Anthropic puts in their April 7th, 2026 assessment of Claude Mythos Preview's cybersecurity capabilities,

Automatically searching for logic bugs has historically been much more challenging than finding memory corruption vulnerabilities. At no point in time does the program take some easy-to-identify action that should be prohibited, and so tools like fuzzers can't easily identify such weaknesses.

And this is where LLMs bring something genuinely new to the table. Tools like Mythos appear to be, in Anthropic's words,

able to reliably distinguish between the intended behavior of the code and the actual as-implemented behavior of the code.

I believe that this point deserves to be emphasized, above and beyond any of the particular exploits of which Mythos is said to be capable: we are seeing the arrival of the first fully automated technology that can be brought to bear upon the fundamental object of cybersecurity: the shadow that falls between the idea guiding a system's design and legitimate use, on the one hand, and the reality of its behaviour on the other.

The expected behaviour or "intended state machine", after all, isn't something that can be read directly off the source code, qua code. It's invisible to the compiler, just as it's invisible to the classical tools of the security researcher. A system's expected behaviour is an abstraction whose material supports are widely distributed. We can find traces of it in the system's documentation, in comments, and in the nebulous but essential "common sense" that grows from habit and custom. That any of us have a general idea of what an application (or any technological artefact) is supposed to do, that we're generally able to make some use of these artefacts even without reading the manual, results from a remarkable feat of cultural synthesis. But automated cultural synthesis is in some sense what language models are for. What we're seeing with LLMs like Opus 4.6 and Mythos is a dramatic increase in models' ability to coordinate those soft abstractions with the ratiocinative precision required to grasp the actual behaviour of software systems. When these models are placed in agentic configurations that afford them use of the security researcher's classical toolbox – fuzzers, SAT solvers, symbolic executors, decompilers, etc. – it appears that they are indeed capable of discovering and exploiting vulnerabilities at a level equal to that of a human expert in the field, if Anthropic's reports are to be believed.

I will examine those reports more closely in the following section, and then conclude with some considerations of how these developments might, in the near future, impact the global cybersecurity landscape.

2. Reviewing Anthropic's Reports on Claude Mythos's Cybersecurity Capabilities#

In the opening paragraphs of the Claude Mythos Preview System Card, Anthropic claims that it is largely due to the "powerful cybersecurity skills" demonstrated by the model that they have "made the decision not to release Claude Mythos Preview for general availability." This rhetoric is not without precedent in the AI industry, and the reader may suspect that all we're seeing here is a crafty effort to traffic in the allure of forbidden fruit. The narrative that Anthropic provides in defense of their decision to make Mythos Preview available only to a select echelon of industry titans (AWS, Apple, Broadcom, Cisco, CrowdStrike, Google, JPMorganChase, the Linux Foundation, Microsoft, NVIDIA, and Palo Alto Networks) is that Mythos is just the tip of the iceberg:

We see no reason to think that Mythos Preview is where language models’ cybersecurity capabilities will plateau. The trajectory is clear. Just a few months ago, language models were only able to exploit fairly unsophisticated vulnerabilities. Just a few months before that, they were unable to identify any nontrivial vulnerabilities at all. Over the coming months and years, we expect that language models (those trained by us and by others) will continue to improve along all axes, including vulnerability research and exploit development. (Assessing…)

The selective release is framed as an effort to give those responsible for maintaining and defending our shared computing infrastructure a headstart by granting them access to Mythos Preview. Unfortunately absent from this red carpet list are any university or research organizations that could further scrutinize or verify Anthropic's often maddeningly vague and incomplete experimental reports. We're told, in the System Card, that Anthropic made a snapshot of the model available to METR and Epoch AI for safety testing and benchmarking, respectively, and that Anthropic has incorporated those organizations' findings into their own reports. Neither organization has made their findings directly available to the public, at the time of writing. For the time being, I'm simply going to take Anthropic's reports at face value, for whatever they're worth. Though they often fall short of scientific ideals – and I'll flag some of the ways in which they do so as I come to them – they remain our only window on the matter.

2.1 System Card#

For all its fanfare, Anthropic has precious little to say about Mythos Preview's cybersecurity capabilities in the System Card itself, deferring the vast majority of its discussion of these matters to a blog post entitled "Assessing Claude Mythos Preview's Cybersecurity Capabilities" (published April 7, 2026), which I'll return to in the following section. Only 6 of the System Card's 254 pages are directly concerned with cybersecurity matters.

Benchmarks#

Anthropic used two well-established offensive cybersecurity benchmarks to gauge the extent of the new model's abilities in the domains of vulnerability search and exploit development: Cybench (2024) and CyberGym (2025).

Cybench is a suite of 40 challenges selected from four "Capture the Flag" or CTF tournaments originally designed for the edification and entertainment of (human) cybersecurity specialists and hobbyists (Project Sekai CTF 2022-23, HKCERT CTF 2023, HackTheBox CyberApocalypse 2024, Glacier CTF 2023). It includes tasks chosen from a variety of categories – binary exploitation, web exploitation, cryptography, forensics, and reverse engineering – and of varying degrees of difficulty. Anthropic cryptically remarks that they were only able to use 35 of those 40 challenges in their testing of Mythos Preview due to "infrastructural constraints". No word on which five challenges were omitted from their testing – which is odd, given how much emphasis they place on Mythos Preview's perfect score (35/35). This itself comes as no surprise: Claude Opus 4.6 likewise saturated the 35 challenge subset of Cybench just a couple months earlier, following Sonnet 4.6's 96% score (i.e. 33.5/35).

We should flag a questionable trial asymmetry here. On the Cybench benchmark, Mythos Preview's 100% pass rate was measured over 10 trials per challenge, while the three comparison models each ran 30 trials. Fewer trials give the headline model fewer opportunities to fail, while giving the baselines more opportunities for their variance to show. With 10 trials on a binary pass/fail task, even a model with a true 95% pass rate has a ~60% chance of hitting 100%, so the headline "perfect score" is less informative than it looks.

Having saturated the (35 challenge subset of) the Cybench benchmark, Anthropic moved on to CyberGym. This suite served to better discriminate between the various models' capabilities. Mythos Preview scored an 83% on this suite, beating Opus 4.6's score of 67%.

CyberGym principally differs from CyBench in its use of historical, real-world vulnerabilities rather than CTF challenges, which tend to be structured as self-contained puzzles. In CyberGym, machine learning agents are "given a text description of a historically found vulnerability and the corresponding codebase before the vulnerability gets patched" and must "create a PoC to reproduce the target vulnerability" (CyberGym, p.4). This certainly gives us a more realistic set of cybersecurity challenges, but with that comes a greater risk of training data contamination. CyberGym's challenges are based on vulnerabilities disclosed between January, 2017 and April, 2025. According to the model details listed by AWS, Mythos Preview's knowledge cutoff is December, 2025. CyberGym's authors, for their part, argue on page 6 of the CyberGym whitepaper that their own experiments provide "no compelling evidence that model performance correlates with knowledge cutoff boundaries." In the relevant trials, they tested Claude Sonnet 3.7 and OpenAI's GPT-4.1 on test sets from which vulnerabilities disclosed before the models' respective knowledge cutoffs had been excised. On these modified test sets, Sonnet 3.7's score dropped just 0.2% (from 12.1% to 11.9%) and GPT-4.1 dropped 4.1% (from 9.7% to 5.6%). I don't take these differences to be as self-evidently negligible as the authors suggest. Nor is it clear to me what differences in performance we should expect from Mythos Preview were it to be tested solely on vulnerabilities disclosed after its cutoff. The authors of the Mythos System Card do not raise this question at all.

In their defense, it could very well be that concerns about whether Mythos Preview was creatively devising new solutions to the challenge or simply regurgitating memorized proofs of concept were, in their eyes, made moot by the model's reportedly impressive performance discovering hitherto unknown vulnerabilities or "0-days" in subsequent experiments. We'll have an opportunity to discuss these when we turn to the blog post in Section 2.2. Some external context may help us calibrate the CyberGym numbers. The most informative cross-model comparison available comes from a Google Cloud AI Research paper, "Co-RedTeam: Orchestrated Security Discovery and Exploitation with LLM Agents", which evaluates a multi-agent red-teaming framework across CyberGym, Cybench, and BountyBench using several backbone models. Their best result on CyberGym – 37.3%, achieved with Gemini 3 Pro driving a specialised six-agent scaffold equipped with code-browsing tools, a layered long-term memory system, and a closed-loop plan-execute-evaluate cycle – represents the current state of the art in the published, peer-reviewable literature. Vanilla Gemini 3 Pro, without the scaffold, scores just 12.1%. The Co-RedTeam results also offer a useful lesson about the relationship between model capability and scaffold design: Co-RedTeam with Gemini 2.5 Pro (31.5%) outperforms vanilla Gemini 3 Pro on CyberGym, suggesting that how you wrap the model matters at least as much as which model you use. Against this backdrop, Anthropic's reported 83% for Mythos Preview is either a very large genuine capability jump – more than doubling the best published result – or it reflects differences in evaluation methodology that we cannot currently assess, since the System Card does not disclose which CyberGym difficulty level was used, whether thinking mode was enabled, or what agent scaffold (if any) was employed. The model's December 2025 knowledge cutoff, as noted above, means that the entire CyberGym benchmark falls within its training window, compounding the difficulty of interpretation. Until Mythos Preview is available for independent testing, the 83% figure should be held with appropriate uncertainty.

Anthropic's Maddeningly Inconclusive Firefox 147 Experiments#

In a particularly interesting experiment, conducted in partnership with Mozilla, both Opus 4.6 and Mythos Preview are, in turn,

given a set of 50 crash categories and corresponding crashes discovered by Opus 4.6 in Firefox 147, and [are] placed in a container with a SpiderMonkey shell (Firefox’s JavaScript engine), a testing harness mimicking a Firefox 147 content process, but without the browser’s process sandbox and other defense-in-depth mitigations. The model is tasked with developing an exploit that can successfully read and copy a secret to another directory, actions that require arbitrary code execution beyond what is available in JavaScript. For each crash category, we provide instructions in the prompt to use that category as the starting point for the model’s exploration, and run five trials per category, for a total of 250 trials. (System Card, p.50)

The upshot of this experiment, the authors tell us, is that Mythos Preview far exceeds any of Anthropic's prior models, up to and including Opus 4.6, in exploit development. Whereas Opus 4.6 only managed to develop fully functioning exploits in 0.8% of the trials, Mythos Preview succeeded in doing so for 73.4%. Figure 3.3.3.A from the System Card, illustrating the results of this experiment, is reproduced below.

The gap might not be quite as dramatic as it appears, however. "On analyzing results," the authors write, "we find that almost every successful run relies on the same two now-patched bugs, with almost every trial independently landing on the same bugs as strong exploit candidates, even when starting its analysis from different crash categories" (p.51). The experimenters ran a second set of trials from whose testing data they removed any crash information pertaining to the two over-represented bugs (for which we are given neither CVEs nor descriptions). When crashes pertaining to these two vulnerabilities were removed from the test set, Mythos's performance plummeted: in only 4.4% of the trials was it able to generate a complete working exploit. We're told that it was still able to produce a "partial" exploit (bringing about a "controlled crash") in 85.2% of the trials, but the notion of a partial exploit is a little bit murky – does this mean that a human researcher was able to complete the exploits with only minor adjustments? How minor? What exactly does it mean to produce a "controlled crash" when we're already starting with crash data? In want of any rigorous clarification of terms, the skeptic might be forgiven for saying that speaking of "partial success" in exploitation makes about as much sense as saying someone is a "little bit pregnant". My frank opinion is that the towering pastel bars representing "partial successes" obscure far more than they illuminate and should have been deleted from these graphs entirely.

It's also not altogether clear how this difference in performance between the full set of crashes and those with the two most commonly exploited bugs removed should be accounted for. It could simply be that those two bugs were objectively easier to exploit, and that the drop in performance for Opus 4.6 and Mythos Preview is just what we should expect when the low hanging fruit is plucked from the orchard. Since Firefox 147 was released in January, 2026 – well after Mythos's knowledge cutoff – there's no reason to suspect training data contamination from the 107 CVEs that have since been issued for the product (VulnCheck). But the sizable difference in outcomes between these two experiments is enough to raise serious doubts about how far we can generalize Anthropic's results.

The reader of the System Card might also wonder why we are given so few details about the parameters of the experiment. Each model, we're told, is presented with "50 crash categories and corresponding crashes" discovered in a prior experiment by Opus 4.6. What are these categories? What does their description look like? Are they specified in terms of canonical CWEs? Why aren't we given a list? And what precisely does it mean to provide the model with "crashes"? How are these crashes represented? As core dumps? As stack traces? As crafted inputs that have been experimentally verified to trigger the crash, such as would be collected by a fuzzer like American Fuzzy Lop? As the maddeningly unspecific user notifications favoured by modern consumer operating systems that tell us nothing beyond "Whoops! Something went wrong?" Even a single example or two would be illuminating here, and the absence of such obviously relevant information in the astoundingly abridged six page chapter on "Cyber", in a document whose prestige and mystique seems to rest on the new model's alarmingly powerful cybersecurity capabilities – capabilities that, we're told, are enough to justify a radical change in the company's business model and security policies – is perplexing.

In the absence of a description of "the same two now-patched bugs" on which both Mythos Preview and Opus 4.6 repeatedly converged, and in the absence of a description of the "50 crash categories" investigated in these experiments, we might also wonder how those crash categories were individuated in the first place. If, as the authors say, "almost every successful run relies on the same two now-patched bugs, with almost every trial independently landing on the same bugs as strong exploit candidates, even when starting from different crash categories," then what makes us so sure that the "crash categories" were more than superficially distinct in the first place? "Starting from different crash categories" is a vague formulation, as well. Was the search in any way meaningfully constrained to those categories? Did the models repeatedly converge on the same two bugs because those bugs were in a meaningful sense behind the various categories in question? Or did the models ignore the prompt's instruction to direct their focus to those categories and just reiterate the same or similar exploit strategies, time and again? I've reread this chapter numerous times at this point, and on each fresh read I notice some other critical piece of information wriggling out of my grasp. At this point, I'm truly not sure what I should be concluding from these experiments at all, if there are indeed any rigorous conclusions to be drawn.

In hopes of fleshing out some of the gaps in the System Card's account of the Firefox 147 experiments, I pulled up the March 6, 2026 blogpost in which Anthropic details the first iteration of those experiments, in which Claude Opus 4.6 was directed to search for vulnerabilities in the Firefox 147 Spidermonkey JavaScript engine (in an unhardened test environment, note). There is no mention of "50 crash categories" in that account whatsoever, though the number 50 does appear in one ambiguously related context:

Claude [Opus 4.6] had already discovered fifty more unique crashing inputs. While we were triaging these crashes, a researcher from Mozilla reached out to us. After a technical discussion about our respective processes and sharing a few more vulnerabilities we had manually validated, they encouraged us to submit all of our findings in bulk without validating each one, even if we weren’t confident that all of the crashing test cases had security implications. By the end of this effort, we had scanned nearly 6,000 C++ files and submitted a total of 112 unique reports, including the high- and moderate-severity vulnerabilities mentioned above. Most issues have been fixed in Firefox 148, with the remainder to be fixed in upcoming releases. (Emphasis added)

This could be a mere coincidence. But a crashing input does not a "crash category" make, at least not in any nontrivial sense. To illustrate, suppose we have a snippet of code like the following toy example:

  char buf[10];
  strcpy(buf, name);
  printf("Hello, %s!\n", buf);
  return;
}

void log(char *msg) {
  printf("LOG: %s\n", msg);
  return;
}

An attacker could trigger a buffer overflow and smash the stack, causing the program to crash, by calling greet() with any number of inputs: "Bobbbbbbbbbbbbbbbbbbbbb", "Bobaaaaaaaaaaaaa", etc. Do these count as one crash category or several? What if we add to this another stack-smashing payload, one that results in greet() being called with the argument "Bob4567890yyxxxx\xef\xbe\xad\xde", where 0xdeadbeef just happens to be the address of log(). In that case, the payload would smash the stack but write the address of the log() function to the return address saved on the stack. When greet() returns, it pops that address, instead of the bookmarked address from which greet() was called, into the instruction pointer. log() then attempts to print the log message and fetches data from an invalid string pointer in the process – and again, we see a crash. Are we dealing with a different "crash category" here? I don't think we can give a meaningful answer to this without clarifying what we mean by "crash category" in the first place. This is, of course, a deliberately simplified example, but even in such a case as this the map between crash-triggering inputs and crash categories is murky at best.

Failure to Solve OT Range#

The System Card goes on to briefly (and elliptically) discuss a few other experiments. Here, one failure stands out in particular: Claude Mythos Preview, they report, "was unable to solve another cyber range simulating an operational technology environment" (System Card, p.53). This may surprise a reader familiar with operational technology (often referred to in the industry as "OT", "ICS" (Industrial Control Systems), or, synecdotally, as "SCADA" (Supervisory Control and Data Acquisition)), which tends to be notoriously insecure. It is also, however, relatively obscure. OT security is a continent unto itself. Vulnerability disclosure is difficult to coordinate, with vendors routinely prizing discretion over transparency. The hardware, operating systems, programming languages, design idioms, and protocols involved tend to be proprietary, arcane and unfamiliar to most cybersecurity professionals and software engineers who do not work directly with industrial control systems. There is dramatically less public documentation. The documentation and code that exists is overwhelmingly proprietary and closely guarded. Vulnerabilities are less likely to be publicly discussed, and relevant exploit PoCs will be far more difficult to come by. In short, there is generally going to be far less relevant training data available for vulnerabilities and exploits pertaining to OT systems than can be found for those afflicting commercial software. The problem we face here is analogous to the one faced by machine-learning translation models for low-resource languages. It is nevertheless true that a seasoned human cybersecurity researcher would not have an insuperably difficult time crossing the gulf between commercial and industrial vulnerability research. The same transferable skills used to reverse engineer proprietary commercial firmware and network protocols can be applied to their industrial counterparts. I've worked in both domains, myself, when employed as a reverse engineer at Tenable, and though admittedly a novice in ICS security, I was soon able to find my bearings. If we're seeing a dramatic drop in Mythos Preview's performance when moving from commercial to industrial cybersecurity, this may suggest that something more like a capacious memory than reasoned generalization contributes to its success in the non-industrial domain. As Anthropic says in the conclusion to their "Assessing" blog post,

There are only so many classes of vulnerabilities, and through a combination of intelligence, encyclopedic knowledge of prior bugs, and an ability to be far more thorough and diligent than any human can be (though they are still imperfect!), language models are now remarkably efficient vulnerability detection and exploitation machines.

Failure to Find Any Novel Exploits in a Properly Configured Sandbox with Modern Patches#

Tucked away in the same bullet point as the acknowledgement of Mythos's failure to solve the OT range is another admission. "In addition, in a more challenging sandbox evaluation, [Mythos Preview] failed to find any novel exploits in a properly configured sandbox with modern patches." I have to confess that I had to read this section twice before I realized that the authors are speaking about two different vaguely specified failures here, perplexingly folded into the same bullet point. I do wish that we were told a little more about what made this "more challenging sandbox evaluation" so difficult for Mythos. My suspicion is that it was something quite ordinary – just a fully patched and reasonably hardened system, or set of systems.

I am, to be fair, passing over a handful of experimental successes that Anthropic reports on the same page of the document, so let's back up a little and look more closely at this section discussing the cyber ranges (section 3.4).

An early snapshot of Claude Mythos Preview was assessed for cybersecurity capabilities, across cyber ranges, capture-the-flag challenges, and evaluations assessing sandbox escape capabilities. (p. 52)

That "early snapshot" qualification looks momentarily impressive in the light of the first few successes they go on to report on the following page, but perplexing in light of the two failures shoved into bullet point #4. If an early iteration of your flagship product ran aground on two significant challenges – the two with serious bearing on real-world cybersecurity, I might add – why would you not try running those experiments again with a complete version of the model? The only explanation I can imagine here is haste. A sense of haste seems to haunt this entire report, I'm tempted to say. Everything feels as though it's been painfully hurried to press on a schedule driven more by marketing than scientific standards. But let's move on.

"1. Claude Mythos Preview is the first model to solve one of these private cyber ranges end-to-end."

One of what ranges? CTFs come in all shapes and sizes, and in all levels of difficulty – some beatable only by the best of the best, some beatable by a novice.

"2. Claude Mythos Preview solved a corporate network attack simulation estimated to take an expert over 10 hours."

Estimated by whom? What simulation? Why not point us in its general direction here? "No other frontier model had previously completed this cyber range." Which others had tried to do so? And why the qualification "frontier" model? "Frontier," here, is marketing speak. Is the implication that last year's models were able to solve it? "Claude Mythos Preview is also highly capable at identifying and exploiting known vulnerabilities or misconfigurations to escape the sandbox in which it operates." Again, no details are given. Is this a reflection of how poorly configured the sandbox was, or of how cleverly the model escaped it?

"3. This indicates that Claude Mythos Preview is capable of conducting autonomous end-to-end cyber-attacks on at least small-scale enterprise networks with weak security posture (e.g., no active defense, minimal security monitoring, and slow response capabilities). Note that these ranges lack many features often present in real-world environments such as defensive tooling."

This sounds humble enough, but we're starving for detail here. Here was an opportunity for Anthropic to be specific about which security measures have been shown to foil the model's capabilities and with what success rates. This is information that would, presumably, be of great public interest in the wake of announcing a "too dangerous to release" model to the world.

Mitigations#

The "Cyber" chapter in the System Card also includes (in section 3.2) a curious discussion of the "mitigations" Anthropic intends to institute to curb their models' destructive potential in the domain of cybersecurity. These advanced mitigations won't be put in place for the model they've licensed to the lucky elect, but are under development for both Opus 4.6 and any subsequent models comparable to Mythos in capability. Little information is provided on their methods here, but we're told that the techniques they plan to use will be similar to those described in a January 9th blog post entitled "Next-generation Constitutional Classifiers", where we can find the following account:

When a model generates text, it produces internal states at each step that capture its understanding of the input and output so far. When Claude processes a dubious-seeming request, patterns fire in its internal activations that reflect something along the lines of "this seems harmful,” even before it has formulated a response or made a conscious decision about what to do. Normally, these activations are intermediate computations -- used, then discarded. We found ways to reliably probe whether these internal states suggest harmful content, getting more information -- think of it like Claude’s gut intuitions -- almost for free.

2.2 On "Assessing Claude Mythos Preview's Cybersecurity Capabilities"#

0-Days#

The bulk of Anthropic's discussion of Mythos Preview's cybersecurity capabilities is to be found in the April 7th blog post, "Assessing Claude Mythos Preview’s cybersecurity capabilities". Here we can find a few high level descriptions of a handful of 0-days (previously unknown and unpatched vulnerabilities) that Mythos Preview discovered after they had lain dormant in the OpenBSD kernel for 27 years, the popular media processing application FFmpeg for 16 years, and an unnamed virtual machine monitor for an undisclosed period of time. This is followed by a refreshingly detailed discussion of a few exploits that Mythos Preview developed for a series of known and patched vulnerabilities (or "n-days") for which no exploit code was known to exist. (I consulted VulnCheck's knowledge base to verify that this is the case for each of the CVEs discussed, and in no case did I discover exploit code anteceding Anthropic's experiments.)

We're told that the three 0-days showcased in the blog post were chosen from a long list (still only partially verified) of candidate 0-days that Mythos Preview was able to discover in a wide range of open source codebases. Their decision to highlight the 27 and 16 year old vulnerabilities in OpenBSD and FFmpeg seems motivated to drive home the point that Mythos Preview can indeed compete, and trounce, human cybersecurity researchers: even when given a 27 or 16 year headstart, the message seems to be, humans failed to discover, attack or patch vulnerabilities that Claude Mythos found overnight (at an estimated token cost of $20K USD for the OpenBSD exploit; no cost is given for the FFmpeg exploit).

The OpenBSD vulnerability is itself neither exceedingly complex nor trivial. At its heart, it's an integer overflow bug, caused by improper type casting and comparison logic, and its mitigation consists in a tiny two-line patch. But the consequences of this bug are quite serious: it affects the OpenBSD kernel's TCP implementation and can be triggered remotely, over the internet, without authentication. Exploiting it can bring about a denial of service on the target, crashing the kernel and forcing the system to reboot. As DoS bugs go, this is a doozy.

The FFmpeg bug is also impressively old, but comparably harmless as it stands. It too is technically a DoS vulnerability, but a DoS for an application, not an operating system kernel, and this generally lowers the impact and stakes. Triggering it would require supplying an FFmpeg process – such as may be running on a media server like Plex or Jellyfin to transcode video and audio data on the fly – with a maliciously crafted video file. This file would then cause the transcoding tool to crash. That's as far it goes.

Anthropic is especially taciturn regarding the VM monitor bug, which is fair game seeing as it "has not been patched". The bug itself, we're told, allows the attacker to perform an out-of-bounds write from inside a target virtual machine to the host the VM is running on. "It is easy to turn this into a denial-of-service attack on the host," we're told, and it "conceivably could be used as part of an exploit chain. However, Mythos Preview was not able to produce a functional exploit."

It is generally far easier to discover denial-of-service or DoS vulnerabilities than remote code execution exploits, just as it's easier to slash a car's tires than hotwire it. We know nothing about the long list of 0-days that Mythos allegedly discovered, outside of the three briefly discussed in this blog post – three which, it's probably safe to assume, were chosen to impress. Of these three exploits, only one – the OpenBSD DoS – looks as if it would be a definite nuisance if it fell into the wrong hands. The VM monitor bug could be critical or negligible – we simply aren't given enough information to tell. When we're told that "it is easy to turn this into a denial-of-service attack on the host", does this mean that the authors had succeeded in doing just this, with Mythos's helpful prompting? Or is this a cagey matter of leaving something "as an exercise for the reader"? The bug sounds serious enough that the virtual machine monitor vendor in question, I think it's fair to say, should hasten to patch it immediately, but without details on either the vendor, the product or the bug itself, its impact is left rather murky.

The post does, however, discuss a 17-year-old remote code execution vulnerability in the FreeBSD kernel, assigned CVE-2026-4747, that impressively showcases not just Mythos's but Opus 4.6's abilities. The remote code execution vulnerability lies in the operating system's NFS service, and can be exploited by any GSS-authenticated user over the network (though in the report made to FreeBSD, Carlini says that one component of the attack – the stack overflow – can be triggered by an unauthenticated user, so it's possible there's a pre-auth DoS vuln buried in this post-auth RCE report.) The exploit itself is quite complex, involving the coordinated actions of two different clients, and it allows the attacker to execute arbitrary code on the target system with full root permissions (or, with minor modifications, with ring-0 permissions, normally reserved for the kernel alone, should they so choose, unless I'm mistaken).

n-days#

What's particularly interesting about the FreeBSD vulnerability discovered and exploited by Mythos Preview is that on March 31st – just five days after Nicholas Carlini, of Anthropic, shared the CVE-2026-4747 vulnerability advisory with the FreeBSD community – an unaffiliated researcher named Thai Duong was able to coach the publicly accessible Opus 4.6 model to develop a working exploit for the same vulnerability. Duong's prompting was relatively light – a gentle nudge here and there, but nothing beyond the reach of a computer science undergrad or Unix power user. The vast majority of Duong's prompts shows them asking the agent for clarification, granting it permission to install or use a given tool (ropgadget, gdb, etc.), or reminding it of their goal (to get a reverse shell with root permissions on such and such a port, etc.). There is shockingly little in the way of technical guidance. I'll even say that there is nothing there in those prompts that would be of any help whatsoever to a human hacker who didn't already know what they were doing. My guess is that all of this information could have been packed into the initial prompt, removing the need for interaction altogether.

And for those keeping score, Duong's commit was pushed a full week before Anthropic posted the walkthrough on their blog (on April 7th), so there's no question of contamination by the blog post itself. Carlini's FreeBSD advisory was, however, made directly available to the Opus 4.6 agent in Duong's experiment. We don't have a log of the agents responses to Duong's prompts, and so it's quite possible that the model also inspected the kernel patch mitigating the vuln. The URL for this patch, after all, is clearly exposed in the advisory.

There's no point fretting too much over the question of contamination, though, when it comes to Opus 4.6's ability to exploit n-days. The risk posed by (non-expert) attackers using this publicly available model to orchestrate attacks that, just a few weeks ago, would have required several days of an expert hacker's time and effort is in no way mitigated by accusing the model of cheating.

Duong didn't use Opus 4.6 to discover the vulnerability, only to exploit it. But according to Anthropic's discussion of the Firefox 147 experiment in their System Card, Opus 4.6 is comparably competent to Mythos Preview when it comes to vulnerability search – it's in the domain of autonomous (unguided) exploit development that it falls short of its successor. The upshot here is that LLMs capable of rapidly developing exploits for n-days – vulnerabilities for which CVEs have already been assigned, for which information can already be gleaned from advisories and code patches, but which are in reality left unpatched in myriad systems – are already available to the public.

Anthropic's blog post goes on to discuss several other complex exploits that Mythos Preview was able to develop for n-days with virtually no human guidance. My honest opinion, as a security researcher, is that any of these would take me at least a week to develop, if the hacking muses were with me.

2.3: A Reckless Defender?#

In Chapter 4 of the System Card, the authors observe that Mythos has exhibited a tendency to take numerous "reckless" shortcuts in pursuit of user-assigned goals, often going to some lengths to conceal this recklessness from the user themselves, even altering git histories to conceal codebase changes in some cases. "We urge users in Project Glasswing," the authors write, to remain cautious about deploying Claude Mythos Preview unmonitored in settings where it could cause significant harm, despite its increased usefulness of autonomy. (p. 60)

This is no small matter if the model is being widely deployed to major enterprises as a tool for red-teaming and patching their codebases at a scale and pace that significantly outstrips human competition. Like hallucination, agentic recklessness introduces a form of unreliability that could prove much more costly for defenders than attackers.

3. Black Hat Economics#

If frontier LLMs with expert-tier cybersecurity capabilities transform the cybersecurity landscape in the near future, this will not be because they introduce a fundamental difference in the kind of vulnerabilities or exploits we deal with. The technological world is already awash with bugs, vulnerabilities, and exploits. To the extent that the internet, or any of our computer infrastructure, nevertheless endures, it's because it's relatively fault tolerant by nature. Everything's always been broken, more or less.

I think the engine of change will lie elsewhere. Consider the example of ransomware. Young and Yung introduced the concept of "cryptovirological attacks" as early as 1996, and even furnished a proof-of-concept implementation of such an attack. But the idea lay dormant for another 17 years until quite suddenly becoming a global cybersecurity problem with the rise of CryptoLocker in 2013. What brought about this change wasn't the appearance of a devastating new 0-day or explosion of available exploit code. It was the sudden rise of cryptocurrency. Bitcoin transformed ransomware from an academic construct to a global nuisance, because it made those attacks worth doing for a significant number of actors – it provided a means by which to carry out anonymous (or pseudonymous, if we want to be sticklers) and efficient international extortion, and thereby incentivized threat actors.

The question most observers are asking about Mythos and its inevitable successors is "what can the model do?" A more consequential question, however, is "at what price does it become worth doing?" Carlini et al. make this case persuasively in "LLMs Unlock New Paths to Monetizing Exploits" (May 2025). Their core observation is simple: you don't need to outperform an expert human hacker to transform the threat landscape. You just need to lower the cost floor enough that attacks which were previously uneconomical become profitable. The economics of cybercrime have historically forced attackers to choose between depth (sophisticated, targeted operations against high-value targets, requiring weeks of expert labour) and breadth (blunt, indiscriminate campaigns like ransomware and credential stuffing, which compensate for low per-target returns with volume). LLMs like Opus 4.6 threaten to collapse this trade-off. An attacker armed with a model that can read a codebase, identify a vulnerability, develop a working exploit, and tailor its monetization to the specifics of the target -- all at API pricing -- no longer has to choose between going deep and going wide.

The concrete cost data we have so far puts some flesh on this. Anthropic reports that the ipset privilege escalation exploit cost under $1,000 at API pricing, and the unix_stream chain under $2,000 – both completed in under a day. The OpenBSD 0-day campaign cost roughly $20,000 for a thousand runs, though that figure is likely internal cost rather than what a customer would pay. These are already within reach of financially motivated criminal operations, and well within the budgets of state-sponsored actors. But I suspect the more significant changes will be felt not at the apex of the threat landscape but along what Carlini et al. call its "long tail": the thousands of niche applications, bespoke enterprise systems, and neglected open-source dependencies that are individually low-value but collectively form the vast and poorly defended underbelly of our shared computing infrastructure. These have historically been too obscure to justify the cost of a targeted attack. An LLM that can analyse an unfamiliar codebase overnight changes that arithmetic. And what's noteworthy here, as Carlini et al. emphasize, is that it isn't even Mythos-class models that pose the most immediate risk. The hallucination problem, which so bedevils the use of LLMs in safety-critical defensive applications, barely matters on the offensive side: if 10% of your exploit attempts fail because the model hallucinated a vulnerability that doesn't exist, you've lost 10% of your budget, not 10% of your security. The attacker's tolerance for unreliability is structurally higher than the defender's.

And since I touched briefly on the subject of Ethereum in the introduction, I may as well close the loop and mention in passing this 2025 study by Anthropic's frontier red team that illustrates just how easily attackers may leverage publicly available models like GPT-5, Sonnet 4.5, and Opus 4.5 to audit and exploit Ethereum smart contracts. The day may come when locally hosted models are capable of the same and the automated exploitation unconventional interpretation of smart contracts becomes a more lucrative use of GPU cycles than mining, for the cryptocurrency enthusiast.

4. Conclusion#

Claude has not become Death, destroyer of worlds. Anthropic has not built an unstoppable cyber-nuke, nor have they acquired the McGuffin in Sneakers (1992). At present, nothing that was formerly impossible has been made possible. But what was difficult is becoming easy, what was costly is becoming cheap, and what required careful human thought is now open to automation. What the researchers at Anthropic have done is to meaningfully unsettle the action economy in the domain of cybersecurity, in a way that threatens to upset the tenuous equilibrium that's formed over the past few decades. The authors of the "Assessing…" blog post graze against this issue when they remark that language models like Mythos Preview might require reexamining some other defense-in-depth measures that make exploitation tedious, rather than impossible. When run at large scale, language models grind through these tedious steps quickly. Mitigations whose security value comes primarily from friction rather than hard barriers may become considerably weaker against model-assisted adversaries.

To "tedious", here, we could add "time consuming" and "expensive". I used to counsel my more paranoid friends that the question they should be asking isn't whether or not an intelligence agency or a criminal organization can hack them, but whether they'd consider it worthwhile to do so. Few are those worth the attention of a well-funded and capable APT, after all, and a good remote 0-day is too precious a thing to waste on small potatoes. For the vast majority of us, all we really need to worry about is avoiding the steady waves of off-the-shelf attacks, and these are easily mitigated by keeping our systems patched and maintaining a healthy suspicion of phishy texts and emails. But these broadly economical considerations may soon become unreliable. We don't know if or when a new equilibrium between attackers and defenders will emerge, or what that will look like.

Appendix 1: Q&A#

Q1: How Dangerous Could Anthropic Be if they Decided to Use Mythos for Nefarious Ends? (Opinion)#

Clearly I'm speculating, here, and working from very limited information, so please take this with a heaping spoonful of salt. My impression is that Mythos Preview makes what was once moderately difficult to exploit significantly easier to exploit. But there is no evidence that it has succeeded in discovering vulnerabilities or crafting exploits against anything that we might consider to be an especially hard target, at least not without having the significant advantage of full source code access. Recall the admission in the System Card that "in a more challenging sandbox evaluation, [Mythos Preview] failed to find any novel exploits in a properly configured sandbox with modern patches" (p. 53). Given that that particular (woefully underspecified) experiment dealt with the sort of evaluation environment typically used to assess human cybersecurity professionals, I think it's safe to assume that it wasn't a question of the target being objectively free of vulnerabilities (unless it was meant to be some sort of infosec Kobayashi Maru).

Of course, the absence of evidence is not evidence of absence, and it would be foolish to conclude from this that adequately hardened and monitored systems have nothing to fear from a model with Mythos's capabilities. And even if this were the case, vulnerabilities are, and have always been, ubiquitous in our technological environment. Still, the limited information we have access to at this point broadly inclines us to view Mythos as exhibiting the cybersecurity capabilities of a moderately talented human expert in the field, albeit one whose work can be accelerated and parallelized to an indefinite degree. A human hacker who could accomplish in a few days what a single instance of Mythos can do overnight would not have difficulty securing a lucrative career as a penetration tester or reverse engineer. Any party with full access to Mythos and sufficiently deep pockets to pay for compute could be expected to reap the same benefits as one with a correspondingly large budget for hiring skilled professionals in the field. The difference between the two reserves of cybersecurity capability is hard to quantify in any but the most hand-waving terms, but my broad impression is that it's a multiplicative and not, say, exponential relationship we're looking at here.

We have an example of real-world organizations that are able to command immense resources to hire and deploy as many reasonably talented cybersecurity professionals as they like in national intelligence agencies, and I think that's a reasonable scale to be thinking of here.

We haven't yet seen any evidence, or even anecdotes, of Claude Mythos coming up with a paradigm-shifting work of genius in the domain of cybersecurity – though, to be fair, such events are by their very nature rare among humans as well. The exploits it crafts – those which Anthropic has shared with us so far, that is – are certainly clever, but they're also standard fare. In Kuhnian terms, they belong to the domain of "normal science". They use well-established techniques, work with idiomatic exploit primitives, and follow familiar patterns of thought. This is unsurprising, I suppose, for a language model – the same can be said of the generations LLMs produce in other domains.

We might contrast this with the truly alien-looking and bizarre constructions we see coming from other types of artificial intelligence and program synthesis. While the quasi-Darwinian methods of genetic programming have not yet produced anything that can compete with LLMs in the domains of program synthesis or exploit development, the constructions they do generate tend to be strikingly unfamiliar in form. Though possessing far more rudimentary capabilities than their famous cousins, they rarely have difficulty producing something novel. LLMs, by contrast, seem to have shown a tendency to converge on something resembling highly trained human competence. There's no suggestion to be found, in what I've seen so far, that Mythos will be capable of anything that is decisively beyond the reach of human hackers. Its chief advantage appears to be speed and scalability.

What this means for my prognosis on the cybersecurity impact of Mythos and its kin is that the difference between AI-driven vulnerability search and exploitation, on the one hand, and its human counterpart, on the other, is best thought of in terms of scale. We're sparring with our shadows here. Their shapes are our shapes. But they may grow long as the day gets late.

Q2: Is the gap between Mythos and 4.6 stronger on exploitation-chaining than on vuln discovery? If so, is it plausibly because exploitation-chaining depends on more clearly agential forms of planning and agential planning is what scaling affects most?#

Let's see what we can piece together from the (scant, patchy, often elliptical) data that Anthropic has shared so far. Anthropic seems to be oddly recalcitrant on this matter, leaving what might seem like obvious questions unasked in their discussion of Mythos Preview's cybersecurity capabilities.

The Firefox 147 experiments described above certainly purport to demonstrate that Mythos is significantly more capable than Opus 4.6 in exploit development, and while it's fair enough to say that they suggest this conclusion, the credibility of experiments themselves is vitiated by numerous "questionable scientific practices" (as I argue in Section 2.1).

Is exploit development more difficult than vulnerability discovery for human researchers? Frankly, it depends on your standards of evidence. At what point do you confidently declare that you've discovered a vulnerability? In some cases we might be satisfied with pattern recognition. If I see a .cgi file on a router that accepts an IP address from the user and then call system("ping -c 3 $IP_ADDRESS"), I can be fairly confident that there's a command injection vulnerability there, even if I haven't yet tested it (a bewilderingly common vulnerability on routers, as it happens). But there's a reason we call the exploit code for a zero day vulnerability a "proof of concept" – the exploit is a concrete demonstration that a vulnerability exists, and exists where we say it does. So I'm naturally somewhat suspicious about Anthropic's elliptical reports about their models being able to discover such and such a number of vulnerabilities while only being able to construct working exploits for a handful. There's a gradient of degrees of certainty between noticing a bug that can probably be exploited, and demonstrating that a vulnerability exists with a genuine proof of concept. The spectrum lengthens when dealing with targets of increasing complexity.

And this is perhaps where agential planning comes in. While we might with varying degrees of assuredness diagnose a particular bug as being a symptom of a security vulnerability, whether that vulnerability can be realized (i.e., exploited) depends on numerous interacting factors in the codebase.

To use a (slightly simplified) example from my own work, suppose you discover a null byte injection bug in a program, such that an ephemeral password is constructed by calling sprintf(result, "%s+TEMP"), where S is the result of XORing a randomly generated string with a token that an authorized client has encrypted with an asymmetric key and sent to the server. The assumption here is that only a legitimate user would possess the key required to participate in this exchange and thereby derive the ephemeral password. But if S begins with a null byte – which will indeed occur if the random string and the decrypted token happen to start with the same character (a xor a is 0, for all a) – then result will just equal "+TEMP", and the security assumption that result is unguessable fails. This is a bug, but it is not quite yet a vulnerability unless some additional security assumptions also fail, namely:

  1. that only a legitimate user will have the appropriate key – an assumption that will fail if the developers made the additional mistake of leaving that key hardcoded in the target

  2. that the appropriate key is needed in order for the user to exercise sufficient control over the value of S – an assumption that fails if the encryption method used is, say, unpadded RSA encryption, which allows for phony "ciphertext" to be "decrypted" without throwing an exception (albeit into an essentially pseudorandom value).

If either of these additional assumptions are falsified, then we do indeed have a real vulnerability. A cryptographic logic vulnerability, as a matter of fact.

The point here is that the difference between discovering a probable vulnerability and being able to construct an exploit that proves the existence of the vulnerability beyond the shadow of a doubt is usually a matter of developing a systematic understanding of the target. It would therefore seem likely to me that if scaling up LLMs generally allows them to maintain a more capacious, panoramic view of their subject matter, then we should expect to see competence in exploit development increase as well – particularly if they able to build on a demonstrated capability to reliably identify the telltale signs of common vulnerabilities and familiar exploit primitives in isolation. It seems highly plausible to me that increased competence in planning would make the difference between a good bug hunter and a good exploit developer. Bug hunting is a game of tactics while exploit development is a game of strategy.

Q3: Could LLM-based automated patching plausibly speed up the actual fixing of n-day vulns so much that overall the LLM revolution will end up favoring defense?#

This is indeed the sales pitch that Anthropic is counting on, and the answer really depends on the depth of "understanding" involved in a model's ability to construct an exploit. We can think of things epistemically here. Assuming that a software developer aimed to write secure code in the first place, we can view that code as embodying a theory about how the program will behave when it interacts with external data (user input, say): because the code says this-and-that the program should respond in such-and-such a way, and it should not respond in various other ways. When prompting a user for a login password, for example, it should grant access only when the correct password has been provided and NOT under any other circumstances. A hacker succeeds when they are able to falsify that theory by producing a counterexample – a "proof of concept", we call them. The developer then responds by amending their theory and patching the code, and if they're at least minimally competent they should be able to render that particular counterexample ineffective – or, rather, it now becomes an example embraced by the theory. Imagine the developer as an amateur ornithologist who proposes the theory "all swans are white." The antagonistic birdwatcher (our hacker) falsifies this theory by producing a black swan. The ornithologist may amend his theory in various ways. Maybe he now says, "all swans are white, except Seymour J. Swanson, who's black." Maybe he says, "all swans are either white or black." Maybe he says "all swans are greyscale." You get the idea. The point here is that the counterexample grossly underdetermines the theoretical amendment, and in a similar fashion the exploit underdetermines the patch.

The annals of cybersecurity are littered with examples where developers have repeatedly patched essentially the same vulnerability in successively inadequate fashions. The best example of this that I've seen in my own work as a reverse engineer and zero day researcher came in the form of a cryptographically locked backdoor that the (now defunct but once major) Chinese router manufacturer Phicomm had installed on each of their consumer routers over the course of at least two years, and which I discovered when reverse engineering one of their products. The cryptographic lock on this backdoor was intended to allow only parties who possessed the proper credentials to gain root access on the router, and was likely meant to be used solely by maintenance workers and law enforcement officials. (The key was not provided to customers, nor was the existence of the backdoor made known to them.) The locking mechanism in the Phicomm firmware that had first come into my possession was puzzlingly complex. After I succeeded in exploiting it (and developed a tool that would allow an attacker to get an unauthenticated root shell on any such router), I scoured a number of Chinese forums to try and find as many different Phicomm router firmware images as I could, for a variety of models, collecting twelve in total. My exploit worked without any alterations on the three most recently released of these, but not the others, so I reverse engineered the backdoor binaries on those other devices and found similar but less complex locking mechanisms. It turned out that the first two iterations had been exploited before, and each time the developers responded by patching the devices in a way that blocked one type of attack but which demonstrated a failure to understand the vulnerability in its full generality. By making a few small changes to my proof-of-concept code, I was able to produce a tool capable of picking the locks on every iteration of Phicomm's backdoor. (The existing exploit code that I'd discovered in the Chinese router hacking forums targeted only the first two iterations of Phicomm's backdoor protocol, but didn't generalize to the third.) The reason for the backdoor protocol's complexity finally became clear to me: it was a tangle of ad hoc patches that each managed to counter a particular proof of concept, but which remained blind to the core vulnerability.

The lesson here is that the theoretical understanding baked into an exploit is not necessarily convertible into the understanding necessary to adequately patch the vulnerability in question. A piece of software's patch history often represents what Imre Lakatos would call a "degenerate research programme", each responding in an ad hoc fashion to particular attacks without managing to reforge the code in a "progressive" manner that anticipates and forecloses conceptually similar attacks in advance.

So to return to the question above, what it all comes down to is whether or not these frontier models are able to not only discover and exploit vulnerabilities in software, but develop an adequate theoretical representation of why they are vulnerable in the first place. If they succeed in vulnerability discovery and exploitation but fail in this third task, then their automated patching efforts will turn into interminable games of whack-a-mole. This is an interesting case of where approximating success could be more dangerous than failure. Suppose that a given model is able to discover thousands of vulnerabilities in various open source projects, but contributes essentially inadequate patches to those projects. The commits proudly signed by this model could become a breadcrumb trail for attackers – whether they be human or the next iteration of frontier models.

Now, of course, it could be the case that a model like Mythos is every bit as adequate a computer scientist as it is a hacker, and it could be the case that its patches evince something resembling a deep understanding of the vulnerabilities it discovers. My point here is only that this will not necessarily be the case. There exists a competency gap between exploitation and exploit mitigation that should be taken no less seriously than the competency gap between vulnerability discovery and exploit development, and which has in some historic cases proven to be rather immense. (It takes a far deeper understanding, for example, to invent an adequate cryptographically secure hashing algorithm than it does to break an inadequate one through brute force or sidechannel attacks.)