TL;DR
- We test GPT-6 Astra and GPT-6.1 Sol on the brand-new game Order of the Sinking Star, released after they were.
- We build an extremely basic harness and drop puzzles which this harness can’t solve (mostly the timing-dependent ones). That leaves about half of the game’s 920 puzzles, which we then randomly sample.
- Given 1M tokens per task, they can solve 86% of these sampled puzzles. (Sol solves 73%, and Astra solves another 13%.)
- We expect that, given cross-puzzle context and more tokens, the models would solve basically all of the turn-based half of the game.
- These still aren’t fully held out: 13% of the puzzles were public before the models came out (i.e., before or during the models’ training): 11% in a June demo, plus a few close copies of old puzzles.
- In theory, post-training can sample the space of games densely enough that most “new” games will land near something already seen. We guess that labs and their suppliers began doing this at scale around the middle of this year; we guess that this is the dominant term in why new models do so well.
- With caveats (single-player, turn-based / pausable, not too weird), we’re tempted to say that Astra is a general game-solving intelligence.
A familiar experience in AI evaluation in 2026 is pre-saturation: you come up with a new test and find to your dismay that the newest models can already do it. For instance, while working on our Pokémon project, we found that, every time we designed a harder test of out-of-distribution (OOD) generalization, a new model came out and passed it, often the same week. Opus 5 and GPT-5.6 Sol showed signs of saturation on our original Pokémon Red experiments: both models made it out of Rock Tunnel, both got all the badges, and so on. So we scrambled the layout of Rock Tunnel, finding that their performance deteriorated sharply. Then GPT-6 Astra came out and aced the same test. So we raised the bar again and had it try Pokémon Brown, a little-known fan-made game. And it just did it.
We built each new test to find what leading models couldn’t do, but the pace of subsequent model releases rendered our experiments obsolete frustratingly quickly. We couldn’t work out whether this was more likely to be explained by improved generalization or the labs’ ever-increasing portfolio of RL environments.
Our hardest Pokémon test was based on a game that had been public for at least two years before Astra’s training run, ran on the same engine as one of the most-documented games in history, and used an engine that had been an AI evaluation target for a year and a half, so it wasn’t a very strong control task. As such, we thought we should test GPT-6.1 Sol and GPT-6 Astra on something truly new.
Enter Order of the Sinking Star#
With the release of game designer Jonathan Blow’s much-awaited Order of the Sinking Star (OotSS) on October 8, we became excited about the possibility of evaluating frontier models on the game’s brand-new puzzles – ones which could not have been included in the models’ post-training, since both GPT-6 Astra and GPT-6.1 Sol were released before the game came out (the former on September 3, the latter on September 29)1.
Of course, the game being brand new does not guarantee that its puzzles require OOD generalization. (Indeed, as we’ll see later, a small number of the puzzles are perfect matches to old puzzles, and most of the game’s mechanics were at least inspired by them.) But we thought this would still be interesting.
Methodology#
We wanted to evaluate the models on OotSS in a setting as similar as possible to that of human players. To this end, we developed an extremely simple harness. The system prompt simply told the model that it was playing a puzzle game, that its task was to solve it, and that it could use the press tool for any of the game’s native controls. Each puzzle attempt began with a screenshot of the puzzle’s starting screen. Once the model replied with its desired actions, the harness played them through before sending another screenshot of the outcome. The game was paused while the model was thinking, so every attempt was effectively turn-based. This continued until the model either successfully solved the puzzle or exhausted its budget of 1M weighted tokens2. The harness also retained reasoning and made use of server-side compaction at 244,800 tokens, following the methodology of OpenAI’s ARC-AGI-3 harness.
Each puzzle attempt was fully independent: models effectively had to learn the game’s mechanics from scratch in each and every single puzzle attempt. Our evaluations were thus harder than what human players encounter (since they can learn the relevant mechanics through simpler pedagogical puzzles before encountering them again in more complex ones).
Due to the deliberately simple harness, we filtered out any puzzle where we could not be certain whether a solution was achievable with it. In particular, we first filtered out any puzzles that required precise move timing, including all puzzles involving the game’s lilies and rovers (game elements that move on their own, in real time, even while the player stands still). We also removed puzzles where we couldn’t replay the (hard-coded) wiki solution through our harness.
This produced a set of 504 OotSS puzzles (out of the game’s 920), from which we sampled 252 for model evaluations, balanced across the game’s worlds and puzzle lengths and random within each group.
Results#
Quantitative analysis#
We evaluated two models: GPT-6.1 Sol and GPT-6 Astra, both at high effort. Since GPT-6.1 Sol is far cheaper than GPT-6 Astra to evaluate, and since GPT-6 Astra effectively outperforms GPT-6.1 Sol in most logic- or puzzle-related benchmarks we’re aware of (such as MazeBench), we decided to only evaluate GPT-6 Astra on the puzzles that GPT-6.1 Sol was unable to solve.
GPT-6.1 Sol solved 183 OotSS puzzles out of the 252 it was evaluated on (73%); and GPT-6 Astra solved 33 of the 69 puzzles GPT-6.1 Sol was unable to solve, with 36 remaining unsolved by both models.
Qualitative analysis#
For a human player, it is straightforward to learn and remember the mechanics from puzzle to puzzle, but we don’t offer the AI models this privilege: they must start from scratch with each attempt. They also see less than a player would. Our harness sends a screenshot only after each batch of moves has played out, so anything that happens mid-move, like a hint popping up or a character sinking, will go unseen. The models usually fail by not discovering the puzzle’s mechanism in time (as opposed to screwing up object recognition in the screenshots or failing to think ahead enough). Given this, it is very impressive that they can solve such a large fraction of the puzzles – most of which were probably not seen in training.
The models start each puzzle clearly lost, but it is clear that they are capable of learning a significant amount in a single session. They succeed in a relatively human-like way: they don’t brute-force puzzles or try lots of key combinations (though they arguably should, given that they try so few things and can spend hours on puzzles they then fail). Eventually, they have an epiphany, and the rest is straightforward.
The models see the board fairly well, but not precisely. They correctly interpret the vast majority of the components in each level, and where they exist relative to each other, but misinterpret a small number of objects or their placement in roughly half the puzzles. They sometimes think movable slime blocks are other characters, wonder if lines in the art are arrows, and at times struggle to see how mirrors are oriented. They also generally struggle with the camera perspective and 3D objects partially concealing others – something that wouldn’t faze human players – but overall display human-level puzzle-solving abilities (or better) once they understand what they are looking at.
The beginning of an attempt can be a frustrating viewing experience. While human players will try all sorts of hopeless strategies and struggle for a while before finding the solution, the GPT-6 models appear far more lost than a human player would be. Part of this behavior is likely due to the models only viewing the game via intermittent screenshots, and part due to them misattributing objects’ positions.
Authors’ opinions#
We had various disagreements about the models’ performances. Finn described the models as not being able to see the puzzle very well; Niccolo disagreed. Subsequent tests settled this: their visual understanding is good but imprecise. Finn sees this imprecise vision as a significant weakness, causing them to struggle to accomplish even basic things, and takes it as evidence that the models have significantly sub-human levels of visual understanding. Niccolo thinks providing models with intermittent screenshots cannot be fairly judged as an equivalent to a human’s visual experience of the game, where they can continuously study and interpret the puzzle while they think about it.
Is providing the screenshot in context only once a handicap? Is it analogous to giving a human just a momentary glance at the puzzle, for instance? We think this analogy is not exact, because the models can continuously attend to the screenshot tokens while they reason. We ran some further experiments, giving the models the ability to view the screen again (i.e., front of context) and to zoom in. These partially improved their accuracy in re-creating level maps, but they continued to make similar errors in solving them.
Opinion (Finn): AI reasoning, as a “system 2”-like process, seems like a poor substitute for true visual recurrence. This is somewhat fundamental to the current models, as they don’t have a visual-cortex-like component for constantly processing and decoding visual information. With continued scaling, the models may be able to make better use of image tokens at test time, or perhaps future model generations will have cortex-like visual processing. Either way, I would expect the models to be superhuman across a wide range of games once their visual understanding becomes more precise. For now, I think labeling these models as general puzzle-solving intelligences gives a false impression of their abilities, while being an accurate statement once you factor out their visual understanding.
But is OotSS OOD?#
As discussed by the game’s creator, OotSS is built on four indie puzzle games – Heroes of Sokoban, Mirror Isles, Promesst 1 & 2, and Skipping Stones. The studio licensed these games to incorporate their mechanics and level designs, and then developed a huge number of new puzzles that mix components of each game with each other. The creator says the original games “comprise and inspire about the first five percent of Order of the Sinking Star’s gameplay,” while the “other 95 percent explores the seemingly never-ending consequences of smashing these four game-mechanical worlds together.” We checked how many puzzles match those from the original games, just to be sure, and found that at least 6% – 56 of 920 OotSS puzzles – are reused puzzles. Our matching method has limits, so this is a lower bound, but the vast majority of the game’s puzzles seem new, even where their mechanics may not be.
The models could also have trained on the 105 puzzles in a free demo released in June. Counting the reused levels too, 116 of the 920 puzzles were public before the models came out, and 36 of them were in our sample. Sol solved all 36, but leaving them out barely changes the results. Sol’s solve rate drops from 73% to 68%, Astra’s stays at 48% on the puzzles that Sol did not solve, and together the models solve 83% instead of 86%. Sol also didn’t play those 36 like puzzles it remembered, using undo or restart in 34 of them, just as it did on the others.
Does it matter how close a puzzle is to the older games? Sol solved all 15 reused puzzles we tested it on. Of the puzzles that were never public, it solved 83% of those built on one world’s mechanics, but only 53% of those that combined worlds. The gap holds for short and long puzzles alike. This fits the dense sampling idea below. But puzzles that combine worlds may simply be harder, so we can’t tell the two apart.
How did the models get so good?#
1. Hypothesis: specific training, but on a dense sample of games#
Our best guess for how the models do this is that OpenAI and other frontier labs are densely sampling the space of games and training on these samples. We further guess that labs or their suppliers began doing so at scale around the middle of this year. Argument:
- Games are among the cheapest RL environments to build: you get the simulator and the sparse reward function for free.
- Human-made games recombine a limited stock of mechanics, and generators and wrappers already cover whole genres: Archipelago for Pokémon, PuzzleJAX for tile puzzles, GlyphBench for 2D games.
- The models themselves can now write “new” playable games, so the supply of such new environments is no longer limited.
- So a lab can cover the space of game mechanics densely enough that most new commercial games will land near something they have trained on.
- A model trained this way would improve on many games at once, hold up on scrambled layouts, lose little on a sequel or a fan hack, and stay weak where environments are slow or costly to run.
In 2023, DeepMind’s AdA showed that you can get fast adaptation to unseen 3D game-like environments by training on a dense space of related, generated tasks. It might be that Astra is the result of industrializing that approach on some large fraction of the whole back catalog of human games and generated games. Our last post described shallow generalization at the scale of one game engine; dense sampling might have extended it to the whole medium.
OpenAI’s gaming jump falls between GPT-5.6 Sol (July 9: 8% on ARC-AGI-3, 96 hours for Pokémon FireRed) and Astra (September 3: 63%, or 99% with OpenAI’s provider adapter harness, 18 hours). Gemini 4 Argon (50%) and Claude Fable 5.2 (81%) have now posted ARC-AGI-3 scores in the same range as Astra, and Opus 5.5 wins 93% of base-difficulty Slay the Spire runs to Astra’s 98%. This weakly suggests a shared input: environment vendors making game engines and selling the same products to several labs.
2. Hypothesis: recurrent depth (“looping”)#
Looping may be behind these improvements. Astra is reported to be a “looped transformer,” a transformer that has recurrent layers (information can run through them multiple times before the model’s next token is produced). This seems likely, given its score with reasoning switched off (96.7% on ARC-AGI-3) suggesting an architectural change, and OpenAI appearing to attribute their dismissal of three members of staff to the leaks that informed this report. GPT-6 Sol (which people say is not looped) scored 4.6% on ARC-AGI-3; GPT-6.1 Sol (probably looped3) scored 52.7%. If GPT-6 Sol isn’t looped, then this is good evidence that looping helps.
3. Hypothesis: generalization from other vision-dependent training#
Maybe game skill is a by-product of progress on other vision-dependent capabilities, such as computer use: for instance, Astra’s gains also show up in Blender, CAD, and robot control.
4. Hypothesis: “gains from pure scale”#
Maybe going up a weight class (from 1T to 6T parameters) helps, but GPT-6.1 Sol also jumped in game performance. That being said, Astra’s game skill was probably distilled into Sol, ferrying the gains from the larger model into the smaller one.
Where does that leave us?#
Consider all the games Astra demonstrates impressive performance on: (newish variants of) Pokémon, NetHack, Factorio, Baba Is You, Portal, Montezuma’s Revenge, and the private games of ARC-AGI-3.4 Now add OotSS.
We don’t believe we can yet call these models general game AIs – especially once you account for all the games that require real-time interaction. However, with further caveats (general at 1) single-player games which 2) can be paused, without 3) too much 3D complexity, without 4) too much cross-episode learning, and 5) considering game solving to be a distinct skill from game playing), we’re tempted to say that Astra is in fact a general game-solving intelligence.
Metadata#
- Cost to replicate the above: ~$3,200 at API pricing, or roughly three weeks of a (late-2026) Codex Pro subscription.
- Data here.
- Code here.
Footnotes#
-
The only exception is an Order of the Sinking Star demo released in June 2026. The demo contained 105 of the game’s 920 puzzles (11%), and 35 of the 252 puzzles we evaluated the models on (14%). GPT-6.1 Sol solved all 35. ↩
-
We followed the Center for AI Standards and Innovation’s convention, with the budget tracking the weighted sum of output tokens (100% weight), uncached input tokens (23% weight), and total input tokens (2% weight). ↩
-
Weak evidence: “GPT-6.1 Sol uses the same base model weights as GPT-6 Sol with two inference passes instead of three,” suggesting – assuming that GPT-6 Sol is a typo – that both GPT-6 Astra and GPT-6.1 Sol are looped transformers. ↩
-
And it does fairly well at Slay the Spire on max difficulty. ↩