ARC-AGI-3 launched on 26 March this year, a benchmark built to measure how far we still are from AGI.
The best model in the world scored 0.37% on it. The next ones managed 0.26 and 0.25, and one scored zero. Ordinary people solved all 135 of its environments. There was $2 million on the table for any AI that could match an untrained human.
Five months later, a model scored 99.9%.
François Chollet, who built the benchmark, had said at launch that saturation was about a year away. Asked whether he still considered around 2030 likely for AGI, he said sooner.
That is the news. Now for the number almost nobody printed: the same model, in the same report, also scored 62.7%.
what the benchmark is
ARC-AGI-3 is a set of small games that arrive with no instructions. You drop a model into one and it has to explore, guess at the rules, work out what the goal even is, and then win.
What separates it from other benchmarks is that it does not only ask whether you solved it. It counts how many moves you took, and it counts harshly: if a human needs ten actions and the model needs a hundred, the model does not get ten percent, it gets one.
That is why the March scores were so low. Models knew how to grind forward by brute force, and this benchmark penalises exactly that.
ARC Prize reported on 3 September what OpenAI's GPT-6 Astra did on it.
what is actually new
A near-100 score is not that surprising on its own, because every benchmark saturates eventually.
What is new is the move count. Astra reached the answer in fewer moves than people on 96% of levels, and on average used half as many moves as they did.
That number only means something once you know who it was measured against. About 500 members of the general public were tested, with no filtering for who happens to be good at puzzles. Then for each level they took the typical number of moves among the players who finished it.
So Astra is more efficient than the median ordinary person, not than the best person available. Even that was not what the ARC team expected. They thought this one would stay the dividing line between people and machines for a while, that a model might eventually solve a problem but would need far more searching to get there. They were wrong.
the model invents its own notation
The most interesting part of the report is not the scores. It is what they saw while the model was thinking.
For each game, Astra improvises a shorthand specific to that game so it can keep track of where it is. A line like Turn 5: P=(24,20), empty, facing west, packing the turn number, where it stands, whether its hands are empty and which way it faces, all into one line.
This is not a programming language. It is an improvised notation invented on the spot for one game. ARC Prize says they have seen models do this before, but that Astra's were more precise and said more in less space.
In a separate setup where the model was also allowed to run code, it wrote itself a handful of tools per game, with names like maze_solver.py and patrol_solver.py.
now the second number
Between the model and the game sits a program that passes messages back and forth. ARC Prize tried two different ones.
With their own, Astra scored 62.7 and cost about $26,000 to run.
With the second, designed by the model's own maker, the same model scored 99.9 and cost about $19,000.
That difference is not in the model. It is in the second one, which lets the model keep its own train of thought between requests, and summarise for itself once the conversation gets long.
The second one did not just raise the score. It was cheaper, close to four times faster, and across the 167 games both of them solved it burned half the tokens.
When one thing is better on every axis, it usually means the other was holding the model back, rather than that this one found a way around. But whichever you report, the number changes.
the number nobody mentioned
Both were run at six levels of effort, from the model thinking hard down to not thinking at all.
With the first, the more the model thought the higher it scored, from 17.5 up to 62.7.
With the second, all six levels landed between 96.7 and 99.9.
So with the right plumbing, how much the model thinks stops mattering almost entirely. All that spread you would have put down to "how well the model reasons" was mostly paying for a bad setup.
There is a smaller oddity in the same table. Under the first, the model scored 35.2 when it did not think at all and 17.5 when it thought a little. Thinking a little was worse than not thinking.
two other evaluators said the opposite
So far this is one benchmark. Put the others beside it.
Epoch AI put the same model first, at 169 points.
Artificial Analysis gave the same model 61, exactly level with its own predecessor and below Claude Fable 5.1 at 66. On several measures in that suite, Astra came out behind the previous generation.
So in one week, one model was scored four ways: on a benchmark, on the same benchmark through different plumbing, top of one leaderboard, mid-table on another.
None of the four is a lie. What changes between them is the ruler.
so is it AGI or not
ARC Prize says flatly that they are not claiming it is.
They give the reason. ARC-AGI-3 is a small, closed world with known rules and a defined goal, and it looks nothing like the real one, which is open and messy. The benchmark measures one corner of what we want from AGI, not all of it.
But Chollet's shift should not be waved away either. The man who built this benchmark, and who has been the most cautious voice in this argument for years, now says progress came twice as fast as he expected and is moving his date forward. That matters more than any score.
The next version, ARC-AGI-4, is due in the first quarter of 2027.
what engineers should take from it
A number without its conditions is not a number.
"99.9%" means nothing until you say through which plumbing, on which games, at what level of thinking and at what cost. The same model becomes 62.7 when one of those changes.
Last time I wrote that a success rate without a baseline can mislead you. This is a step further. There you had to ask "compared to what". Here you have to ask "under what conditions".
For anyone building systems this is good news, because it means much of that gap is in your own hands. Keeping memory between calls, laying out context properly, summarising at the right moment: here those moved the score from 62.7 to 99.9.
The same holds in our own work. In BLZN.AI, which carries a trading strategy through to live execution, no number out of a backtest is worth anything until its conditions sit next to it. Change the fees, the slippage and the date range, and the same strategy goes from profitable to loss-making.
what has changed
Five months ago, front-rank models scored under one percent on this benchmark and people scored 100. Now there is a model that works in fewer moves than the median ordinary person on the same benchmark, and invents its own notation and tools on the way.
But what has been shown is not that machines have caught up with people. What has been shown is that in a small, rule-bound world, wired up properly, a machine works out the rules faster than an ordinary person does.
The distance between those two sentences is as wide as the distance from 62.7 to 99.9.