Everyone has assumed that when you hand an AI agent a note from an earlier task, you are teaching it something it did not know. A team has now gone and measured that. The share of "something it did not know" in the successes is 4.5%. The share of something else is 65.7%.
That something else is the note stopping the agent from wandering off. It adds no knowledge; it holds the path.
This paper asks precisely the question people usually walk past. Everyone has measured whether notes raise the success rate, but few have asked when they help, why they work, and where they break.
how they measured it
Eight thousand one hundred and thirty-five recorded runs, across three benchmarks, with two agent harnesses and two different models. The tasks are real command-line work, meaning things that either run or do not, with nobody needing to offer an opinion. Then, to find out what was actually happening, they went through 528 matched triples one by one: the same task once with a note, once without, and once with the raw trace of the earlier attempt.
There is a methodological detail worth mentioning. The labelling of these runs was done by a model, but they compared the result against independent human coding and got 95.8% exact agreement. Whenever somebody claims a taxonomy, that number is the first thing to ask for.
a note is a rail, not a library
The main result is the paper's opening distinction.
When a note works, it almost always works because it holds the agent to a fixed procedure. The agent already knew what to do, but it would drift mid-task, skip a step, or stop early. The note acts as a rail and keeps it from leaving the track. That mechanism accounts for 65.7%.
The thing everybody assumed was happening, that the note supplies knowledge the agent lacked, accounts for 4.5%.
That difference is not a theoretical nicety; it lands directly on design. If a note is a library, then more is better and more complete is better. If it is a rail, a long and detailed note actively hurts, because a rail has to be short and clear.
And that is exactly what they saw. When they gave the agent the raw trace of the earlier attempt instead of a distilled note, the success rate fell from 61.9% to 55.9%. Worse, agents given the raw trace ran out of time in 10.6% of cases, against 4.4% with the distilled note. A raw trace is full of fruitless searching and low-level debugging, and the agent carries all of it along.
follow the numbers
Now the most interesting part, which I think is the single most important thing in the paper and did not make the abstract.
The question was simple: what happens when the pool of notes gets large. They grew the pool from 5 to 100 and measured retrieval precision, meaning how often the agent picks up the note that is genuinely relevant to this task.
Precision fell from 29.6% to 3.3%. Roughly nine times worse.
Now guess what happened to the success rate. It went from 36.4% to 39.3%. It went up.
Stop on that for a moment. The retrieval layer got nine times worse and the outcome did not move at all. The only thing that can mean is that the note being retrieved was never doing the work we credited it with.
A third number completes the picture. Recall stayed between 54% and 74%, so the right note usually was among the things the agent pulled; it just did not restrict itself to it. It grabbed several candidates, glanced at them, and got on with the job.
a prediction that held
Two weeks ago I wrote here that if you store a note without knowing it moved the agent from failure to success, what you have built is not a notebook but a warehouse. At the time that was an argument. Now it has a number.
A pool built only from successful runs gave a success rate of 0.75. The same pool built only from failed runs gave 0.52.
But the more telling number is elsewhere. When failed runs are in the pool and you keep the label saying which was which, the result is 0.75. The same pool without labels: 0.40.
So merely knowing which experience worked nearly doubles the outcome. The data is identical. The only difference is that you know which is which.
and now the "until they don't"
The paper's title has two halves and the second one matters more.
Notes create a new class of failure that did not exist before. In 10% of the cases where the agent had a note, the note itself was misapplied or ignored. For an agent with no note at all that figure is 0.8%, since naturally there was nothing to misapply.
The shape of it is predictable. The agent runs the procedure mechanically, does not see the conditions under which the procedure is valid, and carries assumptions over from the earlier task that no longer hold. The same rail that stopped it drifting now holds it firmly on the wrong track.
what gets fixed and what does not
There is a clean split here that is useful to anyone building systems.
Notes almost entirely wipe out execution hygiene errors: environment and infrastructure failures such as port conflicts and missing dependencies dropped from 5.3% to 0.2%, and failure to respect the output format from 7.4% to 3.2%.
But two classes of error did not move at all. Logic errors in the algorithm itself stayed between 8% and 11%, in all three conditions. And the agent verifying its work without actually running it and declaring itself done stayed around 12%, again in all three conditions.
The summary is that a note tidies up the walking but does not fix the thinking, and if the agent has the wrong algorithm in mind no procedure will save it. That split is decision-useful: if your system's problem is disorder in execution, a note is the answer, and if it is a mistake in the solution itself, do not waste your time.
now the limitations
The paper limits itself in several places and those need reading.
The mechanism taxonomy comes from a sample of roughly three percent of the records rather than labelling everything, so rare behaviours are probably under-represented. The benchmarks are all terminal and tool work and do not cover long-horizon web tasks or open-ended collaboration, and the number of model and harness combinations tested is small.
And one detail they flag themselves: the pool-size experiment ran on a different model version, so its numbers should not be compared with those in other sections. Only comparisons within that section are valid.
what engineers should take from it
Two things.
First, design the note as a rail rather than an encyclopedia entry: short, conditional, with the conditions under which it is valid written on it. A note that does not say when it applies and when it does not is exactly what produces that new 10% failure class.
Second, and more important, do not look at the success rate on its own. In this very paper retrieval precision got nine times worse and the success rate showed nothing at all. If you were monitoring only the success rate, you would have been looking straight at a thoroughly broken layer for months without seeing it. In the previous piece I wrote that for any cache we have to measure what fraction of lookups actually hit; this paper measured that number, and the answer was close to none.
I have seen the same in our own work. In BLZN.AI, which carries a trading strategy through to live execution, the dangerous case was never a bad result. It was a good result for a reason other than the one we thought. Until you can connect an improvement to the specific component you believe is responsible, a good number is just closing your eyes for you.
so is the notebook dead
No, and this is where this paper goes further than the last one.
The earlier benchmark showed that most of the improvement we were crediting to the notebook came from having recently seen a few similar tasks. This paper says why: what the notebook was doing was procedural reminding all along, and a few recent tasks do that same reminding for free.
So the notebook is not useless. It is simply something other than what we named it. It is not a skill library. It is a rail, and rails are built short and few and precise rather than large and complete. The people working on compressing these pools are, in effect, conceding the same point.
And if one sentence survives from this paper, it is that a note teaches the agent nothing. It only stops it forgetting what it already knew.