An agent that completes a task once has shown it can succeed under those conditions. A team deciding whether to delegate that task needs a different answer: how often does the success repeat?
\nIn an IBM Research article published September 15, 2026, Evelyn Duesterwald and colleagues reported a gap of exactly this kind. On 168 AppWorld tasks, a ReAct agent using GPT-4.1 averaged a 77.4% pass rate across five runs. Only 53% of tasks passed in every run. After the authors added consistency guidelines, the average reached 81% and the share of tasks passing every run reached 69%. PSEEDR read the article and its accompanying technical report dated September 8 on November 1, 2026. The report link points to its first posted version; the article page may have changed since that reading. The numbers here are the authors' benchmark results, not a measurement of business workflows and not a PSEEDR replication.
\nThe distinction matters before choosing a remedy. An average can rise because an agent learns to solve more tasks, because it becomes less erratic on tasks it already handles, or both. An operations team needs to know which change occurred. The same aggregate score can hide very different experiences for a person who makes the same request again.
\nThe notation can obscure the question. Mean@5 averages success over the five attempts. Pass@5 credits a task if at least one attempt succeeds. Pass^5 requires all five to succeed. The IBM study reports Pass^5 next to the average rather than treating them as interchangeable.
\nA small hypothetical example makes the difference visible. Suppose an agent receives five tasks, five times each. On four tasks it succeeds in every run, and it always fails the fifth. Its average success rate is 80%, and four of the five tasks count as consistently solved. Now suppose a second agent succeeds four times out of five on every task. It also averages 80%, but no task passes all five attempts.
\nNeither pattern alone decides which agent is suitable. In the first case, the unsolved task may be identifiable and assigned elsewhere. In the second, every task carries intermittent failure. If the operator can recognize a correct answer cheaply and retry without consequences, occasional success may still be useful. If a failed attempt already creates work or an external effect, the at-least-once result, Pass@5, is a poor description of the user's exposure.
\nThe all-runs measure also depends on how many runs are demanded. Five successful trials are stronger evidence than one, but they do not establish an unlimited guarantee. Comparing Pass^3 for one system with Pass^5 for another quietly changes the standard, because the all-runs requirement gets stricter as the number of runs grows.
\nThe ALTK-Evolve project turns recorded agent experience into guidelines that later tasks can retrieve, without changing model weights. The consistency work adds a diagnostic signal: which choices vary when the model sees the same recorded context again?
\nThe design keeps two activities separate. A production agent interacts with tools and records its trajectory. Offline analysis revisits the model's decision points in that trace. It samples possible responses to the recorded prompts, but it does not execute the alternatives against the original applications. Guidance derived from the analysis then enters the memory used by future runs. That design makes diagnosis possible without treating every captured workflow as safe to replay.
\nFor a workflow owner, the benefit of this separation is also its limit. The analysis can flag a choice that deserves attention even when the observed run happened to succeed. But alternative responses to a recorded context are not a complete set of alternative task outcomes. A different choice could have produced tool results and later contexts that the original trace never contained.
\nConsider an illustrative agent that sometimes selects the first search result and sometimes checks for an exact account match. A useful review asks what evidence makes either choice correct, then tests guidance that addresses that decision. Making the agent choose the first result every time would reduce variation while preserving the wrong behavior for ambiguous searches. A measure of how much a decision varies says nothing about whether the stable choice is the right one. Repeatability needs an outcome criterion alongside it.
\nThe technical report specifies 30 resamples per inference step at temperature 0.5 for its experiment. The article describes a five-completion default for the released diagnostic tool. These are different settings, and the published improvement should not be presented as evidence that the cheaper default reproduces it.
\nThe report estimates roughly 30 times the original trajectory's token cost for that offline analysis. The pipeline it describes admits every generated guideline directly into memory, and the authors leave a validation stage as future work. Those details change the deployment question. A team must decide how often it analyzes traces, which kinds of task justify the expense, and how it tests a new guideline before relying on it.
\nThere is a further limit on transfer. In the similar-task experiment, guidelines came from another variant within the same AppWorld scenario, not from an unrelated workflow. Aggregate results improved, but GPT-4.1's mean accuracy on the hard subset fell by 0.8 percentage points. The article's wording that mean accuracy never dropped describes the same-task results. That is narrower evidence than a claim that guidance improves every workflow without trade-offs. The setup, transfer results and limitations are in the report's sections 4 and 6.
\nA sensible trial starts with tasks whose outcomes the team can judge and whose test environment can be reset. Keep the task conditions stable while repeating runs; otherwise a changed database or permission can look like agent inconsistency. Retain the individual outcomes so an improved average does not hide a new pocket of intermittent failure.
\nNext, separate diagnosis from adoption. A variable decision point is a reason to investigate. A generated instruction is a proposed change. Adoption should depend on what fresh executions do with that change, including related tasks where a formerly helpful rule might no longer apply. This is an operational reading of the evidence, not an additional result reported by IBM.
\nThe goal is not to eliminate every difference in wording or every alternative route through a workflow. Several paths may produce an acceptable outcome. The useful question is whether the variation changes completion, correctness or the amount of human intervention required. A stable but wrong agent and a flexible agent that reliably succeeds should not receive the same verdict.
\nThe study makes one practical addition to an evaluation report: show how many tasks succeed repeatedly next to the average pass rate, with the repetition count and conditions attached. Its guideline method is one candidate intervention, with a real analysis cost and a bounded test base. Before delegating recurring work, a team should know whether its reassuring average describes dependable tasks or a collection of successes that may not recur.
\n" }