A convergence bound in a published theorem held at every iteration up to 223 and first failed at iteration 224. The condition the proof depended on had already been lost after the first update.
\nThe theorem appears in Attention's forward pass and Frank-Wolfe. In the counterexample, three points started with the origin inside the triangle they enclose. One update moved the points so that their new enclosing shape no longer contained the origin. The proof needed that containment to hold at every step. The theorem's stated assumption required it only at the start. A short run could therefore miss the eventual violation even though the missing condition was visible after one step.
\nThe case appears in Hugging Face's August 13 retrospective on its ICML 2026 reproduction challenge. The post is also the source of the headline numbers: 2,226 papers attempted and 6,816 logbooks published. Those numbers count attempts and records. They do not mean thousands of papers were established as correct, and the retrospective itself reports falsified claims, reduced-scale evidence and inconclusive results alongside the verified ones.
\nFor a developer using a coding agent to check research, the practical question is how to translate a paper's claim into an experiment that could contradict it. More runs help only after that translation is right.
\nThe paper studies a simplified limit of self-attention in which token embeddings are treated as moving points. In the case the paper calls negative-definite, Section 3 rewrites each attention update as a Frank-Wolfe optimization step, a method that repeatedly moves a point toward the corner of a convex region selected by a linear rule. Theorem 3.1 in the version examined by the reproduction states that a quadratic quantity for every point stays below a bound that decreases toward zero. Its conditions are a pair of matrix requirements, a fixed step-size schedule, and the requirement that the origin lies in the initial convex hull.
\nA convex hull is the smallest convex shape that encloses a set of points. For three points not on a line, it is a filled triangle. Under the paper's update, each new point is a mix of its old position and a point of the current hull, so each new hull sits inside the previous one. Shrinking is not the same as keeping a particular interior point. A smaller triangle inside the original can leave the origin outside.
\nThat is what happens in the published exact counterexample. The three starting points are chosen so the origin lies strictly inside their triangle, which satisfies the stated assumption. The first step size in the schedule equals one, so each point jumps all the way to the point its rule selects. Two points land on one corner of the triangle and the third on another, so the new hull is a line segment, and the origin is not on it. From then on the points converge to a nonzero location. The theorem's bound keeps falling toward zero, and the tracked quantity first exceeds it at iteration 224.
\nThe reproduction therefore contributes more than a curve that looks different. It names the step in the proof that uses origin membership at every iteration, where the theorem only assumed it at the start. The certificate scopes its verdict to Theorem 3.1 as stated in that version. It also records that the algebraic identity between the hardmax update and a Frank-Wolfe step still holds, and that convergence to the origin would need the stronger, invariant condition. A gap in one theorem of one version is not evidence that the paper's other results are wrong.
\nThis is PSEEDR's reading of the published proof and certificate, not a new reproduction. The public record states its calculations in exact rational arithmetic, so there is no floating-point rounding to argue about. The organizers report that three independent teams found counterexamples, with first violations at 224, roughly 3,800 and 6,416 steps, and that the authors confirmed the finding the same day. Those statements belong to the organizers. PSEEDR has not checked the other two logbooks or the paper's current status.
\nThe delayed failure shows why a test horizon belongs in the result, not only in a configuration file. A run that stopped at iteration 200 could correctly report no observed violation. It could not establish a statement meant to hold at every later iteration. The organizers say this is why other participants marked the same claim as verified: finite-horizon checks stopped too early. They describe the same pattern on a second paper, where several verified verdicts came from checks that stopped before a slow logarithmic growth became visible.
\nThe same distinction applies to experiments with model size, dataset scale or rare events. A small successful case can support the implementation's basic behavior while leaving the paper's scaling claim untouched. Running the program and testing the full proposition are different accomplishments.
\nA useful reproduction brief should spell out the assertion, its assumptions and the region actually tested. For the attention example, checking at each step whether the origin is still inside the hull is more informative than plotting the final objective alone. Watching that condition reveals the logical break at step one, long before the headline inequality fails at step 224.
\nThis does not require unbounded computation. It requires that the conclusion fit the finite evidence. An empirical check can report the range it covered. A mathematical counterexample can disprove a universal claim if it satisfies the stated assumptions and violates the conclusion. A proof needs an argument that covers the remaining cases. These three outcomes should not share one success label.
\nThe challenge made part of that distinction explicit. Its automated judge, running an open-weights model, gave each claim one of four verdicts: verified, falsified, toy (evidence at reduced scale) or inconclusive. The judge was instructed to treat each logbook's self-assessment as untrusted. Reduced-scale evidence and missing artifacts therefore stayed visible instead of being forced into pass or fail. The judge's output is still an assessment to inspect. The categories do not make a flawed experiment reliable.
\nThe retrospective also describes a falsification that did not survive. One logbook claimed the paper's method ran twice as slow as its baseline. The organizers found that the comparison set time per trajectory on one side against time per batch of fifty trajectories on the other. Once the units were normalized, the participant's own measurements supported the paper's reported eightfold speedup. The retrospective does not name the paper or link that logbook, so this account rests on the organizers' description.
\nThis failure differs from the attention case. There, the test eventually exposed a missing condition in a theorem. Here, the reproduction's own comparison was invalid. An agent asked to find a mistake in a paper can produce either kind of report. The organizers say 35 participants submitted formal falsification claims, and that they re-checked every claimed falsification by rereading the paper and logbook and rederiving the math or reimplementing the experiment. An adjudicator has to inspect both the original claim and the attempted refutation.
\nFor performance claims, that begins with the denominator. A time measurement needs to say how much work it covers: one sample, one batch, one complete run or some other unit. Both sides of a speed comparison need compatible units and conditions before the ratio means anything. More precise timing does not repair a mismatch in what was timed.
\nFor theoretical claims, the matching check is whether the candidate counterexample obeys the stated assumptions. The exact certificate lists each one (the matrix conditions, the step-size schedule and the origin inside the initial hull) and records that the counterexample satisfies it. Violating an assumption may mark an interesting boundary, but it does not refute a theorem that excludes that case.
\nThe logbook format helped make these checks possible. Every run published a write-up, the code that ran and the artifacts it produced, with the full agent trace optional. That makes a claim inspectable. It does not make it correct. The units error is a reason to keep the record available: the same data that first supported a dramatic conclusion let the organizers reverse it.
\nThe lesson for an agent-assisted research workflow is more specific than adding a second model as reviewer. Give the agent a claim whose meaning survives translation into a test, and preserve the conditions that distinguish a reproduction from a different experiment.
\nA compact result should let a reader locate the paper version, the claim, the assumptions tested, the range the run covered and the evidence behind the verdict. That is a proposed working format from PSEEDR, not a requirement the challenge imposed. It lets a human direct the next experiment at the uncertainty that remains.
\nIf the apparent failure comes from incompatible units, normalize the comparison before spending more compute. If a proof depends on an invariant, check whether the update preserves it. If a claim concerns large-scale behavior, report the smaller experiment as smaller evidence and choose the next scale deliberately. If the required data or checkpoint is unavailable, keep the inconclusive result instead of manufacturing a substitute verdict.
\nThe retrospective describes a community event with varied agents, compute budgets and methods, and it is written by the organizers of that event. Its aggregate figures are not a representative failure rate for ICML papers, nor a controlled comparison of coding agents. The individual records are more useful for this decision than a ranking would be: they show what let a finding survive adversarial re-checking and what made another collapse.
\nAn agent can run many experiments. The researcher still decides which proposition each experiment can settle. In the attention case, that meant tracking a condition lost after one update rather than trusting a bound that happened to hold for the next 223 iterations. The useful output was the explanation connecting those two events.
\n" }