The Missing Budget Behind a Benchmark Score

Two reports name the same model and benchmark, yet show different scores. Before treating that gap as a contradiction, check whether the model had the same opportunity to solve the task in each run. One run may stop after a single answer. Another may allow repeated submissions and tell the model when it is wrong.

The September 22, 2026 announcement from EvalEval and the UK AI Security Institute describes a practical way to expose such differences. AISI is sharing results, context and configuration through Evaluation Cards, using the Every Eval Ever reporting schema. The release covers the five benchmarks from the main experiment in AISI's accompanying paper, plus two related cyber evaluations that use a partly different set of models.

The useful change is what a result is attached to. A score belongs to a model, a task set and an execution protocol together, because the protocol is part of what was measured. A common reporting format makes that relationship inspectable. Readers still have to decide whether two runs answer the same question. PSEEDR read the announcement and supporting materials on November 1, 2026. This is a retrospective analysis, not a claim of independent reproduction.

Feedback changes the search

The AISI technical report, by Jessica McFadyen and colleagues, evaluates six models on five main benchmarks under two feedback conditions, with five runs, or trajectories, per task in each condition. The shared protocol combines larger token budgets, context compaction and repeated submissions. In one condition, each submission receives an acknowledgement with no information about correctness. In the other, the model learns whether its submission was correct and stops at the first full success. Each trajectory has a budget of 5 million to 30 million target-model tokens, depending on the benchmark, counting input, output and reasoning but excluding the tokens used by any LLM judge.

Those two conditions differ by more than the length of an answer. Consider a hypothetical puzzle solver that proposes three solutions. If a checker rejects the first two and accepts the third, the solver learns which paths failed and when to stop. Without that checker, producing three candidates does not tell the user which one is correct. The application needs its own way to select an answer.

This is why a result obtained with correctness feedback should not silently become the expected success rate of a product that lacks such feedback. The experiment still says something useful about what the model can achieve when a checker exists. But it describes a different system: a model working with a checker, not a model answering on its own.

A token limit is not a complete protocol

The report finds that the return on extra tokens depends on the benchmark. Extending SWE-Bench Pro from the upper end of its typical published budget, about 16 million total tokens, to 30 million added about 0.3 percentage points on average across models and conditions. Humanity's Last Exam (HLE) showed more headroom, particularly when correctness feedback was available. The authors used one shared scaffold and task subsets, so their absolute scores are not directly interchangeable with published leaderboard results.

An evaluator therefore needs more than a large number in a max_tokens field. Is the number a per-call output limit or the total allowance for a whole trajectory? Does it count context that is resent on every call? Are multiple attempts independent, or can a later attempt build on earlier work? What ends the run: a budget, a timeout, a successful check, or a rule against repeated answers?

These questions explain why matching a headline token count can still leave a comparison unequal. An agent that repeatedly interacts with an environment spends its allowance differently from one that produces several independent answers. A model with access to a useful test can spend tokens responding to evidence that another run never receives. The same budget, allocated differently, measures a different thing.

The right protocol depends on the reader's decision. A developer choosing a response service under a latency budget needs a test close to that operating constraint. An evaluator asking whether a well-resourced system can eventually solve a hard task may deliberately allow much more work. Neither result should be relabeled as the other.

Keep the record attached to the score

The Every Eval Ever repository defines structured aggregate records and optional companion files for individual samples. Its current documentation describes fields for generation arguments, available tools, evaluation limits and sandbox configuration. Sample records can retain answers, extraction details, scores, token usage and tool-call traces. Converters support existing evaluation logs from Inspect AI, HELM and lm-evaluation-harness.

This gives teams a practical place to record the conditions that would otherwise disappear when a run becomes a chart. The repository also validates records against the schema and checks that aggregate and sample files agree with each other. Those checks can catch malformed or mismatched records. They do not establish that the original experiment measured the intended capability. These are the facilities documented today, not a claim that every converter or field existed in the September release.

For a developer, the first benefit may be less ambitious than full reproduction. A retained record can explain why a score changed after a harness update, identify a missing sample, or distinguish a new judge from a new model. That is already more useful than a number copied into a slide with its settings left behind.

The same record also makes a reproduction attempt more precise. Another team can identify which configuration it matched and where it departed. If the original model endpoint or environment is no longer available, a structured record preserves that limitation instead of letting an approximate rerun pass as identical.

Verified provenance is a narrower claim

The announcement describes the released results as verified. On the Evaluation Cards platform, the verification guide says that results earn the checkmark when they are submitted through a Hugging Face account associated with the reporting organization's own Hugging Face organization. The checkmark signals that the numbers come from that source. The guide does not define it as a badge for independent replication.

That distinction is useful rather than disappointing. Knowing who supplied a result answers a provenance question. Having a schema-valid record answers a formatting and consistency question. Reproducing the run, and judging whether it represents the intended use, answer further questions. Collapsing these stages into a single notion of trust makes the badge carry more than its stated meaning.

Before using a score difference to choose a model, locate the task set, budget, feedback and stopping conditions behind each number. Where a material detail is absent, keep that uncertainty visible instead of assuming the other report's default. A missing configuration is a reason to narrow the comparison, not proof that the result is wrong.

The contribution of shared evaluation records is that they make this examination possible at a useful scale. The score remains important. Its meaning becomes clearer when the reader can see what the system was allowed to do to earn it.