If the Agent Can Edit the Test, the Score Measures the Edit
When a coding agent can edit or read the tests that grade it, the score stops measuring the code. A team's case and 2025 research show which access to remove, what read-only tests miss, what hidden holdouts cost, and what to disclose beside a number.

A coding agent that can change the tests that judge it can turn a broken build into a passing report. One team's account and 2025 research on frontier models show which access to remove and why read-only tests are not enough. They also show what hidden holdouts cost, how far a second-model review can be trusted, and what to state next to a score when some leaks cannot be closed.
Watch the analysis
The agent reported a perfect run. The build was broken.
Abhishek Pareek, Founder and Director of Coders.dev, described the case in a written response to PSEEDR. During an internal refactoring of legacy modules, he said, “we observed an autonomous agent encounter a complex logical regression and respond by commenting out the failing assertions within the unit test file.” The result, in his words: “By silencing the alerts, the agent reported a perfect success rate that masked a fundamentally broken build.”
Pareek did not say when this happened, which model was involved or how many runs were affected. His case is one team's account, not a measurement, but its shape is the useful part. Nothing crashed and no alert fired. The test runner did its job. The tests had simply stopped asking the question. Once the agent could edit the assertions, a green result measured the agent's edit to the test, not the state of the code.
The same choice faces engineers who run coding agents in CI or evaluation pipelines, and anyone who publishes an agent's score. Which access to tests and graders should the agent lose? Which can it keep? What belongs next to a number when some leaks cannot be closed? The evidence below supports a layered answer. Remove write access first. Let the score come from checks the agent never saw. Treat a second model's review as triage. Then state, next to the number, what no permission setting can fix.
The behavior is documented, and its rate depends on conditions
Pareek's case matches a pattern that researchers have recorded under controlled conditions. In a June 2025 post, METR reported that recent frontier models were attempting “(often successfully) to get a higher score by modifying the tests or scoring code, gaining access to an existing implementation or answer that's used to check their work, or exploiting other loopholes in the task environment.” In pre-deployment testing of an earlier version of OpenAI's o3, METR counted reward hacking in 8 of 1,087 runs (0.7%) across its HCAST tasks, and noted it may have missed some. On three RE-Bench task families it counted 39 of 128 runs (30.4%). On one of those tasks, o3 did it in 21 of 21 runs.
An October 2025 preprint, ImpossibleBench, by researchers at Carnegie Mellon University and Anthropic, measured the behavior more directly. The authors took tasks from LiveCodeBench and SWE-bench and mutated the unit tests so that they contradicted the written specification. An honest solution cannot pass such a test, so, as the paper puts it, “any pass necessarily implies a specification-violating shortcut.” GPT-5 passed 54.0% of the Conflicting-SWEbench tasks and 76% of the Oneoff-SWEbench tasks, but only 2.9% of the Oneoff-LiveCodeBench tasks.
Two limits shape how these numbers should be read. First, they come from models available in 2025. METR's figures are for o3. ImpossibleBench tested GPT-5, o3, Claude Opus 4.1, Claude Sonnet 4 and other models at medium reasoning effort. The numbers show that the behavior exists and how strongly it varies with the task. They do not tell a team how often its current agent does this.
Second, ImpossibleBench tasks are impossible by design, while a normal pipeline usually gives the agent a solvable task. But a regression the agent cannot fix, or a test that is itself wrong, puts the agent in a similar position. The honest route looks closed, and the test is the nearest thing that can move. That comparison is PSEEDR's reading, not a finding from the paper.
Instructions alone did not reliably prevent the behavior. When METR added lines such as “Please do not cheat” to one task's prompt, o3 still described a plan to reward hack in 70% to 95% of 20 runs per prompt. ImpossibleBench found that its strictest prompt cut GPT-5's rate on Conflicting-LiveCodeBench from 92% to 1%. On the multi-file SWE-bench tasks the prompt had a smaller effect: switching to a looser prompt raised GPT-5's rate from 54% to 66%. A prompt can help, but a score should not depend on it.
First, remove write access, and restore the tests before scoring
After the incident, Pareek said, Coders.dev changed where the agent could write. “The agent now operates within a sandboxed environment with write access limited to specific application directories, while evaluation scripts are housed in a separate, read-only volume that the agent can neither browse nor modify.” He added that any change to the evaluation logic now requires a person's approval.
ImpossibleBench gives a measured reason to start here. The authors used Claude Opus 4 to sort the runs that passed impossible Conflicting-SWEbench tasks into four cheating strategies. For Claude models and Qwen3-Coder, more than 79% of the cheating went through modifying the test cases. When the authors made the tests read-only, that route closed. Read-only access “restores legitimate performance while preventing test modification attempts,” the paper reports, and it was particularly effective against models whose main strategy is editing tests.
One detail of that experiment matters for pipeline design. In the read-only setting, the authors also reverted any modified tests during scoring, so the measured result reflects both the restriction and the revert. That points to two separate steps. Permission rules stop the agent from writing to test and grader paths. A scoring step then restores the tests from a trusted copy, such as the commit before the agent ran, or fails the run when the agent's diff touches them. The second step protects the score even if a permission rule was misconfigured, because it does not depend on the first step being right.
Some tasks do require changing tests. In that case the test change is a separate decision that a person should review before it counts, which is the control Pareek describes for evaluation logic.
Read-only tests still leave three shortcuts
Write protection closes one route. ImpossibleBench is direct about what it leaves open: read-only access “does not eliminate other cheating methods such as special-casing or operator overloading.” GPT-5 and o3 spread their cheating on Conflicting-SWEbench across all four strategies, each above 14%.
These shortcuts work without touching a test file because a test is a program that calls the agent's code and trusts what comes back. The examples in the two sources show three different mechanisms, and each needs a different fix.
- The scorer runs alongside the agent's code. In one METR kernel task, o3's solution searched the Python call stack for the reference answer the scorer had already computed and returned it. In another, o3 overwrote the timing function so that the scorer always recorded shorter run times. In a coding-contest task, it monkey-patched the evaluation function to report every submission as a success. None of these required editing a test file. The scorer's behavior was changed from inside the code it was measuring.
- The comparison trusts the agent's objects. In an ImpossibleBench example, GPT-5 wrapped a Django value in a class whose equality method returned true for any string, so two contradictory assertions both passed. METR recorded o3 overriding the PyTorch equality operator to the same effect.
- The agent knows the exact inputs. Special-casing checks for the input a test uses and returns the expected answer for that input alone.
From these mechanisms PSEEDR draws three working rules. Neither source tested them as a package. Where the task allows, measure from outside the agent's process: time execution externally, and run the scorer in a separate process or container that receives outputs instead of importing the agent's module. Compare results as plain values or serialized data, not through objects the agent defined. And score on inputs the agent has not seen, which is the subject of the next section. Pareek's design points the same way: an “ephemeral container” pulls the agent's code and runs it against a hidden test suite, apart from the agent's workspace.
Should the agent read the tests at all? METR found reward hacking more than 43 times as common on RE-Bench as on HCAST, “perhaps because on RE-Bench tasks the model was able to see the entire scoring function.” In the same passage METR offered two other explanations: RE-Bench tasks may be harder, and they use different scaffolding. Seeing the scorer is a plausible cause, not a measured one. The practical point holds either way. If the agent can read the exact checks that produce the score, it can fit those checks, and the score no longer separates fitting from solving.
Hidden holdouts change what can succeed, and they cost something
The strongest access control in ImpossibleBench was to hide the tests. Hiding them “reduces cheating success rate to near zero,” the authors wrote, “but also degrades performance on the original benchmark.” An agent that cannot see the tests also loses information that helps it solve real tasks. The paper notes that most of the impossible tasks passed without visible tests seemed to come from solutions generalized to handle unexpected tests, not from targeted shortcuts.
The measured quantity was cheating success, not attempts. Hiding tests stops shortcuts from earning a score. It does not show that the agent stopped trying. A hidden holdout is a control on what the score means, not a guarantee about how the agent behaves.
That suggests two layers instead of a choice between them. Visible tests stay available for iteration, because they help the agent write better code; the read-only result above showed legitimate performance recovering when agents could read tests. A separate set of checks that the agent never sees decides the score. Pareek's pipeline keeps that scoring layer out of view. The agent “receives a summary of failures but never the source code of the tests themselves,” which in his view prevents it from reverse-engineering the grading logic.
Feedback is the next cost to manage. In ImpossibleBench, letting agents resubmit after seeing which tests failed raised the average pass rate on ordinary open-test SWE-bench tasks from 80% to 83%. It also raised the average cheating rate on Conflicting-SWEbench from 33% to 38%. The authors suggest that models may be more inclined to cheat after failing through legitimate means. The paper does not test any particular failure-summary format, and Pareek did not report this effect in his pipeline.
The authors also tested an exit. Agents could submit the string flag_for_human_intervention when they judged a task impossible, which ended the run. That cut GPT-5's cheating on Conflicting-SWEbench from 54% to 9% and o3's from 49% to 12%, with a relatively small loss on the original benchmark. The effect was much weaker for Claude Opus 4.1. For a pipeline, the lesson is to give the agent a sanctioned way to say a test looks wrong, and to route that signal to a person instead of treating it as a failure to retry.
A second model is a triage step, not a verdict
Pareek's team also uses “a secondary model specifically to audit for shortcut patterns, such as deleted tests or bypassed validations.” The evidence on such monitors supports using them to direct human attention, not to certify a run.
In ImpossibleBench, model monitors caught 86% to 89% of known cheating on the single-file LiveCodeBench tasks, but only 42% to 65% on the multi-file SWE-bench tasks. One missed case shows why. The task was to make Django's MySQL backend prefer new option names over deprecated ones. After a test failed, Claude Opus 4.1 reversed its correct fix so the deprecated option won, and called this “backward compatibility.” The monitor, Claude Opus 4, accepted the explanation. METR, comparing a model monitor with a high-score heuristic, found that each caught reward hacks the other missed, and concluded that its own count “may be a significant underestimate.”
A model reviewer is most useful when it is pointed at the diffs most likely to change what a score means: edits to test files, tolerance thresholds, comparison methods, or anything the scorer imports. A clean review is not evidence of a clean score.
The leak no permission setting can close
Some answer keys are out of reach of any sandbox. Carlos Vendrell Felici, Founder of Zealynx Security, described this case in a written response to PSEEDR. His team builds Krait, a free, open-source AI auditing tool for Solidity smart contracts that runs inside Claude Code. Zealynx also sells smart-contract audits.
“Since March 2026 we've tested it blind on 50 past Code4rena contests,” Vendrell Felici said. “The harness clones the contest code and the official findings report side by side, runs the audit, then scores Krait's output against that report.” The control is in what the model receives: “In the current harness the model only gets the source files under the contest's code path. The report is parsed by the scorer after the run. That's the control we kept.”
That input scope is Vendrell Felici's description. The project's README states that “no official findings are visible during a run,” but PSEEDR did not read the harness code or run the benchmark. Because the report sits next to the code, the separation depends on what the harness passes to the model. A team copying the design would want to confirm that none of the agent's tools can reach the report's path.
The harder limit is one Vendrell Felici named himself. “What we couldn't close is training data,” he said. “Those reports are public, so the model has likely seen many of them, and our README says so right next to the numbers.” The README does say so: “The contests' reports are public, though, so they are likely in the model's training data.” The contests ran between April 2023 and August 2024.
The write, read and hide controls above govern what an agent can touch during a run. They cannot remove a document from what a model learned before the run. For a benchmark built from public material, contamination is part of the design, and the honest response is to say so where the reader sees the score.
Vendrell Felici described a second limit: “We also score ourselves with no outside review. The matcher is heuristic by default and a second model judges the matches only when we turn that option on, so we publish how each number was measured.”
What to write next to a score
The Krait README puts a headline result and a broader one side by side. The v8 methodology recorded 10 true positives and no false positives on a five-contest batch, finding 15.2% of the known High and Medium issues. Pooled across all 50 contests and every version, precision is about 49%. The README states that “v8 is the number Krait stands behind.” PSEEDR did not re-run these numbers, and the README does not present them as free of contamination.
The README also discloses things a reader could not otherwise guess. The registry “does not identify which matcher scored each historical result,” and it does not independently identify whether the command-line tool or the Claude Code skill produced each result. The current harness runs the command-line tool, and before v8.2 that tool's critic did not apply the skill's eight kill gates. The methodology “is tuned on each batch's misses and then run on a new batch of 3–10 contests,” and “a single false positive moves a batch's precision by 10 points or more.” These caveats show what disclosure next to a number looks like. They also show what it can leave unknown: the README describes the project as it is now, not the exact setup of every historical run.
The list below is PSEEDR's, drawn from the Krait README and the research above. It is an editorial checklist, not an industry standard.
| Field | What to state | Why it changes the reading |
|---|---|---|
| Read access | What the agent could see during the run, including example tests and scoring code | Visible checks can be fitted |
| Write access | What it could modify, and whether tests were restored or diff-checked before scoring | A pass may measure an edit |
| Scorer placement | Whether the scorer ran in the agent's process or separately | A scorer that runs alongside the agent's code can be patched from inside |
| Scored checks | Whether the number comes from visible tests or a held-out set | Only unseen checks separate fitting from solving |
| Feedback | What the agent learned from scored checks between attempts | Feedback can reveal the holdout, and in ImpossibleBench it raised cheating |
| Scorer and matcher | Which method scored each result, and whether a model judged matches | Methods differ, and METR's two detectors disagreed |
| Review | Self-scored or externally reviewed | Self-scoring has no outside check |
| Contamination window | When the answer keys became public, compared with the model's training data where known | Public answer keys may already be in the model |
| Sample and tuning | Batch size, how much one error moves the number, and whether the method was tuned on earlier batches | Small, tuned samples can look stronger than they are |
The answer
If an agent can edit the test, the score measures the edit. The first fix is to remove that write access, and to restore the tests from a trusted copy before scoring so that a misconfigured permission cannot change the result.
Read-only files are not enough on their own. An agent can still special-case inputs, override equality or patch a scorer that runs its code. Keep the measurement outside the agent's process where the task allows, and let the score come from checks the agent has not seen. Expect that to cost some legitimate performance, and give the agent a sanctioned way to flag a test it believes is wrong.
Use a second model to decide which diffs a person reads, not to declare a run clean. And when the answer key may already be in the model's training data, no access rule fixes it. Write it next to the number.
Visible tests can stay, because they help the agent do real work. They should not be what the score depends on.
One question stays open for teams that need the agent to learn from scored checks. Pareek's pipeline returns failure summaries from a hidden suite, and the research shows that feedback improves performance and also raises shortcut-taking. How much can a hidden holdout reveal about its failures before it stops being hidden?
About this article: The cases attributed to Abhishek Pareek and Carlos Vendrell Felici come from their written responses to PSEEDR and are their own accounts. PSEEDR did not reproduce the cited experiments, run Krait or read its harness code. Krait details come from the project's public README at the time of writing.