# Before an AI Novelty Check Kills or Clears Your Idea, Ask What It Searched > An AI novelty verdict bundles three claims: a match you can open, an absence that is only as good as the search behind it, and a yes-or-no that can change with wording. Three builders' cases and recent research on LLM novelty judges show which check each claim needs before it decides whether an idea is dropped or pursued. **Published:** 2026-11-02 **Author:** PSEEDR Editorial Team **Category:** enterprise **Content tier:** free **Accessible for free:** true **Editorial format:** analysis **News quality eligible:** true **Source count:** 10 **Word count:** 3486 **Read time:** 16 min **Tags:** Platforms **Canonical URL:** https://pseedr.com/enterprise/before-an-ai-novelty-check-kills-or-clears-your-idea-ask-what-it-searched ![A blue search spotlight lights part of a dark field of abstract document and repository tiles; one lit tile is opened, while many tiles outside the beam stay dim. Text: NO MATCH ≠ NEW. Check what the AI searched. PSEEDR.](https://content-edge-frzs2wwjiq-uc.a.run.app/assets/article-covers/before-an-ai-novelty-check-kills-or-clears-your-idea-ask-what-it-searched.ad5f3660.jpg) --- ## Watch the analysis [Watch the full video on YouTube](https://www.youtube.com/watch?v=SmvYjEvUZvg) · [Watch the Short](https://www.youtube.com/shorts/ROsqUk5Mo_g) In late August 2026, Nick Sawinyh, CEO of [ITS Feed](https://its-feed.com/), was handed three candidate ideas for a coding-agent challenge and picked one: a strict-typing agent, meant to tighten type checking and scored on introducing no new type-check suppressions. Only after he made the pick did his AI collision search look for prior work. The search found some. By Sawinyh's account, PyTorch already kept an agent skill in its repository that performed the same transformation, “with the never-suppress rule almost word for word,” and an existing academic benchmark already covered the evaluation angle. He retracted the pick the same day. Sawinyh described that episode, and a second one from October, in a written response to PSEEDR. He said he has no commercial relationship with any tool he mentioned. The lesson he drew was not that the agent had failed, but that its two kinds of answer deserve different levels of trust: “An agent saying something exists is checkable: you open the link. An agent saying nothing exists is a claim about everything it didn't look at.” That distinction matters to anyone who asks a language model or search agent whether an idea has been done before spending time on it: engineers, technical founders, hackathon entrants and applied researchers. The reply usually arrives as one verdict, but it bundles three different claims. * **A match:** “this already exists, here.” It is a pointer you can open to see what overlaps. Whether the overlap covers the part of your idea that is supposed to be new is a separate question. * **An absence:** “nothing close exists.” This is a statement about a search, and it is only as reliable as a description of what that search covered. That description is often missing. * **The judgment:** “new” or “not new.” It weighs the first two against a threshold, and the way the question is asked sets that threshold. Each claim needs its own check. Verify every match by opening it and comparing it with your idea part by part. Override “nothing close” unless it rests on a source whose coverage you can state, and even then accept it only for the population that source covers. Treat a yes-or-no that changes when only the wording changes as a sign that nobody fixed what “new” meant. ## Opening the link: what the PyTorch file says A match is the one part of a novelty verdict that a reader can check directly, so PSEEDR checked this one. We confirmed the PyTorch file and its history. We did not see what Sawinyh's agent returned or how he worded his idea, and we could not check the benchmark he mentions, because he did not name it. The file is a skill called `pyrefly-type-coverage`: a set of instructions that tells a coding agent how to move a Python file onto stricter checking with Pyrefly, a type checker. Its commit history shows that it was first added to the pytorch/pytorch repository in February 2026, at `.claude/skills/pyrefly-type-coverage/SKILL.md`, and revised in April. The [version committed on June 24, 2026](https://github.com/pytorch/pytorch/commit/41adcae37c80c864d2cb1486ced73f19048178fa) was not changed again until October, when the repository moved its skills to `.agents/skills/`; after an October 3 attempt was reverted the same day, the move [relanded on October 5](https://github.com/pytorch/pytorch/commit/0f30a5acadc0daf21cb85425469996f650367884). The [current file](https://raw.githubusercontent.com/pytorch/pytorch/main/.agents/skills/pyrefly-type-coverage/SKILL.md) has the same text as the June 24 version. A matching file was therefore in the repository when Sawinyh searched in late August. The text supports his description of the central rule. The skill names three error categories, `unannotated-return`, `unannotated-parameter` and `implicit-any`, and says they are “always resolvable; never suppress them.” It repeats the point later: “Never suppress the three target categories.” The rule is narrower than a blanket ban, though. It makes one exception: functions marked as backward compatible must keep their signatures unchanged, because a test compares those signatures against a stored reference, so the skill tells the agent to put a suppression comment on the `def` line instead of adding annotations. It allows narrow ignore comments on other error categories as a last resort. It tells the agent to delete existing file-level suppressions and any suppressions that become unused. It has no scoring or evaluation section. One more detail matters for anyone citing the case: the skill lives in PyTorch's source repository, and the commit that moved it notes that PyTorch's source distribution already excluded the old `.claude` folder and would now exclude `.agents` as well. So the match covered the transformation and the rule at its center. What remained distinctive was the scoring, and by his account an existing benchmark already held that angle. Opening the link turned “it exists” into a part-by-part comparison. On PSEEDR's reading of the file, that comparison supports his retraction. Sawinyh's criticism of his own process is about timing, not about the agent: “The collision search ran after my pick, which was the real mistake.” His fix was procedural: “Since then nothing reaches me for a decision until its search has come back.” ## A match covers parts of an idea, not always the new part In the PyTorch case, the match landed on the core of the idea. A match can also land beside the core. Then opening the link reverses the verdict instead of confirming it. Girish Songirkar, Delivery Manager, Enterprise Software Engineering, at [Arionerp](https://www.arionerp.com/), described that outcome in a written response to PSEEDR. During what he called a recent large-scale system implementation, his team used a large language model to vet an optimization for a warehouse management system. The optimization concerned dynamic slotting, which decides where goods are stored, and it fed real-time labor-fatigue metrics directly into the ERP logic. The model flagged the approach as unoriginal and cited several papers from 2020 and 2021 on automated slotting. Songirkar read the cited papers. “When I audited the cited literature, I found those papers assumed static labor capacity and idealized environments,” he said. His team's approach depended on how human performance changes across a 24-hour industrial cycle, which a model with fixed labor capacity leaves out. He overrode the verdict. The match was real: slotting optimization already existed. It shared the algorithmic frame with his idea, but not the input assumption that made the idea different. Songirkar did not date the episode or name the papers, so PSEEDR could not check them. His reading of them is his own, and his broader statement that the academic literature ignored the problem rests, as far as his response shows, on the papers he checked. He was also judging his own team's work. Kruno Sulić, Founder & SaaS Product Builder at [Cliprise](https://www.cliprise.app/), reported a similar pattern in product work. Around 2025, he used an LLM to judge whether ideas for AI-assisted creator workflows were genuinely differentiated or recombinations of existing AI video and content-production features. “AI often labeled an idea ‘not novel’ because similar components existed, but a manual review sometimes showed the execution angle was still commercially distinct enough to test,” he said in a written response to PSEEDR. These were his own company's ideas, and he describes the pattern from practice rather than from a count. Research describes a mechanism that is consistent with both accounts, without verifying either. In a [study of LLM novelty judges](https://arxiv.org/abs/2610.02022) released in October 2026, researchers at Ai2 inspected 10 cases where giving a judge five abstracts of related work turned a correct verdict into a wrong one. In one case, the judge without retrieval rated an idea that combined two concepts as novel, in line with the peer reviewers. Shown papers that each presented one of the concepts, it dismissed the same idea as “a direct combination of existing ingredients…”. Retrieval also misled judges in the opposite direction. Ten inspected cases illustrate a failure; they do not measure how often it happens. The ideas were machine-learning research abstracts, not warehouse or product ideas. The same study shows that “new” is a threshold rather than a fixed property. Its pointwise judges, which rate one idea at a time, were asked whether an idea clears the novelty bar of a top AI venue. The authors note that an idea can be original in some respect and still fall short of that bar, for example as a straightforward composition of known methods. One reviewer comment in their data faulted a submission as “primarily procedural integration of existing components.” Under that standard, Sulić's differences in packaging and workflow would count for little. For a product team deciding what to build, packaging, workflow, pricing and audience fit can be the whole difference. Patentability is a third standard with its own legal tests, and it is outside the scope of this article. The practical check is to write down what your idea claims is new before you accept any match as a collision. The claim might sit in the mechanism, in an input or assumption, in the target user or packaging, or in the evaluation. Then mark which of those parts each match covers. A match is a collision only when it covers the part you claim, judged by the standard your audience will apply. Songirkar's check worked because he compared the papers' assumptions, not only their algorithms. Some expert review processes already ask for this kind of specificity: in a [2024 human study of research ideation](https://arxiv.org/abs/2409.04109) by Si, Yang and Hashimoto, the review form asked any reviewer who gave a low novelty score to name the similar related work. ## “No close matches” is a statement about a search A match can be opened. An absence cannot. The only way to judge “nothing close exists” is to ask what the search could have seen. Sawinyh's second case shows a version he trusted. In October 2026, he was weighing whether to enter a large Devpost event. Before dispatching any agent, he queried GitHub's repository search directly for repositories created after the event opened that named the event or its sponsor. By his account, that returned 277 public entrant repositories in a few calls. He grepped each of seven candidate concepts against the list. Five were killed outright, and the list caught two near-identical entries that a dispatched AI collision agent had missed. He skipped the event. He labels the 277 repositories and the five-of-seven result as measured. PSEEDR did not rerun his query and does not know its exact text. His rule from that experience is direct: “What I override: its negative findings, unless they rest on a source I can name and know to be close to exhaustive, like that repo list.” GitHub's documentation shows what “close to exhaustive” can mean for such a list. When a repository search does not specify which fields to search, “only the repository name, description, and topics are searched,” according to [GitHub's guide to searching for repositories](https://docs.github.com/en/search-github/searching-on-github/searching-for-repositories). README text is searched only when the query asks for it, and repository search cannot otherwise find content inside a repository. Forks are left out unless the query includes them. A `created:` qualifier filters by creation date. Through the API, search returns “up to 1,000 results for each search,” marks a query that ran out of time with an `incomplete_results` flag, and silently limits results to repositories the caller can access, according to the [REST API documentation](https://docs.github.com/en/rest/search/search#search-repositories). Together, those rules define the population that a query like Sawinyh's covers under default settings: accessible, non-fork repositories, created after a date, whose name, description or topics (or README, if requested) mention the event or sponsor, at the moment of the query. His 277 results were well under the 1,000-result cap, so the cap did not cut the list short. His response does not say whether he searched READMEs, included forks or checked the incomplete-results flag. Within that population, a list like his can come close to complete. Outside it, the list is silent: it cannot see entrants who never named the event, kept their repositories private, reused an older repository or built somewhere other than GitHub, and under default settings it also skips forks. It cannot show that no similar entry exists anywhere, and Sawinyh did not claim that. Its value is that its edges are known. The two cases also differ in the unit being searched. The October collisions were whole repositories that labeled themselves with the event. The August collision was one file inside a very large repository. A repository search over names, descriptions and topics would not reach a skill file inside pytorch/pytorch. Code search, which looks inside files (on a repository's default branch, for files under 384 KB, per GitHub's documentation), is the kind of search that could. A negative claim that does not name its unit leaves open whether the search could have found the kind of thing that collided. A usable “nothing close” therefore names its source, the unit searched (repositories, files, papers or products), the fields, the filters, when the search ran, how many results came back, and whether the result set was complete. It then holds only for that population. An agent report that states none of this gives the reader no way to tell which existing work the agent could have seen. ## Why a stronger search agent does not settle it The obvious fix is a better search agent. The Ai2 study tested versions of that fix on its own benchmark, and the results do not support it as a general remedy. Giving judges related work produced a statistically significant gain of at most 16 points, and only for some models. The researchers also built a much stronger agent: GPT-5.6 Sol at an extra-high reasoning setting, searching arXiv iteratively with no cap on searches. According to [Ai2's summary of the study](https://huggingface.co/blog/allenai/novelty-bench), it scored 0.78 macro-F1, against 0.89 for a Claude Opus 4.6 judge that received five pre-retrieved abstracts, at roughly ten times the cost per decision. Two purpose-built novelty evaluators did worse than the simplest configuration. A Claude Opus 4.6 prompt at low reasoning, with a majority vote over three calls, scored 0.854 at a total cost of $2.59. Idea Novelty Checker scored 0.642 at $30.66, and Sakana AI's AI Scientist scored 0.772 at $82.05. Two limits apply to those numbers. First, they come from machine-learning research ideas written as abstracts and drawn from ICLR 2026 reviews. Second, the scores were measured on a set where AI-generated ideas are assumed to be less novel than human ideas that reviewers strongly endorsed. The authors call these weak labels. One expert checked 30 pairs, chosen mostly from cases the strongest judge got wrong, and sided with the human idea in all 30. That is a sanity check, not a verification of every label. The scores rank configurations on this benchmark; they are not general accuracy rates. Within those limits, the pattern fits the builders' cases. More search produces more retrieved material, not a statement of what was covered, and the judgment step after retrieval can still go wrong in either direction. Sawinyh's repository list was not useful because it was large. It was useful because its edges were known. He adds a related observation and labels it himself: “Impression, not measurement: agents search well along the framing you hand them and poorly outside it.” ## When the verdict flips, the question was not pinned down The third claim in a novelty verdict, the yes-or-no, can change when only the wording changes. Sulić saw this happen. “Different prompts could shift the answer from ‘not new’ to ‘novel combination,’ which told me the model was highly sensitive to framing and probably reconstructing plausibility rather than checking the market in a reliable way,” he said. When that happened, he stopped treating the output as evidence. He used the model mainly to rank ideas, not to kill them automatically. He reported no counts. Research measures the effect. In the Ai2 study, one prompt told GPT-5.4 that reviewers had found exactly one idea in each pair novel and asked which one; another prompt simply asked which idea was novel. Switching between the two prompts changed the model's verdict on more than half of the identical idea pairs (55.8 percent). That finding does not depend on the benchmark's labels, because two opposite answers about the same pair cannot both be right. On the weakly labeled pairs, the same switch moved accuracy from 93.5 percent to 40.9 percent. No single prompt was best for every model: removing a warning against crediting trivial or incremental contributions helped some judges and hurt others. Outside novelty tasks, a [study of framing bias in LLM judges](https://arxiv.org/abs/2601.13537) gave 14 judges logically paired questions, such as whether a text was toxic and whether the same text was non-toxic. Every judge gave inconsistent answers across some pairs. The most consistent judge, GPT-5-mini, was inconsistent 5.69 percent of the time, compared with 0.8 percent when the identical prompt was simply repeated, so the flips were not mainly sampling noise. In examples the authors examined, a judge gave the same answer to a question and to its negation, which is itself a contradiction, and justified the two answers with opposite reasons. A fluent rationale does not show that a verdict was stable. Some flips have a more mundane cause. [JudgeSense](https://arxiv.org/abs/2604.23478), a single-author preprint, traced most of an apparent flip rate on one factuality task to a single prompt template that reversed the meaning of “yes” and “no.” After that was corrected, most judges looked close to fully consistent on that task. Before concluding that a model is unstable, check whether the two prompts asked the same question. Sulić's example sits between these findings. “Not new” and “novel combination” can both be true at once: every component exists, and the combination does not. If the prompt does not say which of those it is asking about, the answer follows the framing. The fix is to pin the question down before comparing verdicts. State whether you are asking about the components, the combination, or commercial distinctness for a named audience. Keep the polarity constant, because “is this new?” and “does this already exist?” invite opposite yes-or-no answers. Ask for the closest existing work and its overlap with each part of the idea, not a bare yes-or-no. If the verdict still moves after that, treat it as unresolved rather than as evidence either way, and fall back on the part-by-part comparison described earlier. A split like Sulić's already shows where that comparison should start: the components exist, so the open question is the combination. ## What to verify, what to override When an AI tool judges whether an idea is new, which parts of its verdict can you verify, and which should you override? Each part earns trust in a different way. Part of the verdict What it is Check before acting on it Override it when “This already exists” A pointer to something you can open Open it and mark which of the idea's claimed-new parts it covers It covers only parts you do not claim as new, under the standard your audience applies “Nothing close exists” A statement about a search Name the source, unit, fields, filters, time, result count and completeness Its coverage cannot be stated, or it covers a different population from the one that matters “New” or “not new” A judgment against a threshold Fix the definition and polarity, and ask for overlap with each part It changes when only the wording changes The order of work matters as much as the checks. Run the search before committing, not after; Sawinyh's August retraction undid a commitment that an earlier search could have spared him. Keep matches and absences in separate lists, because they need different checks. Let the tool rank candidates and narrow the queue, and leave the decision with a person who has opened the matches and can state the coverage behind the absences. Sulić put the division of labor this way. His rule became: “let AI narrow the queue, but let human review decide whether something is merely familiar, genuinely saturated, or still worth building in a sharper form.” The Ai2 authors draw a similar line for research, cautioning against deploying current novelty judges in consequential settings or as a substitute for human peer review. This is PSEEDR's synthesis of three practitioners' accounts and the research cited above, not a tested procedure. It does not say that AI novelty checks are useless. In August, Sawinyh's agent surfaced a real collision, which PSEEDR's check of the PyTorch file supports, and in October his direct query surfaced collisions the agent had missed. The point is narrower: a found match, a reported absence and a yes-or-no are three different claims, and each needs its own check before it decides an idea's fate. One question remains open. Sawinyh's repository list worked because its population could be enumerated. Many of the places where an idea might already exist cannot be enumerated that way: the full research literature, private products, and entrants who never label their work. For those sources, what coverage statement would be good enough to accept “nothing close”? --- ## Sources - https://github.com/pytorch/pytorch/commit/41adcae37c80c864d2cb1486ced73f19048178fa - https://github.com/pytorch/pytorch/commit/0f30a5acadc0daf21cb85425469996f650367884 - https://raw.githubusercontent.com/pytorch/pytorch/main/.agents/skills/pyrefly-type-coverage/SKILL.md - https://arxiv.org/abs/2610.02022 - https://arxiv.org/abs/2409.04109 - https://docs.github.com/en/search-github/searching-on-github/searching-for-repositories - https://docs.github.com/en/rest/search/search#search-repositories - https://huggingface.co/blog/allenai/novelty-bench - https://arxiv.org/abs/2601.13537 - https://arxiv.org/abs/2604.23478