# Semantic Deduplication: Mizchi’s Similarity Bridges Static Analysis and LLM Refactoring

> A Rust-based utility leverages AST parsing to identify logic overlap and prompt AI agents for remediation

**Published:** September 04, 2025
**Author:** Editorial Team
**Category:** devtools
**Content tier:** free
**Accessible for free:** true






**Tags:** Software Engineering, Rust, Static Analysis, Refactoring, LLM, Technical Debt, Open Source

**Canonical URL:** https://pseedr.com/devtools/semantic-deduplication-mizchis-similarity-bridges-static-analysis-and-llm-refact

---

As engineering teams integrate generative AI into their workflows, the volume of boilerplate and repetitive logic within repositories is increasing. Traditional duplication detection tools, such as `jscpd`, rely primarily on text matching or token hashing. While effective for identifying copy-pasted blocks, these tools often fail to detect semantic duplication where the logic is identical but variable names or formatting differ. Mizchi’s `similarity` attempts to solve this by parsing code into an Abstract Syntax Tree (AST), allowing for structural comparison rather than mere string matching.

### AST-Based Precision Over Text Matching

The core differentiator of `similarity` is its reliance on AST parsing to achieve what the documentation describes as "semantic level comparison". By analyzing the tree structure of the code, the tool can identify logic that is functionally equivalent despite superficial differences. This approach aims to reduce false positives common in text-based tools, which often flag coincidental string matches as duplication while missing structurally identical functions written with different syntax styles.

### The AI Refactoring Loop

Perhaps the most distinct feature of `similarity` is its intended workflow. Most static analysis tools output reports designed for human consumption—often static HTML or CLI tables. In contrast, `similarity` is explicitly designed to "output AI-friendly format". The tool generates structured data intended to be fed directly into LLMs such as Claude or GPT-4.

This design decision suggests a shift in how static analysis is utilized: rather than simply flagging technical debt for a human developer to resolve, the tool acts as a context provider for an AI agent. The workflow involves detecting the duplication via AST, exporting the relevant code blocks in a prompt-ready format, and utilizing an LLM to generate a unified, refactored function. This capability supports "intelligent refactoring suggestions", effectively using AI to clean up the redundancy that AI coding assistants often create.

### Performance and Language Support

Built in Rust, the tool is engineered for high concurrency and memory optimization, a necessity for processing the ASTs of large-scale code libraries. However, the maturity of language support varies significantly. The project documentation indicates that TypeScript and JavaScript support is "production ready". Python and Rust support is currently in "Beta", while a long tail of other languages—including Elixir, Go, Java, C/C++, C#, and Ruby—remains "mostly experimental".

### Limitations and Market Position

While the AST approach offers higher precision, it introduces complexity regarding language support. Unlike text-based deduplication, which can often run on any text file, `similarity` requires a specific parser for each language. This dependency explains the disparity in maturity levels across different languages. For teams working primarily in the TypeScript/JavaScript ecosystem, the tool offers immediate utility. However, for enterprise environments relying on Java or C++, the "experimental" status suggests it may not yet be a viable replacement for established tools like SonarQube or PMD.

Ultimately, `similarity` represents a new category of developer tooling: infrastructure designed not just for human developers, but as middleware between codebases and Large Language Models.

---

## Sources

- https://github.com/mizchi/similarity