QURI Launches RoastMyPost: An Open Source LLM Framework for Document Evaluation
Coverage of lessw-blog
In a recent post, lessw-blog introduced RoastMyPost, an experimental application developed by the Quantified Uncertainty Research Institute (QURI) designed to subject blog posts and research documents to rigorous automated evaluation using Large Language Models.
In a recent post, lessw-blog announced the release of RoastMyPost, a new experimental tool developed by the Quantified Uncertainty Research Institute (QURI). As the volume of online technical writing and research continues to accelerate, the capacity for human peer review struggles to keep pace. This release represents a significant step toward "AI-assisted peer review," moving beyond basic grammar correction into substantive, multi-faceted critique.
The core proposition of RoastMyPost is to utilize Large Language Models (LLMs) not just for generation, but for critical evaluation. Rather than relying on a single prompt to review a document, the application employs a suite of narrow, specialized evaluators. This modular approach allows the system to perform distinct checks, including Fact Checking, Fallacy Detection, Math Verification, Link Checking, and Forecast Checking. By decomposing the review process into these specific verticals, the tool aims to provide more granular and actionable feedback than generic LLM summaries.
lessw-blog highlights that the tool is currently optimized for the Effective Altruism (EA) and Rationalist communities. It features direct integration with URLs from the EA Forum and LessWrong, facilitating easy import and analysis of existing content. Furthermore, it includes specialized support for reviewing Squiggle models-a probabilistic programming language frequently used in these circles for estimation and forecasting. This specificity suggests that QURI is targeting high-context, analytical content where factual accuracy and logical consistency are paramount.
The application is designed to handle documents ranging from 200 to approximately 10,000 words. While the developers acknowledge that RoastMyPost is in an early, experimental stage, its release as an open-source project invites the broader community to test the boundaries of automated quality assurance. By making the tool free for reasonable use and providing public examples, QURI is fostering a collaborative environment to refine how AI can be used to vet complex intellectual work.
For researchers and writers, this tool offers a glimpse into a future where drafting and editing are augmented by automated "red-teaming," potentially raising the baseline quality of published research. The focus on specific, verifiable metrics-such as checking broken links or validating mathematical assertions-demonstrates a practical application of AI that complements human editorial judgment.
Key Takeaways
- RoastMyPost is a new open-source application from QURI that uses LLMs to evaluate research documents and blog posts.
- The tool employs specialized 'narrow' evaluators, including checks for facts, fallacies, math, spelling, links, and forecasts.
- It is optimized for the Rationalist and Effective Altruism communities, with support for Squiggle models and direct import from LessWrong/EA Forum.
- The system is best suited for documents between 200 and 10,000 words that contain factual assertions and simple formatting.
- This release serves as a public experiment to explore workflows for AI-assisted document vetting and quality assurance.