# Curated Digest: Why AI Struggles with Experimental Physics > Coverage of lessw-blog **Published:** April 27, 2026 **Author:** PSEEDR Editorial **Category:** platforms **Tags:** Artificial Intelligence, Physics, LLM Evaluation, Scientific Research, Machine Learning **Canonical URL:** https://pseedr.com/platforms/curated-digest-why-ai-struggles-with-experimental-physics --- A recent analysis highlights a critical limitation in Large Language Models: their inability to reproduce numerical results from experimental physics papers, revealing a gap between coding proficiency and true physical understanding. In a recent post, lessw-blog discusses a fascinating and sobering new preprint from Peking University that evaluates the capabilities of Large Language Models (LLMs) in reproducing numerical results from experimental physics papers. As the artificial intelligence landscape rapidly evolves, much of the focus has been on the impressive strides models have made in mathematics, coding, and theoretical computer science. This progress has fueled optimistic projections about the imminent arrival of automated AI researchers capable of autonomously driving scientific discovery. However, this topic is critical because experimental science is fundamentally different from pure mathematics or software engineering. It requires bridging the gap between abstract theory and messy, real-world physical setups. lessw-blog's post explores these dynamics by highlighting a critical blind spot in current AI capabilities: computational modeling and numerical simulation in the hard sciences. Evaluating AI in these specific domains is essential for setting realistic expectations and timelines for the deployment of autonomous scientific systems. The core of the lessw-blog analysis centers on a striking finding from the Peking University study: LLMs achieved a 0% end-to-end callback rate when attempting to reproduce full, numerical results from experimental physics papers. To evaluate this, the researchers utilized the PRBench benchmark, a framework specifically designed to test computational modeling and numerical simulation rather than mere coding proficiency. The post details how, despite their advanced capabilities, LLMs consistently stumble when faced with the nuances of experimental physics. Interestingly, the models are not entirely clueless. The analysis notes that LLMs can successfully answer high-level methodological questions about the papers. Yet, when tasked with the actual execution-data analysis and numerical simulation-they consistently make small, compounding errors that ultimately lead to erroneous final results. The models fail to infer the specific numerical methods required for a given physical setup and struggle to translate complex problem conditions into functional code. This indicates a profound lack of fundamental physical understanding. They can write the code, but they do not understand the physics that the code is supposed to represent. This study is significant as it represents one of the first rigorous analyses of LLM research skills outside of traditional, highly structured domains. It reveals that while AI can mimic the language of science, it currently lacks the deep, contextual comprehension required to execute complex scientific tasks. For professionals, researchers, and technologists tracking the frontier of AI capabilities, this analysis provides a crucial reality check. It underscores the ongoing necessity of human oversight in scientific research and suggests that the road to fully automated AI scientists may be longer and more complex than previously anticipated. To fully grasp the implications of the PRBench benchmark and the specific hurdles AI faces in experimental physics, we highly recommend reviewing the original analysis. [Read the full post](https://www.lesswrong.com/posts/c4Kopt4bY3ZbMdshJ/ai-is-bad-at-physics) to explore the nuances of these findings and what they mean for the future of automated research. ### Key Takeaways * A Peking University preprint reveals LLMs have a 0% success rate in reproducing full numerical results from experimental physics papers. * While capable of answering methodological questions, LLMs consistently fail at data analysis and numerical simulation. * The PRBench benchmark demonstrates that LLMs lack the fundamental physical understanding required to translate experimental conditions into code. * These findings challenge current expectations and timelines for the development of fully automated AI researchers in the hard sciences. [Read the original post at lessw-blog](https://www.lesswrong.com/posts/c4Kopt4bY3ZbMdshJ/ai-is-bad-at-physics) --- ## Sources - https://www.lesswrong.com/posts/c4Kopt4bY3ZbMdshJ/ai-is-bad-at-physics