Sharpening Prediction Skills: The Forecasting Dojo Initiative
Coverage of lessw-blog
In a recent event announcement on LessWrong, the community is organizing the inaugural meetup for a 'Forecasting Dojo,' a practice group dedicated to improving predictive accuracy through deliberate exercises and immediate feedback loops.
In a recent post, lessw-blog announces the first gathering of a new forecasting practice group, scheduled for March 1 via Discord. This initiative, styled as a "Dojo," represents a shift from theoretical discussions of probability to the practical, deliberate practice of prediction skills. For professionals in technology strategy, AI safety, and risk assessment, this grassroots effort highlights the growing importance of rigorous calibration in decision-making processes.
The Context: Why Deliberate Practice Matters
Forecasting is often viewed as an abstract art or an innate talent. However, research-most notably the work surrounding the Good Judgment Project and Philip Tetlock's Superforecasting-demonstrates that prediction is a skill that can be cultivated. The primary obstacle to improving forecasting accuracy is the feedback loop. In real-world scenarios, a forecast made today regarding an AI milestone or market shift might not be verifiable for months or years. This delay makes it difficult for the forecaster to internalize what went wrong in their reasoning or calibration.
The concept of a "Forecasting Dojo" addresses this gap by creating an environment for rapid iteration. By focusing on calibration exercises, participants aim to align their subjective confidence with objective reality-ensuring that when they say an event is "70% likely," it actually occurs 70% of the time.
The Gist: Pastcasting and Immediate Feedback
The lessw-blog announcement outlines a structured agenda for the meetup, emphasizing active participation over passive listening. The core of the session involves a technique known as "pastcasting," facilitated by a tool referred to as "Sage."
Pastcasting is a methodology where participants forecast historical events where the outcomes are already known to the system but hidden from the user. This approach solves the temporal latency problem inherent in standard forecasting. Participants analyze the data available at the time of the event, make a prediction, and receive immediate verification. This allows for an instant "post-mortem" discussion regarding the reasoning used, biases encountered, and the accuracy of the probability assigned.
The session is designed to be accessible, requiring no prior preparation. It will open with a short calibration exercise, move into the Sage pastcasting session, and conclude with a discussion on the group's future trajectory. This format suggests a commitment to building a sustainable community of practice where members can benchmark their skills against peers and historical data.
Why This Matters
For the PSEEDR audience, this event signals a broader trend toward data-driven epistemology. As reliance on probabilistic models (both human and machine) increases, the ability to distinguish between a well-calibrated forecast and a confident guess becomes critical. Initiatives like this provide the training ground necessary to develop those distinctions.
We recommend reviewing the original post to understand the logistics of the meetup and to explore the tools being utilized for these calibration exercises.
Key Takeaways
- The inaugural Forecasting Dojo meetup is scheduled for March 1, 11:00-12:00 CET on Discord.
- The session utilizes 'pastcasting' on historical events to provide immediate feedback on prediction accuracy, bypassing the long wait times of real-world forecasting.
- A tool called 'Sage' will be employed to facilitate these exercises, hiding outcomes to simulate genuine uncertainty.
- The meetup is open to all skill levels and emphasizes practical calibration over theory.
- This initiative represents a move toward deliberate practice in decision science, essential for reducing bias in strategic planning.