Futuur 101: How to Improve My Prediction Skills?
While there are no magic formulas to achieve better predictions in sports, prediction markets, or life in general, decades of solid scientific evidence point to something like a method behind the randomness of real-life events.
Psychology and Political Science professor Philip Tetlock argues that forecasting abilities can be cultivated by everyone. This article gathers insights from his book Superforecasting: The Art and Science of Prediction, co-written with journalist Dan Gardner, which presents compelling scientific evidence for the case of learning how to forecast.
Tetlock's work includes decades-long comparisons between the predictive performance of experts and regular people. His collaboration with Barbara Mellers on the Good Judgment Project also brought him into contact with a vibrant community of "superforecasters," whose skills he analyzed thoroughly.
First Things First
The Importance of Systematic Feedback
Weather forecasting differs from most professional areas in that verifying prediction accuracy after the fact is part of the routine. This systematic approach is one of the keys to improving predictive skills. On Futuur's Portfolio, you can check your personal track record, the changes in value of open positions and analyze wins and losses for resolved markets.
Matching Confidence with Knowledge
The Portfolio is also useful in addressing a key challenge in making reasonable guesses for real-life events: matching our self-confidence with our actual knowledge of a specific topic. According to Tetlock, experience only leads to better results when followed by clear, repeated, and immediate feedback – otherwise, we risk falling into false overconfidence.
Learning from Successes and Failures
Trying and failing is inevitable on this path. Having full records available of how confident we were about an event (measured by the size and frequency of bets) is useful for analyzing our performance. This is especially true for long-term predictions, where memory is not always reliable about our real degree of confidence when placing a bet.
The Base Rate
Learning how to interpret data relating to an event is more relevant for success than formal study. However, data is not always available or clear. While weather forecasters improve their prediction accuracy with the help of solid, recurring statistical records, this will never be the case for predictors of political episodes and other uncertain events. These events’ limited incidence is insufficient to generate data and feedback consistent enough to provide predictions with statistical relevance. Knowing the winning party of a presidential election today isn’t helpful to predict the winner of an election 8 years from now, for instance.
Economists Daniel Kahneman and Amos Tversky have demonstrated that it's very easy to adhere to a biased judgment under uncertainty by hanging onto an arbitrary initial odd. Tetlock argues that this cognitive trap can be avoided by separating the analysis of an event into two parts: an 'external' approach and an 'internal' approach.
The external approach helps us finding out a statistical base rate: a probabilistic approximation of how often certain events occur in a certain type of situation.
The External Approach: Breaking Down Problems
Physicist Enrico Fermi’s famous rapid order of magnitude estimations often relied on simple algorithms. Credit: National Archives, Washington D.C.
The 'external' approach consists of decomposing a hard problem into smaller, easier problems to solve. The physicist Enrico Fermi's piano tuner problem provides an example of this approach, to figure out the incidence of a problem in a broader universe and reasonably fill in the gaps of missing data.
The problem: How to estimate the number of piano tuners in Chicago given only the population of the city?
The solution:
Chicago's population is estimated from an almanac: 3 million people.
Assuming an average family contains 4 members, there must be around 750,000 families in the city.
If one in five families owns a piano in Chicago, we come to 150,000 pianos in total.
Pianos typically need regular tuning once a year.
Some assumptions about the average piano tuner:
a. Tunes 4 pianos a day for 5 days a week.
b. Rests on weekends.
c. Has a 2-week summer vacation.The piano tuner would service 1,000 pianos in one year (52 weeks). So, 150,000 / (4 pianos a day x 5 days a week x 50 weeks) = approximately 150 piano tuners in Chicago.
The result of this reasoning is called the statistical base rate. Even though this is just an abstract and rough data approximation, narrowing numbers down to a probable order of magnitude tends to be a far better starting point than guessing randomly or using a “gut feel” approach. It also leads to even more educated guesses with the further refinement of the 'internal' approach.
Refinement
The Internal Approach: Analyzing Specifics
The next step, the 'internal' approach, consists of analyzing the particularities of the case to adjust the base rate up or down. You may have come to one or more hypotheses that can be directly researched.
Dealing with Conflicting Hypotheses
If the hypotheses conflict, try to investigate individually what would need to happen for each of them to be true. In some contexts, consider the following questions:
Who are the key players in this situation?
How powerful are they in influencing it?
What's motivating each of them?
How much or what exactly do they want?
Gathering evidence from scientific literature, statistical records, news reports, opinions from Futuur's community of users, and other sources regarding the event in question will be crucial to narrowing down possibilities and making fine adjustments to the initial guess.
Good. Now What?
This refinement step acts as an antithesis to the thesis obtained with the base rate, generating a 'synthesis'. However, the resulting synthesis is still just an estimate that should be re-evaluated and potentially changed in light of new evidence.
Make no mistake: it's not easy to master combining sometimes dissonant information! Frequent mindset adjustments may be important to avoid cognitive biases and misplaced (lack of) confidence. Treating beliefs as hypotheses to be tested rather than treasures to keep is a good rule of thumb to achieve progress.
On Futuur, you'll find tools to help you understand and aggregate other points of view in your search for better predictions. Besides contemplating the wisdom of the crowd reflected in the markets' odds, you'll also be able to discuss opinions with other predictors in our markets, community topics, or Discord channel, and follow the best forecasters in our Leaderboard.
Conclusion: Key Takeaways for Improving Prediction Skills
Systematic feedback is crucial: Regularly check your prediction accuracy and learn from both successes and failures.
Use the base rate approach: Start with a broad statistical estimate before refining your prediction.
Refine your predictions: Analyze specific factors and gather relevant evidence to adjust your initial estimate.
Stay flexible: Be willing to change your beliefs in light of new evidence and treat your predictions as hypotheses to be tested.
Leverage collective wisdom: Engage with communities like Futuur to gain diverse perspectives and improve your forecasting abilities.