Multi-Models: Forecasting

(1) Triage (Mental Method)

Forecasting demands careful question selection. Concentrate on issues that are neither overly simple nor hopelessly complex. Rather than attempting long-shot guesses, like predicting presidential elections a decade out, target problems in the Goldilocks zone (Something neither super easy, nor super hard). Smart forecasters don’t try to predict what’s inherently unknowable. They know when to think – and when to let go.

Focus on questions where your hard work is likely to pay off. Don’t waste time either on easy “clocklike” questions (where simple rules of thumb can get you close to the right answer) or on impenetrable “cloud-like” questions (where even fancy statistical models can’t beat the dart-throwing chimp). Concentrate on questions in the Goldilocks zone of difficulty, where efforts pay off the most.

Tetlock also draws from chaos theory to explain why some questions resist prediction. Edward Lorenz’s (famed meteorologist) “butterfly effect” shows how a small change – like a butterfly flapping its wings in Brazil – can trigger a tornado in Texas. That sensitivity makes some systems too unstable to predict. In those cases, no model, no matter how sophisticated, can deliver consistent accuracy. Tetlock calls these cloud-like problems: messy, nonlinear, and inherently unpredictable.

How to Use It:

• Identify the predictability of the issue under consideration.
• Focus efforts on high-impact, moderately complex forecasts.
• Document and update task priorities based on new information.

(2) Break Problems Down – Fermi Method (Mental Method)

When faced with complex questions, superforecasters simplify. They break problems down into simpler, answerable parts. Inspired by famed physicist Enrico Fermi, they help predict unknowns by clarifying assumptions, making educated estimates, and reasoning their way through uncertainty. To demonstrate this concept, consider the following example:

Fermi once asked: “How many piano tuners are there in Chicago?” (without using google!) He began by breaking the question down into simpler answers: “What information would allow me to answer the question?” Fermi reasoned that four key facts were essential: 

1. The number of pianos in Chicago.
2. How often pianos are tuned each year.
3. How long it takes to tune a piano.
4. How many hours a year the average piano tuner works.

He estimated Chicago’s population, explaining, “I’m not sure, but I do know Chicago is the third-largest American city after New York and Los Angeles. And I think LA has 4 million people or so. That’s helpful. To narrow this down, Fermi would advise setting a confidence interval – a range that you are 90% sure contains the right answer… say, 2.5 million people.”

He then reasoned as follows:

“How many hours a year does the average piano tuner work? The standard American workweek is 40 hours, minus two weeks of vacation… So, I’ll multiply 40 hours by 50 weeks to come up with 2,000 hours. [Next] But piano tuners need to spend some time traveling between pianos… I’ll guess 20% of their work hours… so I conclude that the average piano tuner works 1,600 hours.

If 50,000 pianos need tuning once a year, and it takes 2 hours to tune one piano, that’s 100,000 total piano-tuning hours. Divide that by the annual number of hours worked by one piano tuner and you get 62.5 piano tuners in Chicago”

This method also supports Bayesian question clustering. Superforecasters often break a big forecast into sub-questions; each updated independently. This increases accuracy by clarifying what you know and what you assume.

Correct answer: 83 piano tuners in Chicago.

How to Use It:

• Deconstruct larger problems into its smaller, knowable and unknowable parts.
• Clearly outline your assumptions for each step.
• Regularly review and refine your assumptions based on feedback
• Bayesian question clustering:
  • This logic also underpins Bayesian question clustering. This method breaks down big questions into small, related forecasts. In doing so, each component receives analysis and estimates independently. By updating these sub-forecasts over time, they improve clarity and accuracy.

(3) Balance Outside and Inside Views (Mental Models)

Superforecasters start with the big picture, then narrow in. They begin with the outside view – historical base rates – and only then apply the inside view – situation specific details. This method prevents overconfidence and anchors the forecast in reality before adjusting for context.

Tetlock illustrates this concept with the Renzetti family. Frank and Mary live with their son and Frank’s widowed mother. Frank is a bookkeeper; Mary works part-time at a daycare. Asked whether they own a pet, most people focus on these personal details. Frank’s job, their home, their Italian last name – it feels like an easy opportunity to stereotype into a guess.

But superforecasters don’t start there. They begin with the base rate: What percentage of American households own a pet? That’s the outside view. Once that anchor is in place, they consider whether the Renzettis might be above or below the average.

Next comes the inside view, but it’s not a free-for-all. Superforecasters don’t wander through every detail – they investigate. In the (book example) Arafat poisoning case, they would start with a hypothesis – say, “Israel poisoned Arafat with polonium”  – and ask what must be true for that to hold:

So you fill a small library with books and settle in for six months of reading, right? Wrong… If you aimlessly examine one tree, then another, and another, you will quickly become lost in the forest. A good exploration of the inside view does not involve wandering around, soaking up any and all information and hoping that insight somehow emerges. It is targeted and purposeful: it is an investigation, not an amble. For example:

Start with the first hypothesis: Israel poisoned Yasser Arafat with polonium. What would it take for that to be true?
1. Israel had, or could obtain, polonium.
2. Israel wanted Arafat dead badly enough to take a big risk.
3. Israel had the ability to poison Arafat with polonium.

The outside view provides a statistical base. The inside view builds a focused case around it. When used together, they create a forecast grounded in reality – yet responsive to the situation.

How to Use It

• Begin with the outside view. Use base rates to anchor your forecast.
• Then apply the inside view – focused, not scattered.
• Ask: what must be true for this outcome to happen?
• Adjust the base rate only when specific evidence warrants it.
• Avoid stories that “feel right” but float free from data.

(4) Reacting Appropriately to Evidence (Mental Method)

Continuously refine forecasts as new data becomes available. Good forecasters avoid the twin risks of rigidity (underreaction) and whiplash (overreaction). Furthermore, good forecasters avoid both overreaction and inertia by applying Bayesian reasoning – a systematic way to update beliefs based on the weight of new evidence. As Tetlock writes, “The forecaster who carefully balances old and new [information] captures the value in both – and puts it into her new forecast. The best way to do that is by updating often but bit by bit.”

Bayes Theorem: P(H|D)/P(-H|D) = P(D|H) x (P(D|-H) x P(H)/P(-H) or [posterior odds = likelihood ratio x prior odds]. Tetlock explains it this way:

“The theorem says that your new belief should depend on two things – your prior belief (and all the knowledge that informed it) multiplied by the “diagnostic value” of the new information. That’s head-scratchingly abstract, so let’s watch Jay Ulfelder put it to concrete use: In 2013 the Obama administration nominated [former Senator] Chuck Hagel to be defense secretary, but controversial reports surfaced, and a hearing went badly… some speculated that Hagel might not be confirmed by the Senate. “Will Hagel withdraw?””

Suggested: Superforecasters use Google Alerts to track developments – setting them to deliver news daily when volatility is high.

How to Use It:

In this example, it’s important to start with the baseline estimate – your prior – which is derived from the base rates or historical data. For example, only one of 24 official nominees for U.S. Secretary of Defense has ever failed a nomination. That implies there’s a 96% chance of confirmation. But when new evidence surfaces – such as a poor confirmation hearing – forecasters must reassess. As Ulfelder stresses, “Bayes’s theorem requires us to estimate two things:”

1. How likely are we to see a poor Senate performance when the nominee is destined to fail?, and
2. How likely are we to see a poor performance when the nominee is bound for approval?

If bad hearings are common among rejected nominees and rare among confirmed ones, this new data should lower your probability estimate. In Chuck Hagel’s case, one analyst predicted that 95% of failed nominees performed badly, while only 20% of confirmed ones did. Plugging those values into Bayes’ theorem, the estimate dropped from 96% to 83%. Despite media rhetoric, the odds still favored Hagel’s confirmation.

OR

Probability that Hagel will be confirmed after bad hearing news:

Step 1: Start with prior probability: Historically, 23/24 Secretary of Defense nominees were confirmed = Prior (P(Confirmed) = 96% or 0.96.

Step 2: Estimate likelihoods:
1) How often do failed nominees perform poorly = 19/20 or 95%.
2) How often do confirmed nominees perform badly =1/5 or 20%. Thus, P(Bad Hearing | Rejected) = .95, P(Bad Hearing | Confirmed) = 0.20.

Step 3: Bayes’ Theorem
Here’s the simple Bayesian Theorem:

Where: P(Rejected) = 1 – P(Confirmed) = 1 – 0.96 = 0.04

Plugged in: P(Confirmed | Bad Hearing) = 0.20 x 0.96 / (0.20 x 0.96) + (0.95 x 0.04)

Calculate: Numerator = 0.20 x 0.96 = 0.192. Denominator = 0.20 x 0.96 + 0.95 x 0.04 = 0.192 + 0.038 = 0.23 Thus:

P(Confirmed | Bad Hearing) = 0.192/0.23 = 0.8348 OR 83% (probability).

(5) Look for Clashing Causal Forces (Mental Model)

Superforecasters don’t follow a single story. They map competing pressures – political, emotional, economic – and weigh them carefully. Tetlock describes this approach as “dragonfly eye” – a multi-lens view of where “one view meets another and another and another – all of which must be synthesized into a single image.” The best forecasts arise from integrating opposing narratives into a cohesive picture. Like a dragonfly’s compound vision, it’s not elegant – it’s integrative.

This approach also draws strength from the wisdom of the crowd. In 1906, Francis Galton watched hundreds of fairgoers guess the weight of a slaughtered ox. Their individual estimates varied widely. The result? Just one pound off. When pooled, the noise canceled out, and the signal remained. Superforecasters replicate this process in their own minds – combining conflicting arguments, integrating outside and inside views, and refining as new evidence appears. 

Tetlock highlights a forecasting anecdote that aligns with “dragonfly eye” analysis regarding Saudi oil production supporting “on the one hand/on the other” dialectal banter:

“On the one hand, Saudi Arabia runs few risks in letting oil prices remain low because it has large financial reserves,” wrote a superforecaster [trying to determine OPEC production cuts]. On the other hand, Saudi Arabia needs higher prices to support higher social spending to buy obedience to the monarchy. Yet on the third hand, the Saudis may believe they can’t control the drivers of the price dive, like the drilling frenzy in North America and falling global demand. So they may see production cuts as futile. Net answer: Feels no-ish, 80%.
[Outcome: Saudis didn’t support production cuts]”

Instead of anchoring to ideology, they build cognitive range. As Tetlock puts it, “For every good policy argument, there is typically a counterargument that is at least worth acknowledging.” Forecasting requires this art of synthesis – resisting simplistic models and modeling complexity. 

How to Use It

• Identify the strongest counterarguments to your forecast.
• Map opposing drivers clearly: what’s pushing for change, what’s resisting it?
• Synthesize these forces into a single probability.
• Adjust as the balance of power or evidence shifts.

(6) Embrace Degrees of Uncertainty (Mental Method)

Superforecasters are known to wield a sharp tool: precise probabilities that carve clarity from uncertainty. They don’t hedge with “maybe” or “probably.” They express uncertainty in specific, testable terms like 55%, 70%, or 92%. Tetlock writes,“If nothing is certain, it follows that the two-and-three-setting mental dials are fatally flawed.” Binary thinking – yes/no, win/lose – misleads more than it guides.

Tetlock highlights the example of catching Osama bin Laden. In the Situation Room, CIA officers gave varying probabilities that Osama bin Laden was in the Pakistani compound – some said 95%, others as low as 30%. President Obama listened, then said, “This is fifty-fifty -… a flip of the coin.” Obama later said the probabilities “disguised uncertainty” rather than clarified it. He saw the choice as a gamble – and acted accordingly. This is precisely the position Tetlock encourages, writing, “Most of us could learn, quite quickly, to think in more granular ways about uncertainty.”

For example, if you forecast 70% rain, it should rain 70% of the time. Resolution shows how bold and discriminating your forecasts are. A well-calibrated 90% forecast adds more value than a cautious 70%.

How to Use It

• Replace vague terms like “likely” with percentages.
• Distinguish small gaps – 55% is not 65%.
• Track calibration weekly. Use logs to compare predictions and outcomes.
• Use Brier scores** to identify overconfidence and underreaction.
• Make bold predictions when the evidence justifies them.

** Brier scores measure the accuracy of probabilistic forecasts. They calculate how close your predicted probabilities are to actual outcomes. A score of 0.0 is perfect, 0.5 means you’re guessing, and a score of 2.0 means you were completely confident and completely wrong. Lower scores reflect better calibration and sharper judgment.

(7) Balance Confidence and Caution (Mental Method)

Good forecasters don’t waffle and they also don’t swagger. They strike a disciplined balance – bold enough to act, careful enough to consistently revise. Overconfidence leads to bad bets, while excessive caution leads to missed opportunities. Superforecasters do both better by calibrating not just their forecasts, but also their posture.

Tetlock emphasizes that forecasting well requires scoring high on both calibration (accuracy) and resolution (boldness). As Tetlock writes, “They routinely manage the trade-off between the need to take decisive stands (who wants to listen to a waffler?) and the need to qualify their stands (who wants to listen to a blowhard?). Tetlock also warns that forecasters need to “tamp down both types of forecasting errors – misses and false alarms – the the degree a fickle world permits.”

How to Use It

• Avoid defaulting to safe forecasts in the 45-55% range.
• Be willing to take stronger stands when evidence justifies it.
• Track your hit rate across bold forecasts to refine risk-taking.
• Review forecasts where you hesitate and ask: should I lean in or hold back?

(8) Audit Mistakes Without Hindsight Bias (Mental Method)

In the art of superforecasting, mistakes aren’t failures – they’re feedback loops. They urge forecasters to dissect bad calls carefully, without rewriting the past. As Tetlock warns, “Once we know the outcome, that knowledge skews our perception of what we thought.” This hindsight bias creates the illusion that events were always obvious. It blocks learning and breeds false confidence.

Superforecasters avoid this trap by running postmortems. They ask: Did I misread the evidence? Did I confuse correlation with causation? Was I just unlucky? They distinguish between poor reasoning and unpredictable events. Not every miss means your model was broken. As Tetlock further writes, “There is always more trying, more failing, more analyzing, more adjusting, and trying again. Computer programmers have a wonderful term for a program that is not intended to be released in a final version but will instead be used, analyzed, and improved without end. It is perpetual beta.”

More importantly, hindsight bias can appear early in forecasting. Tetlock recalls how experts claimed they predicted the Soviet Union’s collapse – despite years of prior predictions to the contrary. After the fact, everything seems inevitable. But if you think you “knew it all along,” you stop asking why you were wrong.

How to Use It

• Keep a forecast log. Document your reasoning, not just the outcome.
• After each major result, ask: What did I miss? Was the logic sound?
• Don’t assume success proves your model. Luck can mislead.
• Separate signal from the noise. Was the outcome unpredictable or your reasoning flawed.
• Use postmortems to refine – not punish – your judgement.
• Tetlock: Don’t try to justify or excuse your failures. Own them!

(9) Elevate Others, Sharpen Yourself (Mental Method)

Superforecasting isn’t a solo sport. Great forecasters know that collaboration – done right – makes everyone smarter. The ninth commandment reminds us: bring out the best in others and let them do the same for you. That means asking probing questions, challenging assumptions, and welcoming disagreement – not as conflict, but as calibration.

Tetlock stresses this in Superforecasting. He writes:

“[G]roups also let people share information and perspectives. That’s good. It helps make dragonfly eye work, and aggregation is critical to accuracy. Of course aggregation can only do its magic when people form judgments independently, like the fairgoers guessing the weight of the ox. The independence of judgements ensures that errors are more or less random, so they cancel each other out… In so many ways, a group can get people to abandon independent judgement and buy into errors. When that happens, the mistakes will pile up, not cancel out…

…On the other hand, the opposite of groupthink – rancor and dysfunction – is also a danger. Team members must disagree without being disagreeable. Practice “constructive confrontation,” to use the phrase of Andy Grove, the former CEO of Intel. Precision questioning is one way to do that… Suppose someone says, “Unfortunately, the popularity of soccer, the world’s favorite pastime, is starting to decline.” How do you question that claim?: “What do you mean?” lowers the emotional temperature with a question but it’s much too vague. Zero in. You might say, “What do you mean by pastime?” or “What evidence is there that soccer’s popularity is declining? Over what time frame?”

Superforecasting, Tetlock reminds us, isn’t a paint-by-numbers exercise – but it has similarities. You don’t just plug in inputs and apply a formula. Teams must synthesize diverse views, test them openly, and refine them together. That requires constructive confrontation: pushing back without being hostile, and asking precise questions that sharpen – not soften – the argument. Tetlock highlights the paint-by-numbers approach:

Unpack the questions into components. Distinguish as sharply as you can between the known and unknown… leaving no assumption unscrutinized. Adopt the outside view and put the problem into a comparative perspective that downplays its uniqueness and treats it as a special case of a wider class of phenomena. Then adopt the inside view that plays up the uniqueness of the problem… Explore the similarities and differences between your views and others – and pay special attention to prediction markets and other methods of extracting wisdom from the crowds. Synthesize all these different views into a single vision.

How to Use It

• Practice perspective-taking: restate other’s views as clearly as your own.
• Ask clarifying questions: aim to sharpen the points.
• Create a culture where disagreement is safe, not personal.
• Seek feedback from those who think differently.

(10) Master the Error-Balancing Bicycle (Mental Method)

Superforecasting is not perfection – it’s a balance. It’s a balance between two kinds of failure: underconfidence and overreach. Like riding a bicycle, staying upright means making constant micro-adjustments. Every forecast requires a decision: be bolder or pull back?

Tetlock compares forecasting to learning how to ride a bike – not by memorizing rules, but by embracing the feel of drifting and adjusting. “Learning requires doing,” he writes, “with good feedback that leaves no ambiguity about whether you are succeeding or failing.”

How to Use It

• Reflect on each forecast: was I too bold or too cautious?
• Look for patterns in your errors – do you overcorrect or underreact?
• Track forecasts over time and watch how your range narrows.
• Get regular feedback. Use it to fine-tune, not just validate.

(11) Adapt, Don’t Worship Forecasting Principles (Mental Model)

Superforecasters use rules – but they don’t obey them blindly. Forecasting requires flexibility. No principle fits every case; conditions change. Models break, and judgment fills gaps.

Tetlock cites von Moltke: “It is impossible to lay down binding rules.” Superforecasters learn the principles, then apply them with nuance. They know when to follow the playbook – and when to write a new one. Forecasting isn’t following a formula. It’s using the right tool for the job – and knowing when to switch.