Superforecasting: How it’s done

Over the past 10 or so years there has been a lot of hype over superforecasting and superforecasters. Does such skill exist? I emphatically think so.

From Ten Commandments for Aspiring Superforecasters

For instance, don’t ask, “Who will win the world series in 2050?” That’s impossible to forecast and unknowable. The question becomes more interesting when we come closer to home. Asking in April who will win the World Series for the upcoming season and how much justifiable confidence we can have in that answer is a different proposition. While we have low confidence in who will win, we can have a lot more than trying to predict the 2050 winner. At worst, we can narrow the range of outcomes. This allows us to move back on the continuum from uncertainty to risk.

And from Wiki:

A superforecaster is a person who makes forecasts that can be shown by statistical means to be consistently more accurate than the general public or experts. Superforecasters sometimes use modern analytical and statistical methodologies to augment estimates of base rates of events; research finds that such forecasters are typically more accurate than experts in the field who do not use analytical and statistical techniques.[1]

I will use myself as an example. It’s broken down into three parts:

1. mental models, systems
2. risk-reward analysis
3. pattern recognition

A month ago I predicted that as stocks recovered that Bitcoin would significantly lag 3x ETfs and Tesla. The prediction is broken into three parts: stocks going up, Tesla going up, and Bitcoin lagging. The odds of all three of these occurring is 1/8. Although Tesla and stocks going up is positively correlated, hence it’s not exactly a coin flip, this is negated by Bitcoin lagging stocks, because it’s assumed that Bitcoin and stocks are also positively correlated. But then I also predicted that the dollar would keep going up. And that emerging markets would continue to lag US equites. So the odds of getting all 5 of these correct is 1/32.

As shown below, over the past month Bitcoin has greatly lagged 3x ETfs as the Nasdaq has staged a huge rebound:

My idea behind forecasting is developing so-called pillars, mental models, or systems, from which everything else deterministically falls out of it. So rather than having to predict a dozen stocks or markets as individual, independent events, the outcome is predetermined by the system as a sort of ensemble, similar to predicting the trajectory of a ball or the solar system based on Newton’s laws. If you know that capital flows along a gradient that maximizes its returns, it means that although there are hundreds of possible investments, only a handful will consistently continue to reap all of the gains, generally high-IQ ones, which is why the Nasdaq and Bay Area real estate have done so well and emerging markets, foreign currencies, and non-tech sectors have done more poorly, even going as far back as the ’90s. A second system/heuristic is that companies which have market dominance, network effects, ubiquity, and moats will outperform companies which lack those attributes.

This is why forecasting does work for sports betting, because there are too many variables and it’s hard to develop a system or framework around it like you can with stocks, economics, or even foreign policy (such as how I was correct Russia making little progress against Ukraine).

Today I was looking at Super Bowl 2023 odds. The New England Patriots have won the most Super Bowls, a total of six, so it logically follows that the Patriots should be the most favored team, but according to bookmaking site, “the current favorites to win Super Bowl 2023 is the Buffalo Bills at +600, followed by the Buccaneers +750, Chiefs +900, and Packers +1200.” But the Patriots rank at only +4000. This is probably because the odds are not just a function of past wins, but a slew of other variables such as the quality of players, the coach, injuries, and of course, other bettors. Star quarterback Tom Brady in 2019 left the Patriots to join the Tampa Bay Buccaneers, which can explain why the Patriot’s odds are so low in spite of past wins, and why the Buccaneers are much more favored (+700). You can try to bet on teams which have never won, but this introduces considerable potential downside for a tiny payoff. If a team is +10,000 favored to win, it means you are risking $10,000 to win $100 (by taking the opposite side of that bet). With stocks, you are not making all-of-nothing bets but rather betting on trends and systems, which is easier and not as risky.

Having a math background also helps in forecasting. I am able to observe patterns that others can easily overlook, and make calculations to optimize risk vs. reward for investments, such as betting/position sizing. A decade ago I was reading about general relativity. A major takeaway is that geometry, specifically curvature, is a very important part of the theory. This spurred me to took for geometric properties in other processes, like the financial markets. The idea is that the smoothness of the chart, which is a geometric property, is of predictive value. Smoother charts, in which there is the appearance of ‘price compression’ is bullish. Too much variance or roughness/jaggedness in price action is bearish. (However, this does not work for indexes, like the S&P 500, due to variables like pension fund inflows, profits, and economic fundamentals. It only works for closed-system processes which are almost completely a function supply and demand.)

Consider the chart of Bitcoin over the past 3 years (click to enlarge):

Notice the orange circles on the left, and compare to the green circles. What is the difference? The price action denoted by the orange circles is much more smooth and compressed, even adjusting for the y-scale ($10,000 in 2019 vs. $23,000 now), compared to the green circles. Now look at the right-most circle: It’s not smooth, but rather there is a lot of roughness as the prices oscillates between $18,000 and $24,000 (presently at $22,950). So this is strongly indictive that Bitcoin will fall further, as is has done earlier (denoted by the green circles). It will keep falling until there is sufficient smoothness. How many other people observed this? Probably no one else, afik, based on my own research. Patten recondition is an invaluable part of forecasting, not just extrapolating current trends into the future, but seeing things which no else can see and then then using this information to make decisions before it becomes more widely known (this is what is called an ‘edge’ in investing).

This is also related to IQ. For what it’s worth, I took a bunch of pattern recognition matrices tests, which is a component of IQ, and got close to perfect scores on all of them, suggestive of an IQ of 135+ (it does not go much higher because it’s difficult to develop a multiple-choice test that can easily be administered in a timed setting that has enough granularity to distinguish between, say, a 130 vs. 150 IQ). This also agrees with Tetlock’s findings, “The best forecasters scored higher on both intelligence and political knowledge than the already well-above-average group of forecasters. The best forecasters had more open-minded cognitive styles. They benefited from better working environments with probability training and collaborative teams. And while making predictions, they spent more time deliberating and updating their forecasts.”

But I don’t think you need to be that smart, rather you need to have the correct mental models or systems. Even smart people like Peter Thiel got it so wrong with Bitcoin because their models are bad, much like buggy software, not because they are not smart enough. If the software sucks, it does not matter how fast the computer is. People think that economics and finance is harder to predict than politics and geopolitical stuff. It’s usually the opposite. Politicians are unpredictable, as are voters. Who saw Roe v. Wade being overturned? Trump 2016? Brexit? Covid? Same for Russia invading Ukraine…why 2022? In hindsight, you can argue it was due to Biden, no one in 2021 was talking about it. But capital will always seek to optimize…so it’s easier to predict. Hedge funds, major investors are not going to prop up the inferior stuff for too long…the superior stuff tends to win out in the end. Consumers will keep on consuming no matter who is in charge or what the ‘current thing’ is. Famous rich people tend to be better at building long-term brands compared to politicians, an example being Trump. That’s how I correctly predicted in early 2021 that Elon Musk’s brand/popularity and wealth would keep growing despite scandal and the liberal media hating him. Presidents tend to have short careers and are not particularly well-liked, Elon Musk has been relevant since 2010 and keeps growing in popularity. Ron DeSantis and Greg Abbott also keep growing in popularity. Another trend I am predicting is that governors of Florida and Texas will play a much larger role in politics relative to the executive branch. But popularity or importance is harder to score than something easily empirically verified like a stock return.

There is a notable exception, taxation, because it involves capital, which is more predictable than ‘culture war’ stuff. That’s how I was right in 2020-2021 predicting that Biden would not raise taxes and there would be no ‘billionaire tax’, but rather that Biden’s focus would be on the post-Covid economic recovery and the national rollout of the Covid vaccines. This is yet another correct application of a system to make a correct forecast. “Read my lips: no new taxes” is such a famous quote because Bush’s betrayal on taxes was so bad that it overshadowed everything else that was bad about his presidency. Raising taxes is something that isn’t done without extreme deliberation and not without considerable resistance, compared to other policy.

In fairness however, the downside of superforecasting is it’s hard to ascertain individual forecasting prowess in a scientifically rigorous sense. It’s obviously easy to inflate your ‘score’ by making easy predictions “the sun will rise” or making predictions in which success or failure of the prediction is vague. For example someone can predict “X event will happen at ‘Y’ years,” but if the event occurs before ‘Y’, it is correct or not (does “at” include “before”)? Also, not all predictions are weighted the same. Predicting which team will win the NBA finals is a coin flip, yet something like predicting 911 is harder to quantify in probabilistic terms, yet way more important. Is it better or more demonstrative of skill to get many smaller predictions wrong and then get one or two big things right, or the other way around? In the former, due to small sample size it’s harder to disentangle skill from luck. Same if you have 2^x forecasters; by pure luck, you can expect a single person to make x correct forecasts out of x forecasts.