We looked at 2,129 resolved predictions from Metaculus, a platform where thousands of people put real forecasts on the record with confidence scores. Each question typically attracted between 40 and 300 forecasters, with a median around 60. High-profile questions on elections or pandemics drew several thousand. So, how bad are we exactly?
Between 2015 and 2024, the volume of predictions on Metaculus grew steadily, then exploded. In 2020, a global pandemic made the future feel both more urgent and more unknowable than ever. People wanted answers, so they made guesses.
Notice how volume stayed high even after COVID faded. Once we start trying to predict, it's hard to stop.
We grouped all 2,129 questions into 10 topics using machine learning with no manual tagging. Each ridge below shows how much attention a topic attracted, year by year.
Some patterns are hard to miss. COVID-19 went from nothing to the most predicted topic on the platform in a single year, then collapsed almost as fast as it arrived. Russia / Ukraine barely registered until February 2022, then exploded overnight. World events don't build gradually they arrive, and forecasters chase them.
Then there's US Politics. No pandemic, no war, no once-in-a-generation crisis ever knocked it off the top. Year after year, it pulled in more forecasters than anything else on the planet. When something is chaotic enough to keep thousands of analysts guessing indefinitely, that says more about the subject than the forecasters.
For the 1,665 yes/no questions, we scored each one simply: if the crowd's confidence was above 50% and the outcome matched, right. Otherwise, wrong.
Out of 1,665 predictions: 537 right, 1,128 wrong. A coin flip would have done a better job.
Breaking it down by topic reveals clear patterns. Geopolitics and Russia / Ukraine were brutal. The crowd kept expecting conflicts and crises that quietly didn't happen.
AI / Technology was the exception. The tech community was more calibrated, possibly because they actually understood what they were predicting.
Each dot is a prediction. Teal means right. Red means wrong. They're spread along a confidence axis: low on the left, high on the right.
High confidence should predict accuracy. Here's what actually happened: red everywhere, at every confidence level. The crowd was confidently wrong just as often as it was uncertainly wrong.ma
The most embarrassing failures: questions where forecasters were most certain something would happen. And it didn't. All carried the highest confidence score in the dataset.
Across 2,129 predictions, the crowd was wrong 68% of the time. But the sharper finding is in the confidence data: being highly confident didn't improve accuracy at all. The crowd was wrong at the same rate whether it was certain or uncertain. That's not bad luck. That's overconfidence, built in.
Geopolitics was the hardest domain by a distance. Global conflicts don't telegraph themselves, they arrive suddenly, upend everything, and resist the kind of gradual modelling that forecasters are good at. The crowd kept expecting crises that quietly didn't happen, and missed the ones that did.
And then there's US Politics, the single most predicted topic by volume, every year, without exception. Not because it's easy to call, but its volatility is magnetic. When something is chaotic enough to keep thousands of analysts guessing indefinitely, that says as much about the subject as it does about the crowd chasing it.
We know we're not great at predictions. And yet we'll keep making them, because looking ahead is crucial for who we are.