We ask people to predict all sorts of things at CivicScience. Many are strictly related to topics our clients are interested in, like which food trends they think will take off over the next year. Those topics are interesting to analyze with respect to how sentiment evolves over time, but sometimes it’s nice to evaluate how our respondents did right away, like when we ask them to predict who will win NFL playoff games. Can we identify people who are better than average at predicting NFL playoff games? For every playoff game that has been played so far, we asked about 5,000 people to pick a winner.
In identifying good predictors, one important thing to consider is how easy it is for people to predict who wins. The chart below shows the percentage of respondents who correctly predicted the winner for each game. (Respondents who said they weren’t sure who’d win were omitted.)
Generally speaking, our respondents did a good job of predicting the winners. Nearly 80% of respondents predicted that the Chiefs would defeat the Texans in the wild card round, and they ended up winning 30-0, easily the most lopsided game of the eight.
The only game that fewer than 50% of our respondents got right was Chiefs-Patriots from the divisional round. Given that the Patriots were clear favorites and have been the dominant franchise in the NFL for fifteen years now, it might be surprising that more people picked them to lose than win. I think the underlying explanation is that the Patriots are widely disliked. In 2012 we found a sizable minority of respondents who said they thought Mitt Romney would carry California, or that Barack Obama would carry Texas. Even when you ask people what they think will happen, many still tell you what they want to happen.
Chart 2 shows a scatter plot of how individual respondents performed. The points are jittered to give an idea of how many people there are at each position.
Since the rectangles in the bottom left are pretty much completely filled in, you can see that most respondents who made a prediction only made one or two. And if you ever doubted why the Las Vegas casinos have so much money, see how only thirteen out of 356 respondents who made eight predictions got all eight games right — and that’s without considering the point spread.
So what can we learn about respondents who performed well over the past two weekends? First we have to define who the good predictors are. By adding up the percentage of respondents who got each game right, and looking at which questions each individual respondent answered, we can calculate how many games each respondent should have gotten right relative to the average.
For example, if someone made predictions about Chiefs-Texans (78% got it right), Packers-Redskins (62%), and Seahawks-Panthers (60%), their expected correct predictions would have been 0.78 + 0.62 + 0.60 = 2. So if that person went three for three, they would have beaten their expected number. I classified all respondents who beat their expected number as NFL playoff “sharps,” provided they made at least two predictions. (After all, anyone can make one lucky guess.) Those who failed to beat their expected average were classified as “squares.”
That leaves us with 1,507 sharps. Across all eight games, these respondents were correct 82% of the time. Having used the Apriori algorithm to identify which attributes are most strongly associated with sharps, I came up with three broad sets of attributes that define these skilled predictors.
Domain expertise
In other words, sharps are big sports fans. This showed up across a number of attributes, even non-football ones like being a fan of the NBA or the NHL. Super obvious? Yeah, probably. Let’s move on.
Luxury car fans
People who told us they love luxury car brands such as Audi, Land Rover, Mercedes, and Lexus proved to be better predictors than average. Below is a chart that shows the sharps vs. squares breakdown for people who answered our sentiment tracking question about Audi.
As for the official car sponsor of the NFL, only 43% of people who said they love Hyundai were sharps.
Frequent grocery shoppers
Here’s one I didn’t expect to see. A number of grocery-related attributes popped. Basically, if you shop frequently at big chain stores (grocery specific or more general), you’re more likely to be a sharp. Perhaps this is a proxy for football fans who buy lots of food for pre-game tailgates.
Did we find anything earth-shattering here? Probably not. Higher-end, engaged consumers are more likely to follow sports and, therefore, to be able to predict sports outcomes. Indeed, one way to lower your income is to be bad at betting on sports.
But here’s where it can get very interesting. Think about the fact that at CivicScience, we are surveying millions of people every month, who we know a lot about from their response history. Over time, we can find more and more people who are good at predicting sports (and other) outcomes. With a large enough group of those good predictors at our disposal, imagine what we can do as we get smarter about it.
