One of the fascinating aspects of the U.S. election last week was the emergence of the poll aggregators. These were the people that used stats to analyse all of the polls to make predictions about the races. Nate Silver ended up becoming the poster boy for this group when he took a lot of heat in the week before the election from people who didn’t like his predictions.
His predictions which, as they did in 2008, ended up being remarkably accurate. His analysis called all 50 states correctly in the Presidential election, and he was very close to getting the popular vote right as well. His senate calls were also unbelievably good.
Silver explains his approach in his excellent book The Signal and the Noise: Why so Many Predictions Fail, But Some Don’t. In that, he talks about a number of different ways that people try to model the world, with cases looking at the stock markets, online poker, earthquakes, baseball careers, medicine, and, yes, elections.
His approach is Bayesian. This means that you start with a prediction about how an event will turn out, and then modify that prediction as new data arrives. Throughout the process, you get the likelihoods of the outcomes occurring expressed as a percentage. Silver doesn’t show his exact method, but he explains all of the variables that his model accounts for on his Methodology page. Here is what he says about the approach in his book:
Bayes’s theorem says we should update our forecasts any time we are presented with new information. A less literal version of this idea is simply trial and error. Companies that really “get” Big Data, like Google, aren’t spending a lot of time in model land.* They’re running thousands of experiments every year and testing their ideas on real customers. Bayes’s theorem encourages us to be disciplined about how we weigh new information. If our ideas are worthwhile, we ought to be willing to test them by establishing falsifiable hypotheses and subjecting them to a prediction. Most of the time, we do not appreciate how noisy the data is, and so our bias is to place too much weight on the newest data point. Political reporters often forget that there is a margin of error when polls are reported, and financial reporters don’t always do a good job of conveying how imprecise most economic statistics are. It’s often the outliers that make the news.
Here are what Silver’s forecasts looked like throughout the year:
Mark Coddington has a great summary of the discussion of what this means for politics, and Greg Satell tells us what it means for the pundits who got their election predictions so wrong (it’s not good!). While I love the advance that this level of analysis represents, I’m more interested in what will make it go wrong.
Models of financial markets blow up when we get events that are assumed to be extremely unlikely by the models. What would make Nate Silver’s model blow up?
These sophisticated models will blow up when the polls that they rely on for data all have systematic errors built into them. This is what the people criticising Silver were arguing – that the polls were not using the correct distribution of voters, hence unskewedpolls.com.
Contrary to some commentary, this approach wasn’t wrong, or unscientific. It was actually based on a bet – that the distribution of voters would look more like what we had in 2004 than like 2008. That’s not hugely unreasonable. An incorrect distribution of voters polled is what caused the biggest poll screwup in history when the Literary Review predicted that Alf Landon would get 57% of the vote against Franklin Roosevelt. They had predicted the previous five presidential elections correctly.
In reality, Roosevelt won 62% of the vote that election. How did the Literary Review get it so wrong? Their sample was based on telephone directories, magazine subscription lists, and club memberships. The problem is that in 1936, all of those groups tended to be wealthier, so they were much more likely to vote for Landon. The sample was wrong.
The irony of unskewedpolls.com is that Silver’s method actually corrects for potential skew – that is where a lot of the uncertainty comes from in his estimate. When he said that Obama had an 84% chance of winning going into the last week, it meant that in election simulations based on previous outcomes, 16% of the time there was enough variance in the poll data that Romney would have won. The unskewing is built into the simulations.
What could cause all of the polls to be so wrong that this model will break? It will happen when someone comes up with a major innovation in campaigning. One way to do this is to use Clayton Christensen’s disruption theory – which says that one way to disrupt in an industry is to target non-consumers (see this pdf from Business Innovation Factory).
Check out this graph from Andrew Gelman (whose book Red State Blue State is another must-read on the statistics of elections):
The number of eligible voters that didn’t vote at all last week was about 40%. That’s a LOT of non-consumers. If their participation rate were to spike, and if they tended to vote differently than everyone currently voting, it would be unprecedented.
This is the sort of thing that could cause all of the polls to be systematically wrong. And if the polls are systematically wrong in a way that the models don’t anticipate, the aggregating models will also be wrong. That’s how Nate Silver’s model could blow up.
Of course, this is not a strategy that you could execute by starting a couple of months before the election when your candidate is already in trouble. It would take a fair bit of planning. But this is the sort of thing that could lead to a major disruption in U.S. politics.
This type of modelling works really well as long as the underlying dynamics of a system remain relatively stable. They don’t work as well when something new happens.
Political innovation is one thing that can cause such a disruption. If you want to develop an effective third party in the U.S., or jump-start one that’s not doing so well, I’d be looking at the current non-consumers of voting. There sure are a lot of them.
Photo of Nate Silver by JD Lasica used under a Creative Commons license.
Related articles
In a sense, Obama has brought in groups that have traditionally been non-consumers (non-voters), like young people, hispanics, and even lower-income women. He would not have won either election without the innovation of reaching out to these constituencies through social media, and probably more important, by combining these technologies with on-the-ground, community organizing strategies he learned early on in his career.
Still, 40% of the people did not vote.
Maybe if we have them spend a night in jail …
BTW, Alan Lichtman, a historian, has predicted the last eight national elections, starting in 1984. Here’s an interaction between him and Silver, http://j.mp/TF6z6f.
The Obama ground game is definitely a significant innovation Matt – I agree. I was going to talk about that too, but the post had already gotten longer than I wanted. I’ve run across Lichtman’s work in a couple of different places. My suspicion is that his would blow up too in the face of a disruptive innovation, but I’m not sure that we’ll ever see one…