Last night was the Australian federal election and I think it was probably the most gripping and wackiest night in Australian politics that I’ve ever seen. Late last year it seemed impossible that the conservative coalition parties would be able to win government and the current conservative leader almost got the job by accident. Several weeks ago the Prime Minister, Kevin Rudd, was dumped by his party after being one of the most popular Australian PMs in history only 12 months ago. With the new PM, Julia Gillard in place (the first woman to be an Australian PM), the government called a snap election and the book-makers had the government on short odds (about 1.3:1) to retain power, while the opposition were out at about 2.5:1.

The betting was interesting to follow because it has previously been one of the most accurate predictors of the election outcome. With lots of people placing bets, and those with inside information on poll trends placing big bets, it usually becomes a very accurate market for information. In the previous election it had been more accurate than any of the research polls. A week before the election the odds on a conservative victory had blown out to 3:1 and a victory by the incumbent government looked inevitable, even though the polls were saying that the election result would be very close.

All the usual signs pointed to a government win. Very few Australian governments are given just one term in office, the economy is doing very well and unemployment is low and the two seats that had gone with the winning party in every election looked safe for the government. What happened last night was a night of many political firsts, in addition to all of the political chaos leading up to the election.

By early in the evening it was clear that some sort of swing to the coalition was going to threaten the government. In an electorate north of Brisbane, a 20 year old was elected to become the youngest ever parliamentarian. Later in the evening the Green party claimed their first ever seat in the house of representatives and then a former secret service officer, running as an independent, won a seat that used to be assumed to be rusted-on to the governing party. By the end of the night it was apparent that the most likely outcome would be the first hung parliament since the second world war. A bizarre night for sure, but why had the betting market gotten it so wrong?

I think that the market for information in the election failed for the same reason that experts miss disruptive innovation. When we are faced with radical changes in the environment where the conventional operating rules are being broken, the usual informational cues can’t help us. SInce the majority of us are using the same cues we collectively miss what is really going on. There is a much better discussion of the psychology surrounding this issue written by Nasim Tayeb in his book, Black Swans. Markets probably work well in stable environments when we know what information we should be paying attention to in order to make decisions about what price to pay for a bet, a stock, or an innovation. After last night I’m a lot more skeptical about the ability of crowds and markets to make better decisions than individual experts when confronted with radical change.