Experts Miss Disruptive Innovations and Markets and Crowds Do Too.

I wrote a post last week about why experts will often overlook disruptive innovations. Today I’m going to extend this argument by saying that crowds and markets are just as good at missing these step changes, but first I’ll talk about the inspiration for the post.

Last night was the Australian federal election and I think it was probably the most gripping and wackiest night in Australian politics that I’ve ever seen. Late last year it seemed impossible that the conservative coalition parties would be able to win government and the current conservative leader almost got the job by accident. Several weeks ago the Prime Minister, Kevin Rudd, was dumped by his party after being one of the most popular Australian PMs in history only 12 months ago. With the new PM, Julia Gillard in place (the first woman to be an Australian PM), the government called a snap election and the book-makers had the government on short odds (about 1.3:1) to retain power, while the opposition were out at about 2.5:1.

The betting was interesting to follow because it has previously been one of the most accurate predictors of the election outcome. With lots of people placing bets, and those with inside information on poll trends placing big bets, it usually becomes a very accurate market for information. In the previous election it had been more accurate than any of the research polls. A week before the election the odds on a conservative victory had blown out to 3:1 and a victory by the incumbent government looked inevitable, even though the polls were saying that the election result would be very close.

All the usual signs pointed to a government win. Very few Australian governments are given just one term in office, the economy is doing very well and unemployment is low and the two seats that had gone with the winning party in every election looked safe for the government. What happened last night was a night of many political firsts, in addition to all of the political chaos leading up to the election.

By early in the evening it was clear that some sort of swing to the coalition was going to threaten the government. In an electorate north of Brisbane, a 20 year old was elected to become the youngest ever parliamentarian. Later in the evening the Green party claimed their first ever seat in the house of representatives and then a former secret service officer, running as an independent, won a seat that used to be assumed to be rusted-on to the governing party. By the end of the night it was apparent that the most likely outcome would be the first hung parliament since the second world war. A bizarre night for sure, but why had the betting market gotten it so wrong?

I think that the market for information in the election failed for the same reason that experts miss disruptive innovation. When we are faced with radical changes in the environment where the conventional operating rules are being broken, the usual informational cues can’t help us. SInce the majority of us are using the same cues we collectively miss what is really going on. There is a much better discussion of the psychology surrounding this issue written by Nasim Tayeb in his book, Black Swans. Markets probably work well in stable environments when we know what information we should be paying attention to in order to make decisions about what price to pay for a bet, a stock, or an innovation. After last night I’m a lot more skeptical about the ability of crowds and markets to make better decisions than individual experts when confronted with radical change.

Please note: I reserve the right to delete comments that are offensive or off-topic.

9 thoughts on “Experts Miss Disruptive Innovations and Markets and Crowds Do Too.

  1. Thanks for this interesting post, John.

    I think, your conclusion fits quite well to Tim’s summary on the 3 innovation horizons:
    http://timkastelle.org/blog/2010/08/innovation-for-now-and-for-the-future/

    It outlines that different approaches are required along the ‘innovation continuum’. According to the model, crowds and customers are more likely to identify incremental innovation, rather than ‘breakthroughs’. It reminds me of a similar discussion about customer-driven vs. proposal- based innovation, that arised in connection with Verganti’s “Design-Driven Innovation” approach.

    I personally think that many innovation issues in companies result from taking approaches, not being aligned with their actual obectives. The common processes / organizations are mostly oriented to incremental innovation / improvements, as they account for the majority of innovation efforts.

    Another critical question for me is: how to deal with experts in relation to more radical innovation? On the one hand, specialists are clearly required to tap new markets and technologies. On the other hand, you have indicated in your former post that “expertise is valuable, but it also comes with a cost in terms of existing commitments to old ideas”:
    http://timkastelle.org/blog/2010/08/why-do-we-miss-disruptive-innovations/

    Overall, the role of experts seems to be of high relevance for an optimized innovation approach -and tends to be different for the three innovation horizons.

    Regards, Ralph
    @ralph_ohr

  2. Hi Ralph

    this is really insightful and your last comment has got me thinking about the way to use experts without getting trapped in the dominant logic. Thanks for the comment!

    Regards

    John

  3. There is a physical phenomenon known as “retinal fatigue” – when you stare at something long enough and then look at a blank wall, you see a (sometimes blurred) colored image of what you were just staring at. Extrapolate this condition to how we view business conditions, politics, etc….explains a lot!

  4. John, I agree with your overall conclusion about experts and markets often missing disruptive innovations. Agree there is a tendency also for “crowds”

    It was my understanding that markets were never really examples of distributed cognition/wisdom of crowds because individuals could observe what others thought and hence be influenced by it. i.e. “true” wisdom of crowds is in situations where the individual decision is secret and then aggregated.

    I also understood distributed cognition to be most effective when it was aggregating the perspectives of experts, not everyone. e.g. getting a group of graziers to guess the weight of a steer.

    This means that the betting market of the election isn’t a great example of distributed cognition. It’s public, it doesn’t include only experts on the election, and then there’s the added pressure that the odds (what is public) is also influenced by the bookies’ needs to maximise their profit.

    I do wonder what would have happened if, late on Saturday afternoon, you asked some form of expert (e.g. party members manning booths with over 20 years experience, or individual reporters following the election with similar length of experience; or better yet, a collection of Anthony Greens) what they thought the outcome would be?

  5. Hi Deb:
    I think you are right. It’s an irony of the information economy that information is abundant but our basic psychological capacity to process it hasn’t changed for thousands of years.

Comments are closed.