Tim and I get to talk to many managers as part of our research and consulting work. One of the good things about doing this is that we get to see common success factors and inhibitors of innovation.
A particular tension that exists in nearly all of the organisations that we see is between the desire to be innovative and the need to have formalised plans.

Plans are usually based upon projections of the future. The trouble with these projections is that innovations are notoriously hard to predict. This is especially true for more radical and disruptive innovations. The Nobel prize-winning economist Kenneth Arrow summed this up nicely by saying that if an innovation is truly an innovation, then by definition it is impossible to predict.

So, innovations are hard to predict and therefore plan, but to make matters worse we also know that there is something called the “planning fallacy”. This is based upon findings about psychological biases in predictions and decision making. Despite all the planning that goes on in the world, experimental psychologists tell us that we are naturally bad planners for several reasons.

One of these is that we tend to be overconfident about predictions and we use a limited amount of information to make these predictions. And the more we know, the more confident we become and therefore more prone to failure. Tversky and Kahneman have become famous for their work in behavioral economics (Kahneman won the Nobel prize in 2002) but if you want to read a good book on the topic of overconfidence and failure I can recommend Ego Check, written by Mat Hayward who is currently professor in strategy at the UQ Business School.

From this, it would seem that planning is the enemy of innovation. However, research that I am doing with Mat suggests that it is the type of planning that matters more to innovation performance. Based on survey data collected by Andrew Caldwell in his honours year at UQ Business School, we looked at how Australian biotechnology executives developed business plans for innovative products and services. The results from this survey were twofold. Firstly, it is possible to discern two categories of planners. One group attempted to base plans on formalised predictions of future costs and earnings. The other group relied more on being able to stage investments and try things out, without getting bogged down in predictions of profitability. Secondly, when we correlated these planning styles with innovation outputs (measured by patents) it was the second group that was more innovative, while we could actually show a negative correlation between formal planning and innovation.

Planning isn’t necessarily anti-innovation, but planning for innovation means adopting planning routines that involve a portfolio of experiments and the capacity to scale-up successful trials and quickly kill initiatives that aren’t showing signs of success. While there may be several methods for doing this type of planning, the most common and well known is stage-gate. It’s most used for products, but I can’t see why it can’t be used for a whole range of innovations.