Innovation is all about coming up with new solutions to solve problems.

But here’s an interesting question: is the problem that you’re trying to solve a puzzle or a mystery?

The distinction was made by Gregory Treverton and highlighted by Malcolm Gladwell in a piece he wrote on Enron a few years ago.

According to Treverton, a puzzle is a problem that can be solved if you have more information (or the right information). On the other hand, more information doesn’t help with a mystery, which is characterised by high levels of uncertainty, and the need for judgement. Here’s Gladwell:

The national-security expert Gregory Treverton has famously made a distinction between puzzles and mysteries. Osama bin Laden’s whereabouts are a puzzle. We can’t find him because we don’t have enough information. The key to the puzzle will probably come from someone close to bin Laden, and until we can find that source bin Laden will remain at large.

The problem of what would happen in Iraq after the toppling of Saddam Hussein was, by contrast, a mystery. It wasn’t a question that had a simple, factual answer. Mysteries require judgments and the assessment of uncertainty, and the hard part is not that we have too little information but that we have too much. The C.I.A. had a position on what a post-invasion Iraq would look like, and so did the Pentagon and the State Department and Colin Powell and Dick Cheney and any number of political scientists and journalists and think-tank fellows. For that matter, so did every cabdriver in Baghdad.

The distinction is not trivial…

If things go wrong with a puzzle, identifying the culprit is easy: it’s the person who withheld information. Mysteries, though, are a lot murkier: sometimes the information we’ve been given is inadequate, and sometimes we aren’t very smart about making sense of what we’ve been given, and sometimes the question itself cannot be answered. Puzzles come to satisfying conclusions. Mysteries often don’t.

Puzzles are attractive because, as Gladwell points out, they come to clean conclusions. Ironically, by these definitions, all of the Agatha Christie books are puzzles, not mysteries – they can always be solved if you just pay attention to the right information, which is all there for you.

We are strongly drawn to puzzles because of how clear-cut they are.

Unfortunately, many of the big problems that we face are not puzzles, but rather mysteries. Mysteries are messy, and the methods that solve puzzles don’t work for mysteries, and they might actually make them worse.

Jeanne Liedtka and Tim Ogilvie pick up on this distinction in their outstanding book Designing for Growth: A Design Thinking Toolkit for Managers.

They say that incremental innovations are puzzles. The parameters are basically known, we just need to find the right information to develop the innovation that will solve the problem. But then:

There’s another category of problem called mysteries, where there is no single piece of data, there is no level of data disclosure that will actually solve a problem. In fact, there might be too much data and it’s about interpreting all the data that’s there. And that’s a richer, harder problem that requires more systems thinking, that requires prototyping and piloting. That’s really where the designers are often most adept.

Their contention is that the high levels of uncertainty in mysteries requires a different, more experimental approach. Their solution to this is design.

Third, design is tailored to dealing with uncertainty, and business’s obsession with analysis is best suited for a stable and predictable world. That’s the kind we don’t live in anymore. The world that used to give us puzzles but now dishes up mysteries. And no amount of data about yesterday will solve the mystery of tomorrow. Yet, as we’ve already noted, large organizations are designed for stability and control, and are full of people with veto power over new ideas and initiatives. They are the “designated doubters.” The few who are allowed to try something new are expected to show the data to “prove” their answer and get implementation right the first time.

The bulk of the book is taken up with describing tools and processes that you can use to implement design thinking. This is how they picture the process:

If you look at this model, it maps onto the idea management process model that we have discussed here on numerous occasions.

Their model is based around four questions. The first – What Is? – is the place for problem definition.

The next step is asking “What If?” This is idea generation. And it’s the same question that Grant McCracken identifies is critical in his book Culturematic – discussed here. This is the divergent step in the process.

Question three is “What Wows?” This is the idea selection step. Liedtka and Ogilvy outline an method for assumption testing and rapid prototyping here. In other words, experiments.

The final question is “What Works?” This is the execution and diffusion phase of the process. This is where you co-develop with your customers to converge on a solution.

The approach in Designing for Growth is sound. It is a very practical book, with clear instructions on how to implement design thinking in your innovation process, and with plenty of examples and case studies to make the ideas real.

The problems that lead to disruptive innovations are often mysteries. This means that we need a different toolkit to solve these problems than we use when we solve puzzles. Experimentation and design thinking are two excellent approaches to use when facing a mystery.

Which kind of problem do you face right now?