I’ve been running Lean LaunchPad programs for a couple of years now, and all along, I’ve thought that the number of customer development interviews that a team does is a good indicator of how successful they’ll be. And when we measure interview numbers against progress on the Investment Readiness Levels, the correlation is very strongly positive.

Consequently, I thought that the magic in lean startup is customer development. I was wrong.

In our work at Era Innovation we had an experience that invalidated this hypothesis.

We’re doing some work for a corporate client and we’ve got a team (that’s doing ALL of the work and hard thinking on this) that has gone through two LLP iterations looking at two different problems.

The first one was textbook lean startup. We started with a business model canvas, worked out the hypotheses, and the team started interviewing. After more than 130 customer development interviews, they had made two major pivots. They supported the final business model idea with a couple of minimum viable product tests, and everyone was happy, including the clients.

The second problem was trickier. Because the issues seemed so broad, the team launched straight into customer development interviews – and we expected the insights about the best business model to emerge from that data. It turns out that this is about a billion times tougher than we expected.

The difference is that the second time around, we didn’t test hypotheses right from the start.

Ash Maurya wrote a great post about this very issue this week (and he also addresses it in his excellent new book Scaling Lean).

He says:

We start by guessing a new law or a new theory. Then we compute the consequences of our guess. Finally, we compare those computations with experiments or observed experiences. According to Richard Feynman, this simple statement holds the key to science: “If your guess disagrees with experiment, it is wrong. It doesn’t matter how beautiful your guess is, how smart you are, who made the guess, or what their name is … it is wrong.”

Here’s his diagram:

And here’s how he frames it for business:

Innovation experiments are no different. Achieving breakthrough, then, is less about luck and more about a rigorous search. The reason the hockey-stick trajectory has a long at portion in the beginning is not because the founders are lazy and not working hard, but because before you can find a business model that works, you have to go through lots of stuff that doesn’t.

Breakthrough insights are often hidden within failed experiments.

Tina Seelig also talks about this in her book inGenius:

Trained scientists know this well and, therefore, do their best to design experiments that answer an important question, no matter what the specific results. They know that each experiment offers valuable clues on the path to understanding. As the saying goes, “Genius is the ability to make the most mistakes in the shortest period of time.” Each of those mistakes provides experimental data and an opportunity to learn something new. Like scientists, we need to stop looking at unexpected results as failures. By changing our vocabulary, by looking at “failures” as “data,” we enhance everyone’s willingness to experiment. That is a big idea!

The data on lean startup hypothesis is that nearly 70% of them are invalidated. That’s a lot of learning!

And we learn best when we test specific hypotheses and business models. That is why, as Steve Blank says, Build-Measure-Learn isn’t just throwing things agains the wall to see what sticks. That’s a waste of resources, and definitely not lean!

Instead, if we are rigorous in our approach, our pivots will be evidence-based.

In a guest post for Pollenizer, Alistair Croll says:

Adopting a “pivot” mentality isn’t an excuse to randomly walk all over a market. As Eric Ries has explained, a pivot is an adjustment, akin to keeping one foot on ground as you rotate gradually. It’s iterative, honing in on the right product for the right market until it just clicks. Pivoting isn’t hopping.

…

Of course, this sounds like work. A corollary of not pivoting without knowing why is that everything you do must produce learnings. That means interviewing customers—and those who didn’t buy what you were selling. It means building experimentation into your product. It means analyzing results using metrics that drive your business plan, rather than meaningless vanity metrics.

Which brings us back to customer development interviews. It is an essential lean startup tool – you can’t succeed without them.

But we must use them to test hypotheses. Data without hypotheses is much less magical.

The magic in lean startup is hypothesis testing.

Note: Two tools that you can use to track and test your hypotheses are the Lean Startup Progress Board from Strategyzer and all of the great tools from Pollenizer.