Tim Kastelle

Tag: hypothesis testing

  • Those That Get It Don’t Need It, and Those That Need It Don’t Get It

    Those That Get It Don’t Need It, and Those That Need It Don’t Get It

    Here’s a central problem with trying to get any new idea to spread – often, those that get it don’t need it, while those that need it don’t get it.

    It’s a paradox.

    This leads to problems for people that have new ideas.

    Problem 1a: you end up talking to the wrong people. It is easiest to talk to the people that get it – even though they don’t need your idea. Back in my startup days, we often went to First Tuesday in Brisbane to try to build our network. We’d talk to all the other local startups about the problems that we shared, and it was great – they really got it!

    Did it help us build our business? No. No, it did not.

    We were talking to the people that were easy to talk to – the ones that got it. But they didn’t need what we had built. To grow a business we had to talk to the people that needed our ideas. That was a lot harder. It was frustrating, and difficult. Mainly because:

    Problem 1b: you might be solving a problem that people don’t yet realise they have. This problem is the opposite of the first one. This makes it really hard to talk to them, because when they hear your idea, they’ll hate it.

    When this happens a lot, you’re in what Seth Godin calls the Gulf of Disapproval:

    Here’s what he says:

    Start at the left. Your new idea, your proposal to the company, your new venture, your innovation—no one knows about it.

    As you begin to promote it, most of the people (the red line) who hear about it don’t get it. They think it’s a risky scheme, a solution to a problem no one has or that it’s too expensive. Or some combination of the three.

    They need your idea, but they don’t get it.

    The key to solving this paradox is to find the small number of people that will get it and that need it. Even for huge breakthrough ideas, this original group is usually pretty small.

    These people are the blue line in Godin’s drawing.

    Here is how Steve Blank describes them:

    Earlyvangelists are a special breed of customers willing to take a risk on your startup’s product or service. They can actually envision its potential to solve a critical and immediate problem—and they have the budget to purchase it. Unfortunately, most customers don’t fit this profile.

    Earlyvangelists can be identified by these characteristics:

    • They have a problem.
    • They understand they have a problem.
    • They are actively searching for a solution and has a timetable for finding it.
    • The problem is painful enough that they have cobbled together an interim solution.
    • They have, or can quickly acquire, dollars to purchase the product to solve their problem.

     

    How do we find these people? We start by building a model of who we think they are, and what we think they need. This model is almost certainly wrong. We fix that by going out and talking to these people to learn about the problems that they are actively trying to solve.

    Because we’re trying to identify problems, we’re not pitching during these conversations. We’re learning. If we do that enough times, we’ll figure out what a small group of people really need right now, and, with luck, we can build it for them.

    People usually can’t explain what they need, especially if the idea is genuinely new. So you need to look for evidence of problems. In my experience, the sign that we’re really onto something is in Blank’s fourth point – when we find people that have already hacked together a solution of their own. This is strong evidence.

    It turns out that our two groups of people aren’t mutually exclusive – there’s a small overlap:

    When we have a new idea, our job is to figure out what these people need, who they are, and how to find them. Once we’ve done this, then more people will start to get the idea, and more people will also start to need it.

    That’s the only way to cross the Gulf of Disapproval.

  • The Magic in Lean Startup is Hypothesis Testing

    The Magic in Lean Startup is Hypothesis Testing

    I’ve been running Lean LaunchPad programs for a couple of years now, and all along, I’ve thought that the number of customer development interviews that a team does is a good indicator of how successful they’ll be. And when we measure interview numbers against progress on the Investment Readiness Levels, the correlation is very strongly positive.

    Consequently, I thought that the magic in lean startup is customer development. I was wrong.

    In our work at Era Innovation we had an experience that invalidated this hypothesis.

    We’re doing some work for a corporate client and we’ve got a team (that’s doing ALL of the work and hard thinking on this) that has gone through two LLP iterations looking at two different problems.

    The first one was textbook lean startup. We started with a business model canvas, worked out the hypotheses, and the team started interviewing. After more than 130 customer development interviews, they had made two major pivots. They supported the final business model idea with a couple of minimum viable product tests, and everyone was happy, including the clients.

    The second problem was trickier. Because the issues seemed so broad, the team launched straight into customer development interviews – and we expected the insights about the best business model to emerge from that data. It turns out that this is about a billion times tougher than we expected.

    The difference is that the second time around, we didn’t test hypotheses right from the start.

    Ash Maurya wrote a great post about this very issue this week (and he also addresses it in his excellent new book Scaling Lean).

    He says:

    We start by guessing a new law or a new theory. Then we compute the consequences of our guess. Finally, we compare those computations with experiments or observed experiences. According to Richard Feynman, this simple statement holds the key to science: “If your guess disagrees with experiment, it is wrong. It doesn’t matter how beautiful your guess is, how smart you are, who made the guess, or what their name is … it is wrong.”

    Here’s his diagram:

    innovation experiments

    And here’s how he frames it for business:

    Innovation experiments are no different. Achieving breakthrough, then, is less about luck and more about a rigorous search. The reason the hockey-stick trajectory has a long at portion in the beginning is not because the founders are lazy and not working hard, but because before you can find a business model that works, you have to go through lots of stuff that doesn’t.

    Breakthrough insights are often hidden within failed experiments.

    Tina Seelig also talks about this in her book inGenius:

    Trained scientists know this well and, therefore, do their best to design experiments that answer an important question, no matter what the specific results. They know that each experiment offers valuable clues on the path to understanding. As the saying goes, “Genius is the ability to make the most mistakes in the shortest period of time.” Each of those mistakes provides experimental data and an opportunity to learn something new. Like scientists, we need to stop looking at unexpected results as failures. By changing our vocabulary, by looking at “failures” as “data,” we enhance everyone’s willingness to experiment. That is a big idea!

    The data on lean startup hypothesis is that nearly 70% of them are invalidated. That’s a lot of learning!

    And we learn best when we test specific hypotheses and business models. That is why, as Steve Blank says, Build-Measure-Learn isn’t just throwing things agains the wall to see what sticks. That’s a waste of resources, and definitely not lean!

    Instead, if we are rigorous in our approach, our pivots will be evidence-based.

    In a guest post for Pollenizer, Alistair Croll says:

    Adopting a “pivot” mentality isn’t an excuse to randomly walk all over a market. As Eric Ries has explained, a pivot is an adjustment, akin to keeping one foot on ground as you rotate gradually. It’s iterative, honing in on the right product for the right market until it just clicks. Pivoting isn’t hopping.

    …

    Of course, this sounds like work. A corollary of not pivoting without knowing why is that everything you do must produce learnings. That means interviewing customers—and those who didn’t buy what you were selling. It means building experimentation into your product. It means analyzing results using metrics that drive your business plan, rather than meaningless vanity metrics.

    Which brings us back to customer development interviews. It is an essential lean startup tool – you can’t succeed without them.

    But we must use them to test hypotheses. Data without hypotheses is much less magical.

    The magic in lean startup is hypothesis testing.

    Note: Two tools that you can use to track and test your hypotheses are the Lean Startup Progress Board from Strategyzer and all of the great tools from Pollenizer.

     

  • How to Make Good Lean Startup Hypotheses

    How to Make Good Lean Startup Hypotheses

    When teams start out with lean startup, they often build hypotheses that are too precise – we assume we know more than we do. Imagine that we’re trying to build a startup called ScotchFinder – it’s like AirBnB, but for Scotch! Our first hypothesis often looks like this:

    We believe that: people will be happy to share their rare scotch with others.

    We will test this by: interviewing 30 people outside a liquor store.

    We are right if: more than 70% or people say they like the idea of ScotchFinder.

    Obviously, it’s better to start with this than it is to go into interviews blindly. However, this hypothesis/test combination can be improved in several ways.

    First off, who are the “people?” We often try to approach interviews as though we are looking for a representative sample of the population. This is especially true for scientists, as this is how we’re trained to do surveys.

    But that’s not the best way to do customer development. Remember, we’re looking for our first market segment – a specific group, with a specific problem that we can solve. Once we zero in on this group, our sample might end up being very unrepresentative, and that’s fine.

    The second problem: what kind of liquor store? Are trying to make something for people that already buy premium scotch? Then we’ll want to do our surveys outside a store with lots of good Scotch. What if we’re trying to get people to start drinking Scotch? Then we might be better off doing interviews at places selling lots of beer. What if we’re aiming to get people that normally drink Scotch in bars to start drinking it at home? Maybe we need ScotchFinder drones!

    ScotchFinder drone!

    For each of these three segments, we need to find people in different places, because they’re all living their lives differently. ScotchFinder may eventually serve all three groups of people – but right now, we need the group that is dissatisfied with what they’ve currently got.

    The final problems are with the last question. If we set up the hypothesis like this, then we’ll spend too much of our interview time pitching ScotchFinder, and not enough learning about what problems people are running into when they try to drink Scotch.

    Finally, again, this isn’t a poll. The majority doesn’t rule. We’re looking for a market segment that we can absolutely dominate. What if we do our interviews (or do a survey), and 97% of people say that they (correctly) think that ScotchFinder is a stupid idea. Then it’s dead, right?

    Not necessarily. What if the 3% that do like ScotchFinder are all the same? What if they’re all well educated, double-income no kids couples, living in the same part of town, and for some reason, that part of town has no good liquor stores? The ScotchFinder Drone may well work! And if it does, this might be a segment that we can dominate, which is a great way to start out.

    In our early interviews, we need to set the bar pretty low for our hypotheses. We don’t know enough yet to use percentages, as though our 30 interviews are the same as a mini-survey. We’re actually looking for themes.

    This is what Eric Ries says about this in his upcoming book The Leader’s Guide:

     

    Don’t bog new teams down with too much information about falsifiable hypotheses, and try to resist the urge to critique teams’ hypotheses when they are early in their Lean Startup journey. Because if we load our teams up with too much theory, they can easily get stuck in analysis paralysis. I’ve worked with teams that have come up with hundreds of leap-of-faith assumptions—page after page after page in a tiny spreadsheet of all the things that had to be true for the project to take effect. They listed so many assumptions that were so detailed and complicated that they couldn’t decide what to do next. They were paralyzed by the just sheer quantity of the list.

    Strategy: Simplify.

    Take the photo-sharing product example. We might start out by saying “100% of parents want to share photos.”

    Now we have a prediction that we can test. Creating an MVP has gotten much easier. I can test out my assumptions with the first ten parents I find—and I can find them anywhere: online, in a coffee shop, or wherever. Over time, I can come up with more specific, sophisticated experiments.

    Remember, our goal is to get our teams to write down whatever it is that they believe already—no matter how sloppy, ill-conceived, or foolish those beliefs are. It’s extremely difficult to talk entrepreneurs out of a bad idea; it’s much better to let reality be their teacher.

    Here’s another way to frame our initial hypothesis that might yield more usable learning:

    We believe that: double-income no kids couples in high-power jobs need a great way to socially share Scotch.

    We will test this by: interviewing 30 DINKs (and include a guess about where to find them)

    We are right if: we find a common Scotch-related problem shared by several (more than 6 or so) of them, which they are currently trying to solve.

    That looks vague, but that’s where we start. Once we have the problem identified, then things will become a lot more specific as we move to IRL 4 and above.

    We need to start our lean startup process with discovery – and that is harder to hypothesise. But we can’t look for false precision, that will lead us down the wrong path.

    Note: Over the past year, I’ve been running (with help, of course!) a bunch of Lean LaunchPad programs with the Commonwealth Science and Industrial Research Organisation (CSIRO) aimed at increasing the impact of all the great research that they’re doing. This is part of a series reflecting on what we’ve learned through the course of six programs involving 40 research projects and more than 250 people. The other posts are:

  • A Minimum Viable Product is an Object for Learning

    A Minimum Viable Product is an Object for Learning

    The idea of a Minimum Viable Product (MVP) is frequently misunderstood. From the name, people often think of it as a prototype, but really, it’s a learning object.

    Here’s an example from our first Lean LaunchPad program. The ASPIRE team was in that group – and they’ve recently launched their platform. Here is how they describe it:

    ASPIRE is an online marketplace which intelligently matches your business with potential purchasers or recyclers of your waste by-products. It saves on your disposal costs and cuts the amount of waste going to landfill. ASPIRE is run by the CSIRO in collaboration with local councils and business networks.

    Register your outputs and inputs to find a match.

    That’s a pretty clear statement of their value proposition. How did they test it? Their first thought was to build the website, but they had neither the time nor the money to do that during the LLP program.

    Here’s what they did instead:

    ASPIRE Concierge MVP

    The hypothesis that they needed to test was:

    We believe that small manufacturers in Victoria will value a service that helps them recycle waste streams.

    To test that, they got 58 people in the room from 32 small manufacturers. Then they did on paper what they proposed to do with their website – list the industrial waste streams they produce, along with the process inputs that they need. Then they looked to see if there were any matches between firms. And there were – 62 of them, in fact, nearly two per firm!

    Now, obviously, this is a very primitive version of the platform that they ended up building. The launch version has a lot of smarts built into it to do that matching automatically.

    But this is a great example of an MVP. They didn’t build a small version of their final product, they built an experience that validated whether or not people had a genuine need for it. The results were an overwhelming “Yes!”

    This is a low-fidelity MVP. The Investment Readiness Level scale includes steps for both a low-fidelity MVP (IRL 5) and a high-fidelity MVP (IRL 7). A high-fidelity MVP is something like a prototype.

    I think that the best way to think about a low-fidelity MVP is a pretotype – and pretotype.org has a bunch of great resources to help you test and validate your ideas quickly and cheaply. That Resources page includes free .pdfs of two books on the topic, and they are both excellent.

    The Pretotyping Manifesto includes a great set of principles to help you build your low-fidelity MVP:

    Pretotyping Manifesto

    Here are some more resources to help you build and test an MVP:

    The Lean Startup approach is based on the Build-Measure-Learn loop. When you combine hypothesis testing with low-fidelity MVPs, you get into the Build-Measure-Learn loop very early in the process.

    It’s a great way to learn if you’re really delivering value for your customers. The best way to think of an MVP is an object for learning.

    Note: Over the past year, I’ve been running (with help, of course!) a bunch of Lean LaunchPad programs with the Commonwealth Science and Industrial Research Organisation (CSIRO) aimed at increasing the impact of all the great research that they’re doing. This is part of a series reflecting on what we’ve learned through the course of six programs involving 40 research projects and more than 250 people. The other posts are:

  • What Assumptions Underlie Your Business?

    What Assumptions Underlie Your Business?

    Every social activity that we undertake is subject to underlying assumptions.

    Think about music – here is a great section from the book Art Worlds by Howard Becker:

    Consider what changing from the conventional Western chromatic musical scale of twelve tones to one including forty-two tones between the octaves entails. Such a change characterized the compositions of Harry Partch. Western musical instruments cannot produce these microtones easily, and some cannot produce them at all, so conventional instruments must be reconstructed or new instruments must be invented and built. Since the instruments are new, no one knows how to play them, and players must train themselves. Conventional Western notation is inadequate to score forty-two-tone music, so a new notation must be devised, and players must learn to read it. (Comparable resources can be taken for granted by anyone who writes for the conventional twelve chromatic tones.) Consequently, while music scored for twelve tones can be performed adequately after relatively few hours of rehearsal, forty-two-tone music requires much more work, time, effort and resources. Partch’s music was often performed in the following way: a university would invite him to spend a year. In the fall, he could recruit a group of interested students, who would build the instruments (which he had already invented) under his direction. In the winter, they would learn to play the instruments and read the notation he had devised. In the spring, they would rehearse several works and finally would give a performance. Seven or eight months of work finally would result in two hours of music, hours which could have been filled with more conventional music after eight or ten hours of rehearsal by trained symphonic musicians playing the standard repertoire. The difference in the resources required measures the strength of the constraint imposed by the conventional system.

    The difference in the resources required also measures the amount of assumptions in the conventional system. Here’s another example: money has no inherent value. How long would it take to navigate daily life if we had to negotiate trading terms for every single thing that we needed? It would require all kinds of extra resources. We have many other sets of conventions like this that allow us to interact – things like time, every single language, standards of dress, and so on.

    Conventions like this save us tons of time and hassle every single day. My brother Gabriel is a fantastic violinist, arranger, composer and generally excellent music guy. Here is his response to the Becker quote:

    I was very lucky a year ago to be part of an “audience” of four squeezed in to the (big!) room at Montclair State where Harry Partch’s own instruments are housed, on the occasion of the final dress rehearsal of several of his landmark works before their reprise at Lincoln Center later in the week…pretty freakin’ amazing music!!

    IF (and it’s a huge ‘if’) one really has something so different and valuable to create musically than the millions before, then MAYBE eschewing centuries if not millenia of tradition for the sake of incredible INefficiency and frustrations and utter lack of infrastructure may be worth it… Maybe, yes, for example, in the very very rare case of Harry Partch…

    Becker is making the point that constraints enable creativity, as you can also see in that statement from Gabriel.

    This is very circuitous route to a very important point: every time we build a business model, there are a TON of assumptions that lie underneath it, and some are more obvious than others.

    Here is what it means for lean startups: every assumption is an innovation opportunity.

    You start with your first proposed business model, which consists of a bunch of guesses.

    Business model guesses

    The next step is to convert these guesses into testable hypotheses – this is why the National Science Foundation in the US has called the Lean LaunchPad approach “the scientific method for startups.”

    There are plenty of tools for doing this – we’ve used Strategyzer. Here is what one of their hypothesis testing cards looks like:

    Hypothesis testing

    The structure for a business hypothesis and test is:

    • We believe that…
    • To validate this belief, we will…
    • And we will measure…
    • We were right if…

    Once you’ve done this, then you have to go do customer development interviews, and record your results. Was the hypothesis right? If so, great! Move on to test the next one. If not, you need to build another hypothesis to test.

    At its simplest level (simple, but not easy), business hypothesis testing lets you test out the hypotheses that must be true for your business model to work.

    At a deeper level, you can be like Harry Partch and think about the fundamental assumptions that underlie the area you’re working in.

    I talked about money – there are all kinds of assumptions with it, like:

    • The value of money is guaranteed by government fiat. Challenging that assumption led to Bitcoin.
    • Money requires currency. Challenging that assumption led to mobile money platforms in Africa like Sente, where money is stored in phone accounts.
    • People keep money in banks. African mobile money platforms also challenge this assumption, where a huge percentage of the population has skipped banks entirely.

    Assumptions aren’t automatically bad. For day-to-day things like language and money, assumptions save us a great deal of time and effort.  Nevertheless, when we are doing new things, it is useful to articulate the assumptions that underlie the work. Challenging these assumptions can lead to potential innovation opportunities. At the least, clarifying assumptions gives us the hypotheses that we must test to improve our chances of success.

    Note: Over the past year, I’ve been running (with help, of course!) a bunch of Lean LaunchPad programs with the Commonwealth Science and Industrial Research Organisation (CSIRO) aimed at increasing the impact of all the great research that they’re doing. This is part of a series reflecting on what we’ve learned through the course of six programs involving 40 research projects and more than 250 people. The other posts are:

    Another note: if you’re interested in Harry Partch, here’s a good documentary about him and his music.