Three Innovation Lessons from the Outcome of the Netflix Prize

You probably heard about the contest that Netflix started in 2006 to crowdsource improvements in their recommendation algorithm. They offered a $1 million prize to anyone that could improve the accuracy of the recommendation algorithm by at least 10%. In 2009, a team of people hit the target, and won the prize.

Awesome, right? The team got their big check, Netflix got their performance improvement, and everyone ended up happy.

Well, sort of.

It turns out that Netflix has never implemented the algorithm that won the prize. Mike Masnick has an excellent article outlining this surprising turn of events.

Here is part of what Netflix says about it:

We evaluated some of the new methods offline but the additional accuracy gains that we measured did not seem to justify the engineering effort needed to bring them into a production environment.

Why not? Because the way their customers use the service has shifted:

Streaming has not only changed the way our members interact with the service, but also the type of data available to use in our algorithms. For DVDs our goal is to help people fill their queue with titles to receive in the mail over the coming days and weeks; selection is distant in time from viewing, people select carefully because exchanging a DVD for another takes more than a day, and we get no feedback during viewing. For streaming members are looking for something great to watch right now; they can sample a few videos before settling on one, they can consume several in one session, and we can observe viewing statistics such as whether a video was watched fully or only partially.

This is pretty amazing, and there are some very interesting lessons here:

  1. It’s hard to fit new innovations into old business models: the recommendation algorithm was built for rentals, where the recommendations have to be right. So there was great value in making them more accurate. For streaming, when people can sample a movie before they watch all of it, there is much less pressure to get all of the recommendations absolutely correct. Trying to fit streaming into the old business model has been giving Netflix fits for most of the past year or two, and this is just more evidence of how hard it is to fit new innovations into old business models.
  2. You have to break connections to make room for your new ideas: the engineering issue is an interesting one. Apparently a fair bit of effort was required to code the new algorithm into all of the existing processes. And that’s always the case with a new idea. You can’t just parachute new ideas into existing slots in the economy. First, you have to make space for them. You do this by breaking connections.
  3. Optimising when your environment is changing is very dangerous: this is the big lesson. Netflix was optimising rental while the entire structure of the industry was changing. What they really needed to be doing was get the business model for streaming right. However, this wasn’t at all obvious when they started the contest in 2006. You always need to be aware of what’s going on around you. This has an enormous impact on what kind of ideas you should be testing out. If the environment is changing rapidly, then making yourself more efficient is risky – it’s quite possible that you’re perfecting a process that could be obsolete in a couple of years.

It’s been astonishing to watch Netflix struggle with the business model for streaming. They have been a textbook case of disruption for they way they changed the structure of the movie rental industry. And now they’re having huge problems adapting to the latest change in industry structure.

It’s kind of scary when things move this fast. That’s a big part of why being risk averse and not innovating is actually more risky than you think.

At least the guys that wrote the algorithm got their $1 million.

Student and teacher of innovation - University of Queensland Business School - links to academic papers, twitter, and so on can be found here.

Please note: I reserve the right to delete comments that are offensive or off-topic.

9 thoughts on “Three Innovation Lessons from the Outcome of the Netflix Prize

  1. I recently had a discussion with a colleague of mine on the role of coordination in supply chain management. I argued that if you want to work on supply chain sustainability, then you need to rethink your business model (otherwise it just stays business as usual). Coordination, matter less to achieve this innovation, than finding out what to coordinate in the first place.

    We got into quite a heated debate over this, because I couldn’t build the argument that coordination equals optimization. I like this example of Netflix a lot! I’ll throw it into our debate, and hope it turns into a dialogue :-)

    Btw, have you had a chance to look at my latest blogpost? Keen to have your thoughts on it!



    • Thanks for the comment Bart! I definitely agree with your ideas about new business models for supply chain sustainability. I hope the Netflix example helps you build your case!

      I just read the post, which is a good one. I wrote a comment, but I think WordPress might have eaten it. Just to be on the safe side, here is what I said:

      “Interesting post Bart! I find Gray’s connected company idea to be very appealing. And it seems intuitively right to me as well. So I think it’s particularly interesting to see organisations such as banks acting in a more connected way. It’s also fantastic that they were willing to sponsor your business model innovation workshop. Both are very encouraging signs! I’m not sure that any of the banks here in Australia will be doing similar initiatives any time soon…”

  2. Tim, you hit the nail on the head with #1: “It’s hard to fit new innovations into old business models”. If more companies would take the time to make their existing business models explicit, they would have an easier time evaluating new opportunities. Many people assume that all innovation is awesome and believe that a blockbuster idea is a silver bullet. However, marshaling resources and getting an innovation to market often takes internal transformation or external separation. Existing success metrics and distribution channels exert huge inertia. I wonder if “fit” is possible–maybe you always have to support innovations in parallel to existing business?

    • Thanks for the thoughtful comment Olaf! There has been a fair bit of academic attention paid to your last question, and the results are mixed. It is starting to look as though most of the time, though, that it is best to support innovations in parallel rather than trying to fit them into existing models. Tough question though.

      I enjoyed your review of Business Model Generation, too, btw.

  3. I find this an intriguing example of user-generated innovation (sort of – meaning the way they use the product) outpacing innovation contests. It’s as if Charles Lindbergh flew across the ocean to find the contest had changed from planes to jetpacks. (Okay, an extreme example, still this is such a potent example of how fast change happens).

    • Interesting analogy. It is a bit of a strange case, but it tells you something about the pace of change in some industries right now.

Comments are closed.