You probably heard about the contest that Netflix started in 2006 to crowdsource improvements in their recommendation algorithm. They offered a $1 million prize to anyone that could improve the accuracy of the recommendation algorithm by at least 10%. In 2009, a team of people hit the target, and won the prize.

Awesome, right? The team got their big check, Netflix got their performance improvement, and everyone ended up happy.

Well, sort of.

It turns out that Netflix has never implemented the algorithm that won the prize. Mike Masnick has an excellent article outlining this surprising turn of events.

Here is part of what Netflix says about it:

We evaluated some of the new methods offline but the additional accuracy gains that we measured did not seem to justify the engineering effort needed to bring them into a production environment.

Why not? Because the way their customers use the service has shifted:

Streaming has not only changed the way our members interact with the service, but also the type of data available to use in our algorithms. For DVDs our goal is to help people fill their queue with titles to receive in the mail over the coming days and weeks; selection is distant in time from viewing, people select carefully because exchanging a DVD for another takes more than a day, and we get no feedback during viewing. For streaming members are looking for something great to watch right now; they can sample a few videos before settling on one, they can consume several in one session, and we can observe viewing statistics such as whether a video was watched fully or only partially.

This is pretty amazing, and there are some very interesting lessons here:

1. It’s hard to fit new innovations into old business models: the recommendation algorithm was built for rentals, where the recommendations have to be right. So there was great value in making them more accurate. For streaming, when people can sample a movie before they watch all of it, there is much less pressure to get all of the recommendations absolutely correct. Trying to fit streaming into the old business model has been giving Netflix fits for most of the past year or two, and this is just more evidence of how hard it is to fit new innovations into old business models.
2. You have to break connections to make room for your new ideas: the engineering issue is an interesting one. Apparently a fair bit of effort was required to code the new algorithm into all of the existing processes. And that’s always the case with a new idea. You can’t just parachute new ideas into existing slots in the economy. First, you have to make space for them. You do this by breaking connections.
3. Optimising when your environment is changing is very dangerous: this is the big lesson. Netflix was optimising rental while the entire structure of the industry was changing. What they really needed to be doing was get the business model for streaming right. However, this wasn’t at all obvious when they started the contest in 2006. You always need to be aware of what’s going on around you. This has an enormous impact on what kind of ideas you should be testing out. If the environment is changing rapidly, then making yourself more efficient is risky – it’s quite possible that you’re perfecting a process that could be obsolete in a couple of years.

It’s been astonishing to watch Netflix struggle with the business model for streaming. They have been a textbook case of disruption for they way they changed the structure of the movie rental industry. And now they’re having huge problems adapting to the latest change in industry structure.

It’s kind of scary when things move this fast. That’s a big part of why being risk averse and not innovating is actually more risky than you think.

At least the guys that wrote the algorithm got their $1 million.