In December of 1992, there were 50 websites on the internet. A year later, when they started building Yahoo, we had jumped to 623. So if you were going to build a search engine, what would be the best way to index things? Actually, at the time there weren’t any search ‘engines’ – we’d go to a search ‘index’ for information about the net. When the numbers were small, it made a lot of sense to build these by hand. People could look at each new site as it was built, and figure out in which categories it best fit. Why would you need an algorithm or an engine to do this? After all, it would take longer to make one than it would to look at everything that was there on the net at the time.
This is what Yahoo’s first home page looked like:
Each of these links (only 32,000!) was categorised by hand – at the time, indexing the internet was a craft. It took skill, some people were better at it than others. When we chose between Yahoo, Lycos or AltaVista, the choice was usually based on which had better people working for it. If one of them had someone that knew more about the area we were interested in, we would get better search results from that index. It was idiosyncratic, and often very frustrating when you tried to find something specific.
A bunch of stuff then happened, the indices added bots to search for new pages automatically, and they started figuring out ways to rank results more effectively. But there was still a significant amount of craft that went into building a search site. Then we had Google, with their algorithms. When I first used Google in 1997 or so, I was amazed at how much better the search results were. I could actually find what I wanted! At scale, Google’s algorithms were much more effective than Yahoo’s craft.
This poses a dilemma for innovators. When we start out with our new idea, first off, we just need to make sure that it works. Usually, this takes craft. The problem is, the methods that we use at the start often lock is in as our market grows. In the face of Google in the late 90s, the original search engines argued that human judgment provided better search results than an algorithm. The problem that they had was this:
Craft worked ok when there were 30,000 pages to index, but it didn’t work so well when there were 30 million. DMOZ tried to crowdsource indexing, and even that couldn’t keep up.
The transitions from markets that reward craft to those that reward scale is difficult to make. This is the point that Geoffrey Moore made in Crossing the Chasm – many innovative new firms fail when they have to make that jump. His book is a good starting point for developing a strategy to scale your new ideas.
A thought that I’ve had recently though is that in markets that reward scale, there are often innovation opportunities within niches for craft. Google search results are still based on popularity, so I think there actually is an opening for expert indexing of topics where popular isn’t necessarily best (you have the chance to be a good filter). When you’re competing against giants, there are bound to be some openings – and finding these are often the way that disruptive innovations start (see the Scott Anthony video here for more on that).
The other interesting thing is that when your big-scale market is eventually replaced by something else, it opens up craft opportunities again. For example, you can still buy buggy whips. Hand-crafted, beautifully made buggy whips. Despite the fact that cars made them obsolete over a century ago.
So there are two main points. First, if you’re bringing an innovation to market, figure out how it will scale. If you don’t, you’re likely to lose out to a fast follower that has. We tend to think that things increase along a straight line, not exponentially. Craft often scales along a straight line, but it doesn’t do very well on an S-Curve (which is what everything actually follows!) Second, if you’re looking to disrupt a scaled market, figure out a way to take advantage of craft. If you can figure out a way to scale craft, then you really have a winner!