In December of 1992, there were 50 websites on the internet. A year later, when they started building Yahoo, we had jumped to 623. So if you were going to build a search engine, what would be the best way to index things? Actually, at the time there weren’t any search ‘engines’ – we’d go to a search ‘index’ for information about the net. When the numbers were small, it made a lot of sense to build these by hand. People could look at each new site as it was built, and figure out in which categories it best fit. Why would you need an algorithm or an engine to do this? After all, it would take longer to make one than it would to look at everything that was there on the net at the time.
This is what Yahoo’s first home page looked like:
Each of these links (only 32,000!) was categorised by hand – at the time, indexing the internet was a craft. It took skill, some people were better at it than others. When we chose between Yahoo, Lycos or AltaVista, the choice was usually based on which had better people working for it. If one of them had someone that knew more about the area we were interested in, we would get better search results from that index. It was idiosyncratic, and often very frustrating when you tried to find something specific.
A bunch of stuff then happened, the indices added bots to search for new pages automatically, and they started figuring out ways to rank results more effectively. But there was still a significant amount of craft that went into building a search site. Then we had Google, with their algorithms. When I first used Google in 1997 or so, I was amazed at how much better the search results were. I could actually find what I wanted! At scale, Google’s algorithms were much more effective than Yahoo’s craft.
This poses a dilemma for innovators. When we start out with our new idea, first off, we just need to make sure that it works. Usually, this takes craft. The problem is, the methods that we use at the start often lock is in as our market grows. In the face of Google in the late 90s, the original search engines argued that human judgment provided better search results than an algorithm. The problem that they had was this:
Craft worked ok when there were 30,000 pages to index, but it didn’t work so well when there were 30 million. DMOZ tried to crowdsource indexing, and even that couldn’t keep up.
The transitions from markets that reward craft to those that reward scale is difficult to make. This is the point that Geoffrey Moore made in Crossing the Chasm – many innovative new firms fail when they have to make that jump. His book is a good starting point for developing a strategy to scale your new ideas.
A thought that I’ve had recently though is that in markets that reward scale, there are often innovation opportunities within niches for craft. Google search results are still based on popularity, so I think there actually is an opening for expert indexing of topics where popular isn’t necessarily best (you have the chance to be a good filter). When you’re competing against giants, there are bound to be some openings – and finding these are often the way that disruptive innovations start (see the Scott Anthony video here for more on that).
The other interesting thing is that when your big-scale market is eventually replaced by something else, it opens up craft opportunities again. For example, you can still buy buggy whips. Hand-crafted, beautifully made buggy whips. Despite the fact that cars made them obsolete over a century ago.
So there are two main points. First, if you’re bringing an innovation to market, figure out how it will scale. If you don’t, you’re likely to lose out to a fast follower that has. We tend to think that things increase along a straight line, not exponentially. Craft often scales along a straight line, but it doesn’t do very well on an S-Curve (which is what everything actually follows!) Second, if you’re looking to disrupt a scaled market, figure out a way to take advantage of craft. If you can figure out a way to scale craft, then you really have a winner!
Tim,
This reminds me my own experience more than a decade now. When I obtained my first degree, which was in computer science with a specialisation in 3D graphics, I came to the realisation that I was at the end of an era. 3D graphics (before Toy Story) was a craft that is “hand made” my talented programmers. But then, quickly, all the knowledge that I had learnt (or will learn) about 3D graphics in my $50,000 degree could be put in a tiny chip that costs $50.
I have only 3 options then: continue the arms race with the chip and push my knowledge and skills further in the high-end (R&D), play support or maintenance role to the 3D chips (the same role that people at yahoo and google play today), or branch out in other unexplored directions.
Going the R&D way meant going to graduate school and committing myself to a long-term career in a highly competitive environment. Playing a support role had little appeal to me and lacked the sense of satisfaction that I was seeking. So I chose the latter option, which meant I had to basically throw away all the specialised knowledge and skill that I had spent years to develop.
In my situation, the only truly useful (and long-lasting) knowledge that I gained from that degree from its “general education” component: all those courses in anthropology, philosophy, organic chemistry, accounting, political history that had nothing to do 3D graphics but I was MADE to take. And I am grateful for that!
That’s an excellent example of the disruption that’s caused at the transition points. It’s pretty bad going the other way too, when the market stops working at scale – lots of people end up losing their jobs as it reverts to a craft…
‘In December of 1992, there were 50 websites on the internet.’
I remember some librarians at the time complaining about how poor the internet’s catelog system was. And how what was really needed was good application of the Dewey system by librarians. They could still be right.
Possibly – although the Dewey Decimal system is hopeless for IT-related classification – everything gets crammed into an incredibly small range of numbers. So we’d still need some innovation there!
“DMOZ tried to crowdsource indexing, and even that couldn’t keep up”
Interesting to think that there has been sort of a resurgence via crowd sourcing filtering (diggit etc).
Good point Sam. Of course, diggit, reddit etc. aren’t trying for comprehensive coverage – which is probably the biggest difference between them and DMOZ (and that’s a HUGE issue…).
If you like Hagel and Singer’s “unbundled corporation” then “craft” would correspond to “product innovation” and and “scale” would correspond to “infrastructure management”. The interesting thing is that a third route is implied: “customer relationship management”. In the example in the article, this would mean a company that provides highly specialised indexing to a niche of customers with whom they nurture a close relationship. Perhaps such services already exist…?
Graham