After John & I yelled at him for an hour last friday, our PhD student Sam MacAulay was still gracious enough to continue talking with me. In the course of discussion he asked a good question – what actual impact complexity science has had on management studies? We talked about it for a while, and concluded that complexity has primarily been used as a metaphor. However, I did point out as an example that Didier Sornette has done superb work in modeling the stock market as a non-linear complex system. Elizabeth Garnsey has also done some really nice research in this area (see for example the interview that is linked to on her webpage, or the excellent book she co-edited with James McGlade Complexity and Co-Evolution). But in the main, I had to agree with Sam that we probably haven’t made as much out of complexity theory as we could.

Nevertheless, Sornette’s book Why Stock Markets Crash is one of the best pieces of practical complex systems thinking that I’ve run across. So I was very pleased to see his work discussed in this article from physicsworld.com. The article also talks about the work of Nassim Nicholas Taleb and Lee Smolin, two other people I’ve got a lot of time for. The main thrust of the article is that scientific research may be too conservative these days as researchers focus more on taking a safe approach to facilitate publishing, while not spending enough time on the riskier big questions. While black swan events (things with a low probability of occuring, but which have a large impact if they do actually arise) are often negative in financial markets, they are sought after in science and innnovation.

I found the article through the excellent Nesta Connect blog. The angle that they thought was most interesting was that the article proposes the formation of markets for research. That is an interesting angle, but not the one that grabbed me. I was struck by this section:

Some scientists, he suggests, are what we might call “hill climbers”. They tend to be highly skilled in technical terms and their work mostly takes established lines of insight that pushes them further; they climb upward into the hills in some abstract space of scientific fitness, always taking small steps to improve the agreement of theory and observation. These scientists do “normal” science. In contrast, other scientists are more radical and adventurous in spirit, and they can be seen as “valley crossers”. They may be less skilled technically, but they tend to have strong scientific intuition — the ability to spot hidden assumptions and to look at familiar topics in totally new ways.

To be most effective, Smolin argues, science needs a mix of hill climbers and valley crossers. Too many hill climbers doing normal science, and you end up sooner or later with lots of them stuck on the tops of local hills, each defending their own territory. Science then suffers from a lack of enough valley crossers able to strike out from those intellectually tidy positions to explore further away and find higher peaks.

This is a nice description of innovating on fitness landscapes, an idea that John and I have been talking about in our classes. And it is particularly important now. There are numerous reports coming out showing that companies are becoming increasingly conservative in their innovation efforts as a reponse to current economic conditions. In Smolin’s analogy, companies are reverting to only undertaking hill climbing. Many have never thought much about valley crossing in the first place, while many others are pulling resources out of more speculative ideas.

This is as wrong for business as it is for science. Innovation is central to survival, and successful innovation requires both hill climbing and valley crossing. Now more than ever it is critical for firms to figure out a way to continue to invest in areas that have breakthrough potential. Newspapers and record companies have collectively been very effective at getting to the top of their hills. And these industries are in big trouble now because they have completely missed on out the discovery of new, better hills. It’s a challenging process to manage in good times, and probably even harder now. But it’s still essential.

Note: The photo at the start is from the very useful resource, the Australian Bird Image Database.