The conflict between efficiency and insight

One key step in innovating is generating novel ideas – and this is based on insight.  So how can we increase insight?  That is the topic of the excellent book Seeing What Others Don’t by Gary Klein (check out Harold Jarche’s review here).

Klein starts by looking at the impact that insight has on performance, using this diagram:

According to Klein, performance improvement results from a combination of reducing errors and uncertainty, while increasing insights.  This is a fundamental tension in organisations.  As Paul Sloane says:

Businesses are good at getting better but poor at getting different.

Jeffrey Phillips outlines how firms manage this balance, and why it is easier to focus on efficiency at the expense of insight and innovation:

Businesses, with timelines and incentives that are somewhat different, focus on short term financial results, which tends to shift the balance between innovation and efficiency toward efficiency.  Most initiatives that focus on improving efficiency have an immediate, and positive financial impact.  Thus efficiency is rewarded, and initiatives that are rewarded are repeated.  Innovation often has a negative short term impact – costs without an immediate benefit – so innovation is far less likely to produce a short term financial benefit, and therefore much more difficult to do.  Slowly, over time, the scales shift from a balance between efficiency and innovation to ever more efficiency and increasingly less innovation.  Eventually efficiency is well understood and easily accomplished, but it has ever decreasing marginal returns.  Innovation, on the other hand, becomes more difficult the less it is practiced, and is viewed as risky, uncertain and become even less likely to be taken up.

How insight works

How can we address this problem?  Klein’s answer is to increase insight.  To figure out how to do this, he studied 120 cases of insight, looking for commonalities.  Which he didn’t find.  Instead, he found five forms of insight, some of which contradict the others: contradictions, coincidences, connections, curiousity and creative desperation.

His basic premise is that these forms of triggering events cause us to change the stories that we tell ourselves, and, ultimately, others.  It is this change in story that leads to performance improvements.  Here is how Klein maps the process:

One of the routes to insight is curiousity – this is based on keeping an open mind.  This is how many scientific discoveries come about.  But contradictions generate insights through being skeptical – this is the opposite of keeping an open mind!  Klein’s insight is that it is not the type of action that defines an insight, but rather the outcome – the change in the story.

This illustrates a very common error that we make when we study things – we don’t classify them very well.  Previous insight researchers have focused on the act that led to the insight.  This led to arguments about whether they were always generated by a flash of inspiration, or if they incubate and build up over time.  It turns out they do both.  And both open minds and skeptical minds can generate important insights.

The critical step is changing the story that we tell ourselves about how what we’re thinking about works.

Increasing the up arrow

Klein agrees with Phillips that the natural tendency in organisations is to over-emphasize the down arrow of efficiency at the expense of the up arrow of insight.  And he has some ideas for improving you insight capabilities.

The first step is to understand the different forms of insight, and how they work to create changes in behaviour.  This is the change in story that we have already discussed.

Inside of organisations, Klein advocates using insight advocates.  These are people that collect and share stories of how insights have been generated and successfully acted upon inside the organisation.  We know that one way that people become more innovative is by having role models.  Actively gathering and telling stories of insight will help to create these role models within our organisations. It helps to build community too.

Another tool is to build some capability in helping people gain insights.  Klein says:

Many professionals have a dual mission – to gain their own insights and then to enable others to grasp these insights or even to reach ones that are different but valuable. Therapists are continually trying to gain and give insights in their sessions with patients. Artists and writers also traffic in insights even though they rarely have a direct relation with their audiences. Historians are engaged in seeking and sharing insights. And, of course, teachers are also in the insight business.

Managers should be on this list as well.    I’ve said before that removing obstacles is one of the most important jobs that managers have.  This includes removing obstacles to insight.

Finally, Klein talks about the need for willpower – removing gumption traps.  This can mean being open to changing your behaviour and goals in response to insights.  Klein says:

Organizations demonstrate willpower when they act on insights, particularly insights about their primary goals.  An insight about a gola isn’t about being flexible and adapting plans in order to reach the original goal.  It’s about changing the goal itself.

Insight is a critical part of innovation.  If we are to innovate, we must be able to generate insights.  This can’t be done on a schedule, but we can make changes that improve our odds.  The tools that we need to use reduce our mental and organisational rigidity.  They disrupt our thinking, and they change our stories.

This often makes people uncomfortable.  But it’s what we need to do to find new ideas.

Related articles

• The Remarkable Ways We Gain Insights
• Here are five things you should know abo
• When Efficiency is the Enemy of Innovation
• how to train your brain to see what others don’t