Five Forms of Filtering

filtering

We create economic value out of information when we figure out an effective strategy that includes aggregating, filtering and connecting. The three steps interact and reinforce each other – and successful information-based business models have all three. We can undertake business model innovation by changing our methods in these three areas, or by changing where in the value network the processes take place. I’ve run across a few things recently that have gotten me thinking about filtering – and it made me realise that we have another classification issue here. Here is how Clay Shirky frames it:

So, the real question is, how do we design filters that let us find our way through this particular abundance of information? And, you know, my answer to that question has been: the only group that can catalog everything is everybody. One of the reasons you see this enormous move towards social filters, as with Digg, as with del.icio.us, as with Google Reader, in a way, is simply that the scale of the problem has exceeded what professional catalogers can do. But, you know, you never hear twenty-year-olds talking about information overload because they understand the filters they’re given. You only hear, you know, forty- and fifty-year-olds taking about it, sixty-year-olds talking about because we grew up in the world of card catalogs and TV Guide. And now, all the filters we’re used to are broken and we’d like to blame it on the environment instead of admitting that we’re just, you know, we just don’t understand what’s going on.

Filtering is what helps us deal with the vast amount of information available to us. We try to filter information so that we end up with something that is relevant to us – it helps us learn something, it helps us solve a problem, it helps us develop a new hypothesis about the world around us. These are all connections – and this is what really drives value creation. However, we can’t connect without some filtering going on. So filtering is important, and it’s a term that includes several different sub-types. I can think of at least five forms of filtering.

The five forms of filtering break into two categories: judgement-based, or mechanical.

Judgement-based filtering is what people do. At its most basic level, we have naive filtering. This is what we do when we don’t know anything about the information that we are trying to filter. This is a fairly complex internal process, and there are plenty of models available for what is happening in this step. It’s basically everything going on in the ‘Sense’ step in this diagram by Harold Jarche:

As we gain skills and knowledge, the amount of information we can process increases. If we invest enough time in learning something, we can reach filter like an expert. I previously explained how this process can work in bird-watching.

However, even experts can’t deal with all of the information available on the subjects that interest them – that’s why they end up specialising. One way to increase the amount of information that gets filtered is by relying on a network. This can be a network of learners, as in the Connectivism course run by George Siemens and Stephen Downes, it can be a group of people with a similar interest, like all of us talking about #innovation on twitter, it can be large groups of otherwise unconnected people as in Shirky’s examples on places like Digg and Delicious. Networks expand our reach enormously.

There can also be expert networks – in some sense that is what the original search engines were, and what mahalo.com is trying now. The problem that the original search engines encountered is that the amount of information available on the web expanded so quickly that it outstripped the ability of the network to keep up with it. This led to the development of google’s search algorithm – an example of one of the versions of mechanical filtering: algorithmic.

Algorithmic filtering is used by most of the filtering tools available on the web. They can cover the entire web, like google does, or sub-sections of it, like an RSS feed does. Howard Rheingold describes many of these sorts of tools in his post and video on Mindful Infotention.

Rheingold also provides a pretty good description of the other form of mechanical filtering, heuristic, in his piece on crap detection. Heuristic filtering is based on a set of rules or routines that people can follow to help them sort through the information available to them.

Why is filtering important? Understanding the variety of filters available explains why there are often arguments about the discrimination process, and over what role a particular filter plays. If someone writes about filtering, and they mean ‘algorithmic filtering’, a reader thinking of filtering in terms of ‘network filtering’ is likely to misunderstand the discussion. Another source of confusion is that some people talk about filtering not as a search for useful information, but as a way to block information that annoys them.

Filtering by itself is important, but it only creates value when you combine it with aggregating and connecting. As Rheingold puts it:

The important part, as I stressed at the beginning, is in your head. It really doesn’t do any good to multiply the amount of information flowing in, and even filtering that information so that only the best gets to you, if you don’t have a mental cognitive and social strategy for how you’re going to deploy your attention. (emphasis added)

Filtering and connecting is what leads to important skills, like pattern recognition (described well by Venessa Miemis).

Finally, we can use these ideas about filtering to help with business model innovation by changing where it takes place in the value network. One of Shirky’s points is that since Gutenberg, the economic logic of publishing required publishers (of books, music, movies) to act as filters in order to maximise their investment. As publishing and filtering has shifted out to human networks, publishers no longer need to fill this role. Someone (or some network) needs to, and since that creates value, it’s something that can perhaps be monetised.

You can see this in investing. You can put money in Berkshire-Hathaway, where investment choices are run through the personal expert filter of Warren Buffett. Or you can invest in individual stocks recommended by a broker- which is filtering through an expert network. Or you can take advantage of DIY investing, where you do your own filtering, probably aided by some heuristic filters as well. Three different investing business models based on three different filtering methods.

People or networks filling the filtering role now are creating significant value – and people trying to come up with innovative business models in these fields should be thinking about how they can create value through filtering. Of course, the filtering needs to be part of an overall aggregate, filter and connect strategy – which is at the core of successful information-based digital business models.

(Thanks to John, Phil Long & Nancy Pachana for talking to me about these ideas- of course, none of this is their fault. Special thanks to Phil for editorial suggestions.)

Student and teacher of innovation - University of Queensland Business School - links to academic papers, twitter, and so on can be found here.

Please note: I reserve the right to delete comments that are offensive or off-topic.

Leave a Reply

32 thoughts on “Five Forms of Filtering

  1. Thank you for making this area of filtering easier to understand going forward.

    Interestingly, Seth Godin also mentioned Clay Shirkey in his latest post.

  2. This worked out really well Tim. Expert groups might be better at filtering information than individuals but I wonder how this changed in the face of rapid change or discontinuity. Does the problem of groupthink then outweigh the superior filtering capacity. Remind me to tell you about the immediate reactions of people in the world trade centre on 9 11.

  3. Thanks Matt! This one was hard work, so I’m glad you think it came out ok! Not surprised that Godin (or anyone, for that matter) mentions Shirky – he’s definitely a leading thinker in this area…

  4. Thanks John – the groupthink issue is definitely an important one. Not quite sure how I’d fit it into this, but it probably should be considered. Thanks for your help on this!

  5. Hi Tim!

    I would like to add another aspect to filtering, which I’m still trying to figure out how to really say, but it’s something many of us have agreed upon as we’ve been thinking about the Junto platform.

    I know that many people want reductionist models and everything clearly explained and repeatable, but I strongly believe that we are moving towards a new kind of thinking that is inclusive of holism and intuition, and that there is an entire repository of ‘knowing’ that we are just learning (relearning?) how to tap back in to.

    I think this is based in energy, and that we are able to become more open to “feeling” its frequencies.

    for instance, this notion is really coming from direct observation on Emergent by Design, and watching the way ideas have been built and refined over the months, and the flows of participation from people for whom the messages have “resonated.”

    just as you have laid out your ‘aggregate-filter-connect’ pattern, i have also observed the pattern. (i think it might just be The Pattern… the life cycle of any process – I talked about it with a metaphor in yesterday’s post, making an analogy between building trust, and growing a garden).

    another way i’ve heard the pattern described is from a ‘meta’ level as “rock-tree-cloud-star-rock”;

    another way has been “Empty (Or quiet) > Initiator (Or excitement action) > Binding (Assembly) > Crash or Propagation > Empty”

    another way has been “Attractor (curiosity) – Challenge (tension) – Opportunity (inspiration) – Strategy (hope) – Test (confidence) – Decision (resolve)”

    do you see the pattern?

    however you want to think of it, it’s an iterative process that is going on around us at all times, that is us as well, as we are always taking in sensory information and deciding what to do with it.

    but back to the point, yes we can rely on networks for distributed filtering, but as rheingold said, just receiving filtered information, even if it’s high quality, doesn’t do anything.

    what i’ve begun to notice (and now want to see if we can make it a more cognitive process), is that certain information resonates.

    once we begin to see the thought process as a pattern, or the innovation process as a pattern, we begin to mentally filter information into the appropriate place within the pattern for digestion/action.

    this is seeming like a higher level mental functioning (which i am only just beginning to practice), but it is making a lot of sense.

    so it’s not just about filtering information into “folders”… like, oh, tim just tweeted a great article, i’m gonna tag it “innovation” in my delicious links…. but rather, if the article is truly immediately valuable to me, to be able to say “ah, this idea about filtering fits directly into the “x” stage of this process/project i’m working on.

    and once you’re collaborating with a group who is also able to think this way, i am seeing that that is how rapid innovation looks like it could happen.

    requirements: everyone needs to have a version of that bigger mental model of The Pattern. (i’ve been collaborating with around 20 people, and each person has described it to me, without any prompt from me…. so there is something to that). secondly, everyone has to have a clear understanding of the vision of the project or problem they’re working on – the big vision.

    if we are hoarding the vision (keeping information in silos), only a little is revealed to each person in the group, so how could they possible know how to form a cohesive strategy for accomplishing the goal? when everyone knows as much as possible, *how* and *where* and *why* information fits into a process becomes more intuitive.

    yikes… sorry tim, didn’t mean to go off like that! (free flow thinking before 7am!)

    so yeah, i’d love your take on this, as it’s something that’s just starting to form in my mind, and has given me an explosion of ideas on how to build collaborative software programs that put an emphasis on the human/intuitive aspect of innovation and problem solving, and a little less emphasis on the technology tools themselves.

    i think we’re all a hell of a lot more creative than we give ourselves credit for, but we’re locked into thinking like our machines, and being programmed by our programs. i’m trying to break free of that, throw everything aside that i’ve “learned,” and experiment with something that is by nature less definable, but that i think is key to building higher intelligence.

    – @venessamiemis

  6. Thanks for that Venessa! It’s worth writing the post just to get a response like that…

    My brief reply is that I pretty much agree with all that you say. Breaking this down to just ‘filtering’ was extremely reductionist, and consequently, incomplete. I am mainly trying to get my head around what different people are talking about when they say ‘filter’.

    But it is definitely the ‘connecting’ that is the key in everything – and that I think maps on to what you’re saying pretty well. This:

    “so it’s not just about filtering information into “folders”… like, oh, tim just tweeted a great article, i’m gonna tag it “innovation” in my delicious links…. but rather, if the article is truly immediately valuable to me, to be able to say “ah, this idea about filtering fits directly into the “x” stage of this process/project i’m working on.”

    to me is definitely a filter + connect step, which is where all the action really is.

    I do see the pattern you’re talking about – & I think you’re right, we’re all tugging at strings that are part of the same ball of yarn. Not sure how it’ll work out, but it’s certainly an interesting area!

    The garden post yesterday was excellent, btw. I’m not commenting when I don’t have anything more substantive than that to add, but I think you’re on a great track right now and I’m excited to see where you end up.

  7. Nice one Tim, and very timely. I’ve been doing parallel research on mechanisms of community filtration for my PhD system on online scenario planning.

    Would love to see some follow up thoughts on the mechanisms of filtering, i.e., voting, averaging, scoring, etc.

    Have you seen Malone’s work on Harnessing Crowds: Mapping the Genome of Collective Intelligence? From the MIT Center for Collective Intelligence. It has an excellent schema to various tasks related to online community problem solving.

  8. Thanks for the feedback Noah! The mechanisms are definitely worth exploring more deeply – I’ll definitely give that some thought. I’d be interested to hear what you’ve learned too (although if I have to wait for the PhD to be finished, that’s ok!).

    I haven’t seen Malone’s work, & it looks very useful. Thanks for the tip!

  9. Tim,
    As always a great post. This week I was on a lite-Twitter diet because of a conference involvement and so just catching up on some of the reads.

    I totally like the way you framed the filtering. After reading it, the one thing that popped in my mind was maybe an additional method. I will lay it out and you can tell me if it already falls in one of your categories.

    In addition to the methods you list out, I was thinking there is what I would call “probabilistic filtering” – quite distinct from heuristic and algorithmic filtering. Not to get too mathematical, but this type of filtering is founded more on Bayesian inference and Shannon’s information theory. The key here is that using probabilistic pattern recognition and removal of redundant and unneccessary content, one can extract the essence of any document or conversation. Not the best of example, but an analogy would be interpreting a conversation in a loud, crowded room.

    Anyway, just wanted to throw this out. Great post once again.

    Regards,
    Ned

  10. Thanks for the feedback Ned. That’s a really interesting example. When I was writing it, I was thinking that anything involving pattern connection would be a case of filtering + connecting, so not purely filtering. Now I’m not so sure. I’ll have to think about it some more. In any case, that’s a great addition!

    The main drawback though is that if that really is another form of filtering, then I’ll lose the alliteration in the title, unless I go with something like Six Sorts of Selection… :-D

  11. Hi Tim,

    I’d like to throw some thoughts into this mix on Filtering, i thinnk we all understand that filtering is really a transition to a “better something”

    What i noticed in regards to “filtering” and next level connections is “how/when/why/what” it happens.

    I’ll give you an example.
    One of Venessa’s posts on The Future of Networks is the starting point or point of contact i began with. I automatically created a mindmap through mindmeister to capture in real time the comments that were being left. (Difficult but doable)

    The first level then i found was participation.
    The blog post was the point of contact and points of interest, also the first pattern Venessa described as Attractor/Curiousity.

    I automatically created a stem from the focal point which was the post and created the folder “Participants” the second pattern that emerged from people leaving comments was they were shring links as well, thus i kept track of every link that was provided.

    but more importantly what i understood was the convversation that was going on, which is pattern two from attractor/curiousity towards Tension. This tension is the conversation that goes on through twitter and blog posts,

    Eventually all that tension/conversing is transitions into opportunity.

    The conversation that is going on from those points of interests are conversing towards a resonating degree of harmony. they are feeding the system putting tension through conversation that eventually breaks into a new realm of ideas otherwise unnoticed if the conversing wasn’t going on.

    in that opportunity then comes hope and straegy.

    Filtering thus, is the grind work that goes on, the constant conversing is putting tension on the conversaters, it’s processing information through each tweet, each debage and so forth…

    The next thing you know, your sense or conscious awareness begins to attract what resonates from the constant tension, next thing you know your in sync with a trusted network, who are in sync with each other in “vision” but are on the edges of there own networks, feeding the trusted network with higher levels of information and understanding….

    i hope this helps in regards to filtering

  12. Thanks for the great comment Spiro! I definitely agree that the next level of connection is where the action is – that’s why I keep talking about aggregate-filter-connect. This post was looking at filtering from a very reductionist perspective, just trying to clarify the differing mechanisms.

    But overall, value is generated through connections, as your example shows very well!

  13. Hi Tim

    I think the bottom line for me after months of observation around these types of topics is, it can’t be planned to gain higher insight, we can plan projects around these insights or like Venessa said at the moment it arises it fits here and it’s placed accordingly.

    But now i see that the level of connections come from an improvised state of discussion.

    The problem is and i think that’s why it’s important like you and others who experiment with models, because the closer we observe the patterns that are coming from these connections the better we can model them at least to a degree of helping others understand them.