Information mining the social way

Using del.icio.us network feature to find relevant links

How do we search for something?

The Internet is vast and mostly a lot of lost pages with no links between one another originally the humble Hyperlink provided means to move from one page to another but unless you knew an address or stumbled on a link your chances of finding the page you want is slim. Then came the humble search engine and this changed or did they?

We all know the principle of search engines like Google and if you don’t there are plenty of articles on the mechanics of the web from Ho John Lee’s site though they are a bit technical you could also crawl through my de.icio.us accounts search tag and SEO tag. Search engines allow us to search through vast quantities of known links which are picked up either by users submitting those links or through a bot finding them, and while the page ranking system and similar algorithms used by the search engines are very clever they can not guarantee the content of the site. This became a problem in the late nineties as more and more link farms appeared in the search engines listings. Around the time the search engines developed managed directories also appeared including DMOZ these directories were maintained by humans rather then bots and often provided more relevant results, but these two were easy to fool principally because most categories only had one or a small group of editors.

So with the very abbreviated version of searching the web move on to more recent times and the craze of del.icio.us a very simple web site originally set up as a means to store your personal bookmarks it provided a useful place to centralise what for many was becoming an increasing mess particular if like me you moved from computer to computer. It has advantages for companies and groups centralising links to sites as well but the biggest advantage was its ability to view other peoples bookmarks and tags. This feature means that I can search through other peoples links to see what they have found this open several useful features;

1. I can find targeted information based on keywords with more accuracy and with a level of trust.

2. I can discount links from untrustworthy users

3. I can find trends and patterns in possible customers/competition

4. I can track reactions and opinions

del.icio.us like all social bookmarking has a built in system for getting rid of bad links by being time based the more relevant links should by routinely appearing at the top of tag pages, while the bad links sink down. This is obviously not the case for individuals but rather the collective as a whole. This type of linking has a downside popular linking can drown links easily if a site is featured on Digg or Slashdot you can be sure it will flood del.icio.us as well.

The biggest problem is that del.icio.us and similar site have to many users or rather you are trying to view to many users tags but by developing a network of known users with similar interests you can reduce the amount of search while still keeping relevancy. The network feature of del.icio.us does just that. By adding users with similar interests to myself to my network I can take a daily digest of posts and work with just these links, using some thought you can search within your network of users to find older information. But how do you find these similar users, well in the first instance looking at tags which match yours and finding users who provide good links is a good way to start. But the real advantage of the del.icio.us network is looking at the networks of those you have in your network.
For example if we look at my network

we can see that I have 15 people I watch on this account I regular add and remove people from my network but normally I have around 15 people. Out of the above group I will have found 2 maybe 3 from their web sites, another 2 from del.icio.us tags the rest come from the first 5 own networks in effect I’m trusting the original 5 to have only useful people in their networks and therefore already have a certain amount of trust for the newer members, and I can always un-subscibe those that provide duff links anyway.

Does all this work?

I thinks so by using the network feature of del.icio.us to create a trusted subset of people with common interest I can often find relevant sites quickly and easily, while del.icio.us is not yet a substitute for a search engine its not far of and is becoming more and more my first point of call.

This has been a bit of a long post but I hope to cover more tricks with del.icio.us and using it for information mining paticularly with the del.icio.us API. By the way if you were wondering how the screen shot was taken, check out screen saver a firefox plugin (thanks to graywolf pointing it out on his blog)