Nilesh's Weblog

Invisible Web

The latest issue of Businessworld India (14th October 2002) has a very interesting article on web searches (page 50). Charles Assisi has given some very good site links for searching what the everyday Googles and Altavistas cannot give you. Even me, with a 6^1/2 year web experience, never came across some of these links!! For you lazy bones who do not want to get off the keyboard, I put it all in brief with my perspective.

I did my scouting over the web for what Charles want to say. It is called the Deep Web or the Invisible Web. There a white paper on how the Deep Web works. Says this FAQ on the Deep Web -

The Deep-Web is content that resides in searchable databases, the results from which can only be discovered by a direct query. Without the directed query, the database does not publish the result. When queried, deep Web sites post their results as dynamic Web pages in real-time. Though these dynamic pages have a unique URL address that allows them to be retrieved again later, they are not persistent.

The amount of information stored in the Deep Web is about 750 terabytes as compared to the 19 terabytes that the conventional engines crawl. How does one access the Deep Web? There are three cardinal rules for searching the Deep (I give them here in brief) -

Think databases. The result "India genome databases" will give a better result than "India genome" for a researcher interested in human genome research in India. Similarly, "India FDI database" is better than "India FDI". This rule has a simple logic which even I realised after spending sometime on search engines. The conventional spiders crawl static pages. Once they come across dynamic content, they simply stop there. Google is one of the culprits. As you know today, more and more sites are becoming dynamic. Hence more the difficulty in reaching the content stored in databases.
There are a few "deep" search tools. Like I said I haven't heard of these before -

• Bullseye (Not free)
• Lexibot (Not free)

Trial and Error. I am sure most of us might have learnt this by now. If you do not get what you want in the first search, try changing the search terms. e.g. If I am looking out for a specific tutorial on camera shutter speeds, the search terms "photography tutorial shutter speeds" and "photography shutter speeds tutorial" give slightly different results. The latter pulls up a page that I find relevant, on the first page itself. So do not get disappointed on your first search result. Try tweaking a bit.

Search Multimedia. Look for multimedia content like pdf files, audio, video. Some good sites for searching multimedia content (alongwith Google, which I feel is good at multimedia searches) are -

• Direct Search
• Invisible Web
• Completeplanet

Mind you, these engines do not search for generic information. They answer only to specific queries. So you might get frustrated using them for generic things. Here are some more links -

• SearchPDF
• Askeric
• FindArticles
• Librarian's Index to the Internet
• Tool Kit for the Expert Web Searcher
• Digital Librarian
• Infomine
• XRefer

And this isn't what the Deep Web is all about. There is more to it. Do a bit of research yourself and you'll enjoy searching from now.