8 Ways for Searching the Dark Web - Beyond Google!
Posted by: Rea Maor In: Search Engines - Tuesday, July 3rd, 2007Google this, Google that, Google something else. But there’s a lot more to the web than Google, and in fact Google only shows you a tiny bit of what’s going on. You’ve heard of the “deep web” or “dark web” - the part not normally indexed by Google, and maybe even not by Yahoo, MSN, or Ask. Well, here’s a little list presenting a few “rabbit holes” into that vast, uncharted territory!
Dogpile - In the first place, you can check the Big Four (Google, MSN, Yahoo, Ask) search engines all at once with Dogpile. Since other search engines use the indexes from one of these four, chances are if you cannot find it here and you are sure it exists, it is “dark” to the web.
Clusty - Clusty is a more comprehensive search, finding those deep, dark crevasses that other search engines pass over. I’d nominate Clusty for “the dark-web Google” Of course, the more inclusive you make a search engine, the more spam sites it will pick up, so you might have to wade through a lot of garbage.
USA Library of Congress - Good for finding research materials for scholarly interests.
Nelson Search - If you’re looking for a journalistic piece, you can’t beat the self-proclaimed search engine for journalists. If there was a news story on it, it’s here.
Intute - Well, if the bots don’t do a good enough job of weeding out the spam sites, how about giving the humans a go? Intute is the only search engine which uses only web pages quality-checked by human researchers - guaranteeing that you’ll never get a spam hit!
AltaVista - What better way to search the dark web than use a dark search engine? AltaVista died in popularity when Google came out, but it’s still kicking, and it returns hits similar to Clusty.
Wayback Machine - Maybe the page you’re looking for no longer exists? that’s OK, the Wayback Machine should have an index of it. The only snag here is, you have to give it an exact URL. Once you have that URL, you can find the entire history for the domain - sometimes through several owners!
Bloglines - A search engine just for finding blogs. Anything that’s a blog is here, and these days the web is mostly blogs anyway!
Note that the whole thing behind the dark web is that it is mostly made of sites that are one of (a) spam sites rejected for quality (b) personal home pages, bulletin board archives, and other stuff not generally of interest to the public (c) academia, which lives in its own world (d) government, which lives in its own world (e) criminal and underground sites that aren’t in too great a hurry to be found!
Popularity: 25% [?]
Related Posts:
- Search Engines - a look under the hood
- Search Engine Study - part 3: The Great Search Engine Era
- It’s Traffic Day (Or: Digg is Over-Rated)
- Search Engine Study - part 1: Search Engines Defined
- Google and the Concept of a Landing Page
July 3rd, 2007 at 15:27
Great article and furtermore a great topic idea. I’m studying library and information science and I’ve learned in Uni that only about 1/10 of HTML pages are catalogued to any degree by search engines.
The rest is mostly intanets and/or password protected.
So in fact, Google, Yahoo, MSN and the other search engines only catalog the free internet, both free of charge and free access.
I see you tagged this as SEO, search engine optimization, isn’t it the exact oposite category? SEI perhaps, i.e. search engine invisibility.
July 3rd, 2007 at 15:32
Good thinking, i’ve actually changed the original category from SEO to Search Engines in General.
July 3rd, 2007 at 18:15
Sweet!
July 3rd, 2007 at 19:59
Good list, Dogpile is more a listing of sites from various search engines and you can see how each one stacks sites in relation, but it’s still not going to reach the invisible or dark web.
July 4th, 2007 at 0:00
Check out the Open Archive Initiative search engine at http://oaister.org. This is a really superb academic resource with links to almost 12 million digital artifacts from 850 institutions.
July 4th, 2007 at 2:23
Check also http://www.50matches.com/ Which only crawls web sites that were bookmarked or voted for by people, in sites like del.icio.us, digg and reddit.
July 4th, 2007 at 2:32
Wow, i got digged - Amazing.
July 4th, 2007 at 3:21
Another mechanism which beats even search engines is dotcut (www.dotcut.com) - dotcut finds ‘dark’ material by only searching for pre-registered search terms. It them indexes thousands of sites every day and notifies you of any new results - long before the search engines even come across them.
christo
July 4th, 2007 at 3:29
[...] read more | digg story [...]
July 4th, 2007 at 4:17
good links! thanks
July 4th, 2007 at 4:23
You forgot Technorati for the blog searching, it offers multiple search options.
July 4th, 2007 at 5:39
Nice post! Thought I’d share my 2 cents … I’ve just started using CoReap.com which enhances Google and Yahoo web search for social bookmarking. Quite neat!
July 4th, 2007 at 9:30
“Intute is the only search engine which uses only web pages quality-checked by human researchers - guaranteeing that you
July 4th, 2007 at 9:35
[...] Now Rea Maor suggests 8 ways of searching the web outside of Google. [...]
July 4th, 2007 at 9:36
[...] of Google’s Reach Posted July 4, 2007 What to do when things you are searching for don’t show up in Google, but they do exist on the web.
July 4th, 2007 at 9:48
[...] “rabbit holes” into that vast, uncharted territory! wonder woman comic cover art, matureread more | digg [...]
July 4th, 2007 at 9:56
http://oedb.org/library/college-basics/research-beyond-google
July 4th, 2007 at 10:28
[...] read more | digg story Posted by computerszone Filed in [...]
July 4th, 2007 at 10:38
[...] read more | digg story Posted by technologyblogs Filed in Uncategorized [...]
July 4th, 2007 at 13:00
great list of engines
July 4th, 2007 at 13:33
[...] Google this, Google that, Google something else. But there
July 4th, 2007 at 14:32
Great insight about it! These search engines do have their niche when we use it..
July 4th, 2007 at 14:52
i dont like Dmoz and they dont like me back…
July 4th, 2007 at 21:51
“Note that the whole thing behind the dark web is that it is mostly made of sites that are one of (a) spam sites rejected for quality (b) personal home pages, bulletin board archives, and other stuff not generally of interest to the public (c) academia, which lives in its own world (d) government, which lives in its own world (e) criminal and underground sites that aren
July 5th, 2007 at 2:33
[...] 8 Ways for Searching the Dark Web - Beyond Google!
July 5th, 2007 at 5:52
[...] Full article is here [...]
July 5th, 2007 at 8:32
[...] 8 Ways for Searching the Dark Web - Beyond Google!
July 6th, 2007 at 3:29
About Intute being the only human-powered search engine, well, there’s another one called Mahalo which is relatively new and entirely run by humans.
July 6th, 2007 at 8:06
Mahalo will not list a website or a specific result (PDF document, etc.) unless you submit it to them or they get paid for it. Even if you submit it, it goes under human editorial review, and it’s my understanding very little makes the cut, so to speak. So it’s not a great expansive search engine, but more like Overture on a strict diet.They aren’t listing my site, and until they do, it’s not a good search engine at all
July 7th, 2007 at 8:26
and you can always use “ChaCha” to have humans assisting you in search and err… if you don’t trust machines (sorry, get a little bit carried out becoz of the transformers movie!).
July 7th, 2007 at 11:32
That’s ok
this movie should come with a warning label…
July 7th, 2007 at 15:40
The answer isn’t human edited results if you want a better search engine. It costs Mahalo nearly $20 to display just one search result. With that kind of overhead their profits would have to be astronomical for them to compile more than the smallest of indexes. That is truly a dream that can’t come true. How long would the biggest search engines last if displaying just one result cost that much? And who wants human edited search? Yes, edit out the spam and garbage by hand since you must, but let us find the rest of the sites out there without your direction, and without your censorship: your so-called “editorial” control.
The next big search engine will win by being much like Google, but will crawl the web and using algorithms to determine rank by quality of results, not how many sites did or didn’t link to it, or how much money they paid for their position. I don’t think any privately or publicly held company can ever achieve what I’m thinking of: a more decentralized server run by everybody and nobody (like p2p) that will serve up better results, and be more democratic–without all the ads!
July 8th, 2007 at 4:10
[...] caminos para buscar en las webs oscura, sin [...]
July 24th, 2007 at 17:28
What about astalavista.box.sk?
August 9th, 2007 at 2:10
[...] 8 ways of searching the areas of the Internet long-forsaken by Google. I personally like Clusty. [...]
September 1st, 2007 at 21:42
[...] Alternatives to Google (AskReaMaor) [...]
September 10th, 2007 at 15:38
I think it will be interesting if Yahoo team up with MSN
February 12th, 2008 at 16:35
Just to add one more in the list -
Twing - its a search engine to search forums !!