8 Ways for Searching the Dark Web – Beyond Google!

Posted by: Rea Maor In: Search Engines - Tuesday, July 3rd, 2007

Google this, Google that, Google something else. But there’s a lot more to the web than Google, and in fact Google only shows you a tiny bit of what’s going on. You’ve heard of the “deep web” or “dark web” – the part not normally indexed by Google, and maybe even not by Yahoo, MSN, or Ask. Well, here’s a little list presenting a few “rabbit holes” into that vast, uncharted territory!

Dogpile – In the first place, you can check the Big Four (Google, MSN, Yahoo, Ask) search engines all at once with Dogpile. Since other search engines use the indexes from one of these four, chances are if you cannot find it here and you are sure it exists, it is “dark” to the web.

Clusty – Clusty is a more comprehensive search, finding those deep, dark crevasses that other search engines pass over. I’d nominate Clusty for “the dark-web Google” Of course, the more inclusive you make a search engine, the more spam sites it will pick up, so you might have to wade through a lot of garbage.

USA Library of Congress – Good for finding research materials for scholarly interests.

Nelson Search – If you’re looking for a journalistic piece, you can’t beat the self-proclaimed search engine for journalists. If there was a news story on it, it’s here.

Intute – Well, if the bots don’t do a good enough job of weeding out the spam sites, how about giving the humans a go? Intute is the only search engine which uses only web pages quality-checked by human researchers – guaranteeing that you’ll never get a spam hit!

AltaVista – What better way to search the dark web than use a dark search engine? AltaVista died in popularity when Google came out, but it’s still kicking, and it returns hits similar to Clusty.

Wayback Machine – Maybe the page you’re looking for no longer exists? that’s OK, the Wayback Machine should have an index of it. The only snag here is, you have to give it an exact URL. Once you have that URL, you can find the entire history for the domain – sometimes through several owners!

Bloglines – A search engine just for finding blogs. Anything that’s a blog is here, and these days the web is mostly blogs anyway!

Note that the whole thing behind the dark web is that it is mostly made of sites that are one of (a) spam sites rejected for quality (b) personal home pages, bulletin board archives, and other stuff not generally of interest to the public (c) academia, which lives in its own world (d) government, which lives in its own world (e) criminal and underground sites that aren’t in too great a hurry to be found!


Related Posts:


39 Responses to “8 Ways for Searching the Dark Web – Beyond Google!”

  1. webjourneyman Says:

    Great article and furtermore a great topic idea. I’m studying library and information science and I’ve learned in Uni that only about 1/10 of HTML pages are catalogued to any degree by search engines.
    The rest is mostly intanets and/or password protected.
    So in fact, Google, Yahoo, MSN and the other search engines only catalog the free internet, both free of charge and free access.
    I see you tagged this as SEO, search engine optimization, isn’t it the exact oposite category? SEI perhaps, i.e. search engine invisibility.

  2. Rea Maor Says:

    Good thinking, i’ve actually changed the original category from SEO to Search Engines in General.

  3. Tim Says:

    Sweet!

  4. Online TV Says:

    Good list, Dogpile is more a listing of sites from various search engines and you can see how each one stacks sites in relation, but it’s still not going to reach the invisible or dark web.

  5. Bob F Says:

    Check out the Open Archive Initiative search engine at http://oaister.org. This is a really superb academic resource with links to almost 12 million digital artifacts from 850 institutions.

  6. nm Says:

    Check also http://www.50matches.com/ Which only crawls web sites that were bookmarked or voted for by people, in sites like del.icio.us, digg and reddit.

  7. Rea Maor Says:

    Wow, i got digged – Amazing.

  8. christo Says:

    Another mechanism which beats even search engines is dotcut (www.dotcut.com) – dotcut finds ‘dark’ material by only searching for pre-registered search terms. It them indexes thousands of sites every day and notifies you of any new results – long before the search engines even come across them.

    christo

  9. I Only Wish » 8 Ways for Searching the Dark Web Beyond Google Says:

    [...] read more | digg story [...]

  10. Balakumar Muthu Says:

    good links! thanks

  11. Jose Says:

    You forgot Technorati for the blog searching, it offers multiple search options.

  12. Donna Says:

    Nice post! Thought I’d share my 2 cents … I’ve just started using CoReap.com which enhances Google and Yahoo web search for social bookmarking. Quite neat!

  13. EmmanuelOga Says:

    “Intute is the only search engine which uses only web pages quality-checked by human researchers – guaranteeing that you

  14. Register Square » Search without Google? Says:

    [...] Now Rea Maor suggests 8 ways of searching the web outside of Google. [...]

  15. How to Find Things Outside of Google’s Reach « Linux and Unix Links of interest Says:

    [...] of Google’s Reach Posted July 4, 2007 What to do when things you are searching for don’t show up in Google, but they do exist on the web.

  16. Prime News Blog » Blog Archive » wonder woman hentai 8 Ways for Searching the Dark Web Beyond Google Says:

    [...] “rabbit holes” into that vast, uncharted territory! wonder woman comic cover art, matureread more | digg [...]

  17. qwerty Says:

    http://oedb.org/library/college-basics/research-beyond-google

  18. 8 Ways for Searching the Dark Web Beyond Google « Computers Zone Says:

    [...] read more | digg story Posted by computerszone Filed in [...]

  19. 8 Ways for Searching the Dark Web Beyond Google « Technology Blog Says:

    [...] read more | digg story Posted by technologyblogs Filed in Uncategorized [...]

  20. Coop Search Says:

    great list of engines

  21. Weird Daily » Blog Archive » 8 Ways For Searching The Dark Web Says:

    [...] Google this, Google that, Google something else. But there

  22. Charles Lau - Internet Business Review Says:

    Great insight about it! These search engines do have their niche when we use it..

  23. Rea Maor Says:

    i dont like Dmoz and they dont like me back…

  24. Marah Marie Says:

    “Note that the whole thing behind the dark web is that it is mostly made of sites that are one of (a) spam sites rejected for quality (b) personal home pages, bulletin board archives, and other stuff not generally of interest to the public (c) academia, which lives in its own world (d) government, which lives in its own world (e) criminal and underground sites that aren

  25. Nerdcore — links for 2007-07-04 Says:

    [...] 8 Ways for Searching the Dark Web – Beyond Google!

  26. The dark web beyond google « Lessons from the Web Says:

    [...] Full article is here [...]

  27. links for 2007-07-05 « notations / lurking chihuahua Says:

    [...] 8 Ways for Searching the Dark Web – Beyond Google!

  28. Mo Says:

    About Intute being the only human-powered search engine, well, there’s another one called Mahalo which is relatively new and entirely run by humans.

  29. Marah Marie Says:

    Mahalo will not list a website or a specific result (PDF document, etc.) unless you submit it to them or they get paid for it. Even if you submit it, it goes under human editorial review, and it’s my understanding very little makes the cut, so to speak. So it’s not a great expansive search engine, but more like Overture on a strict diet.They aren’t listing my site, and until they do, it’s not a good search engine at all :)

  30. syahid ali Says:

    and you can always use “ChaCha” to have humans assisting you in search and err… if you don’t trust machines (sorry, get a little bit carried out becoz of the transformers movie!).

  31. Rea Maor Says:

    That’s ok :) this movie should come with a warning label…

  32. Marah Marie Says:

    The answer isn’t human edited results if you want a better search engine. It costs Mahalo nearly $20 to display just one search result. With that kind of overhead their profits would have to be astronomical for them to compile more than the smallest of indexes. That is truly a dream that can’t come true. How long would the biggest search engines last if displaying just one result cost that much? And who wants human edited search? Yes, edit out the spam and garbage by hand since you must, but let us find the rest of the sites out there without your direction, and without your censorship: your so-called “editorial” control.

    The next big search engine will win by being much like Google, but will crawl the web and using algorithms to determine rank by quality of results, not how many sites did or didn’t link to it, or how much money they paid for their position. I don’t think any privately or publicly held company can ever achieve what I’m thinking of: a more decentralized server run by everybody and nobody (like p2p) that will serve up better results, and be more democratic–without all the ads!

  33. Entradas en las blogosferas.7 - Carrero Bit Says:

    [...] caminos para buscar en las webs oscura, sin [...]

  34. Ck Says:

    What about astalavista.box.sk? :wink:

  35. A Very Many Things « citizen geek Says:

    [...] 8 ways of searching the areas of the Internet long-forsaken by Google. I personally like Clusty. [...]

  36. How to rip Netflix, modern hobos, LSD & the CIA, the 11th dimension and other random links « Hope2012 Says:

    [...] Alternatives to Google (AskReaMaor) [...]

  37. pete Says:

    I think it will be interesting if Yahoo team up with MSN

  38. Alpesh Bhalala Says:

    Just to add one more in the list -

    Twing – its a search engine to search forums !!

  39. Abhishek Says:

    can anyone clearly differentiate b/w dark web & deep web.?

Leave a Reply