Long, long ago on a planet far, far (oops wrong story)…
So, long, long ago when the World Wide Web (WWW) started to get the public’s attention and the dotcom gold rush was in full swing, people had trouble finding things on the internet. Much like a very large library, people were getting lost and started to turn to some very smart folks to help them find what they were looking for, yes librarians. Now this new approach followed the first few indexing attempts, the bots and web crawlers.
Primitive Web Search:
By December of 1993, three full fledged bot fed search engines had surfaced on the web: JumpStation, the World Wide Web Worm, and the Repository-Based Software Engineering (RBSE) spider. JumpStation gathered info about the title and header from Web pages and retrieved these using a simple linear search. As the web grew, JumpStation slowed to a stop. The WWW Worm indexed titles and URL’s. The problem with JumpStation and the World Wide Web Worm is that they listed results in the order that they found them, and provided no discrimination. The RSBE spider did implement a ranking system. Source http://www.searchenginehistory.com/
You can read a great deal more about the history of search from The Search by John Batelle. Here are another couple of good resources to get a primer on the history of search
http://www.webreference.com/authoring/search_history/
http://www.searchenginehistory.com/
http://en.wikipedia.org/wiki/Web_search_engine
So, most folks agree that one of the early search companies, Yahoo!, became the king of search for quite a while, in internet years that is… Search might be a misnomer. Many experts would define the initial Yahoo! as a directory versus a search engine. Unlike Google, the current rival, Yahoo! had relied on humans, not Larry’s Page Rank or algorithms to index the WWW. ?Here is the Yahoo! story.? While a human curated directory has many advantages to the machine, it can also create problems. Think sex, power and politics. I remember some “hub-bub” about where to put “Jews for Jesus.” Does that get listed in the Christian or Jewish religion section of the directoy?
Now fast forward 14 years. While Google continues to dominate search, some new trends are a foot. The Wired

article captures a good chuck of the “new, new” thing. Human curated web index, either by paid staff or user created content models. Some examples are:
Mahalo
Cha Cha
Squidoo
Funny, that sounds like a song I’ve heard before. Yahoo!, not to mention DMOZ and several other projects, were already relying on human curators. Here are a few more that I have known about for a long time and/or I just grabbed from a quick search…
Blogged
Rollyo
About.com
Thanks to Dahn Tamir for suggesting www.mturk.com that has a whole different direction on human powerd “answers”.
Of course we can not ignore all the noise about semantic search. Danny Sullivan [need to find article] gives a great review of the last decade of unmet expectations. Then there is the latest news like Microsoft buying Powerset and FAST. Not to mention some significant developments for companies like Cognition Siderean and Zoominfo Cycorp
So which will it be, humans or the machine?
I say, why not both… using a smart machine combined with input, feedback and validation from subject matter experts might deliver the best search experience yet.
Danny Sullivan does a great job of reviewing some of what’s what Search 4.0: Putting Humans Back In Search
- Search 1.0 (1996): Pages ranked using “on-the-page” criteria
- Search 2.0 (1998): Pages ranked using “off-the-page” criteria
- Search 3.0 (2007): Vertical search results blended into regular search results
Well, we will all continue to use the search tool that gives us the best results for our own needs.
Bring it on… human, machine, personal search – just help me find what I’m looking for!