Search Engine History – Web Search Before Google
second of three posts on the history of the Search Engines, I look at the pioneers of the early search market, including the very first web crawler, WWW Wanderer. Did you know that Disney used to be one of the biggest players in the business? Or that Altavista was more technically advanced, in many ways, in 1998 than Google is now? Read on!
The pioneering Web Search Engines
Really, the point at which modern search engines first begin to appear is after the development and popularisation of the MOSAIC browser in 1993. In 1994, Internet Magazine was launched, together with a review of the top 100 websites billed as the ‘most extensive’ list ever to appear in a magazine. A 28.8Kbps modem was priced at $399 and
At this point and for the next 4-5 years, it was just about possible to produce printed and web-based directories of the best sites and for this to be useful information for consumers. However, the rapid growth in the number of www sites (from 130 in 1993 to over 600,000 in 1996) began to make this endeavour seem as futile as producing a printed yellow pages of all the businesses, media and libraries in the world!
Whilst WAIS was not a lasting success, it did highlight the value of being able to search – and click through to – the full text of documents on multiple internet hosts. The nascent internet magazines and web directories further highlighted the challenge of being able to keep up with an internet which was growing faster than the ability of any human being to catalogue it.
In June 1993, Matthew Gray at MIT developed the PERL-based web crawler, WWW Wanderer. Initially, this was simply devised as a tool to measure the growth of the world wide web by “collecting sites”. Later, however, Gray (who now work
About The Author
