Directory and Robot Search Engines

The differences between a directory and a robot search engine

It is all too common for people to generalise the term "search engine" to refer to one of the many directories available to web surfers.

Directory search engines

In a directory, websites are categorised like a library. One of the most famous of all web directories is Yahoo. Yahoo and other directories may offer a search facility, however the primary goal is to categorise the web pages, usually by their subject.

Robot search engines

A robot search engine is generally automated and uses what's known as a "spider" or a "crawler". A spider/crawler is an automated programme that follows links from one web page to another and then reports back to the parent search engine with the results found.

The search engine spider will read through the HTML content of the pages it comes across and look for particular key words, phrases and other elements, which it will consider when building a searchable index.