SEO : How Search Engines Work

A search engine sends out an online software program known as a "spider" or "crawler" which follows links from site to site, sucking up page content and storing it back in the search engine database.

A spider does not see a website like a human does. It is not interested in nicely formatted content. It consumes raw content - text, images, and video. When it has digested all text, images, and video on a page it follows any links it finds to other pages, or other sites, to get some more text, images, and video. It never sleeps and, if it thinks the site is popular, comes back regularly to check for updated content.

To see how a spider views your website, look at your site using a text-only browser such as Lynx (available free from http://lynx.isc.org )

Later, when a searcher types in some keywords, the search engine compares the keywords to the raw content in its database and shows what it considers to be the most relevant results, with the most relevant or most popular site at the top. By each search result it puts a link back to the original nicely formatted content that the spider originally visited.

You can track how often a spider has visited your website by looking at your website server log, or visitor tracking program.

The Web from a search engine spider's perspective

95% of us use IE6/7 or Firefox to browse the web, which interprets HTML code such as this:

and show us a nicely formatted page like this:

But we tend to forget that a search engine spider - a basic text browser without "eyes" - sees the same page like this:

The spider can only read text (either the readable text on the page or the text contained behind the scenes in image tags and comments). A spider cannot see the formatted styles, images, animation or any kind of visual effect.

And it can only discover new pages by following the hyperlinks from page to page... so if if can't find a link then it can't continue on its journey.

	Do... download the Lynx text browser and see for yourself how Google sees your website (press 'g' then type in your URL and hit enter)
	Don't... assume that just because your page looks great to people that it will look great to a search engine.

lynx.isc.org - Lynx text browser