I, robot

Steve (steve@mordred.gatech.edu)
Mon, 11 Dec 1995 15:16:30 -0500 (EST)


Hey all,

I may have missed it in the archives, but I didn't see any topics on which
sections of a document a robot or spider or other Web traveller might examine
to determine content information.

I suppose that this sort of information could be contained in the <META> tag,
like

<META HTTP-EQUIV="Keywords" CONTENT="Audio/Video Equipment">
<META HTTP-EQUIV="Keywords" CONTENT="Laserdiscs sold here!">

but is this indeed what robots look for besides (I guess) the first few lines
of the document itself?

If there isn't a specific indexing method that is standard for all search
engines/robots/crawlers, shouldn't there be one?

thanks,

steve

_____________________________________________________________________________
Can't think of anything right now, other than the traditional e-mail address:
steve@mordred.gatech.edu