Re: Searchable Indexes: LISTEN NOW!

Tim Berners-Lee (timbl@www3.cern.ch)
Thu, 1 Jul 93 11:38:34 +0200


95 registered servers (level 0)
99 servers refered to by level 0 (level 1)
172 servers referred to by level 1 (level 2)
174 distinct servers in levels 0-2.

Going this deep takes long enough (an hour or so). I use a filter at
each stage to cut out known slow sites (typically Eastern Europe) or
known buggy servers. I also have to clean the links quite a bit for
references with local hostnames only (not FQDN) and a small
amount of junk. [Obviously mailing the webmaster at sites
with bad links would be a possibility.]

This is only for interest. I don't generate a index. The engine is
just a bunch of scripts using www -listrefs.

Tim