[tech] who added these rules?

[Ian McKellar]

On Sat, Nov 27, 1999 at 01:28:20PM +0800, Ben Rampling wrote:
> Me, oddly enough, and we all know how stupid I am. A robots file would've
> almost certainly have been useless, even if a user agent specific to that
> spider, and no other spider, could've been divined.

The spider was htdig - a spider designed for the purpose of indexing local
sites and intranets. Any legitimate global web crawler would have a different
useragent.

Ian

-- 
Ian  McKellar | Email: yakk(a)yakk.net     | Web: http://www.yakk.net/
Fax: +61 (8) 9265 0821 / +0 (775) 205 0307 | Home: +61 (8) 9389 9152
If God didn't want us to eat animals, he wouldn't have made them out of meat.