[tech] who added these rules?

Mark Tearle mtearle at tartarus.uwa.edu.au
Sat Nov 27 22:52:04 WST 1999


On Sat, 27 Nov 1999, David Manchester wrote:

> Well, presumably someone in UCC and on staff...
> 130.95.128.65=testbox.ucs ... ..128.65=erebus.ucs
> 

Yes, I use erebus as the machine that I run my web browser on,
it's amazing how slow Netscape still is on a machine that fast.


On Sat, 27 Nov 1999, Ben Rampling wrote:

> Date: Sat, 27 Nov 1999 13:28:20 +0800 (WST)
> From: Ben Rampling <ben at ucc.gu.uwa.edu.au>
> To: tech at ucc.gu.uwa.edu.au
> Subject: Re: [tech] who added these rules?
> 
> > On Sat, Nov 27, 1999 at 10:34:55AM +0800, Mark Tearle wrote:
> > > Hi
> > > 
> > > Who was the moron who added these rules on mako?
> > > 
> > > 00050 deny ip from 130.95.128.65 to 130.95.13.11
> > > 00050 deny ip from 130.95.13.11 to 130.95.128.65
> > > 
> > > robots.txt is the write way to do it
                        ^^^ correct
> > 
> > I assume some idiot who is too slack to:
> > 1) Discuss it with wheel and
> > 2) Actually be aware of the robot exclusion standards.
> 
> Me, oddly enough, and we all know how stupid I am. A robots file would've
> almost certainly have been useless, even if a user agent specific to that
> spider, and no other spider, could've been divined.
> 
> Regards,
> Ben Rampling

Why Ben?  The code under htdig/Server.cc on erebus certainly looks like 
it checks the robots.txt file.  
To quote Chris Macdonald  - "This answer was bullshit."

On Sat, 27 Nov 1999, Ian McKellar wrote:

> On Sat, Nov 27, 1999 at 01:28:20PM +0800, Ben Rampling wrote:
> > Me, oddly enough, and we all know how stupid I am. A robots file would've
> > almost certainly have been useless, even if a user agent specific to that
> > spider, and no other spider, could've been divined.
> 
> The spider was htdig - a spider designed for the purpose of indexing local
> sites and intranets. Any legitimate global web crawler would have a different
> useragent.
> 
> Ian
> 

Yours
Mark
-- 
Mark Tearle - mtearle at tartarus.uwa.edu.au

Captain Bipto: We did win, didn't we?
Blaznee: No, but if we think fast enough we might just live to lie about it.





More information about the tech mailing list