[geeklog-devel] geeklog-devel Digest, Vol 3, Issue 6

[Michael Brusletten]

I have seen an increase in the daily amounts from yahoo on a few hundred hits to over 2000 hits daily, all coming from different IPs
of theirs.  My site now tops over 5000 hits total daily, and sometimes more.

Michael

----- Original Message ----- >
> Message: 7
> Date: Thu, 7 Jun 2007 14:57:55 +0200
> From: "Dirk Haun" <dirk at haun-online.de>
> Subject: [geeklog-devel] Yahoo bot becoming a pain
> To: geeklog-devel <geeklog-devel at lists.geeklog.net>
> Message-ID: <20070607125755.1501718254 at smtp.haun-online.de>
> Content-Type: text/plain; charset=US-ASCII
>
> Is anyone else seeing excessive visits from Yahoo's Slurp bot on their sites?
>
> I've been watching this on geeklog.net for a while now but never really
> measured it until now.
>
> At one point today, of 44 active sessions, 30 were for instances of the
> Slurp bot. Which means that within the last 120 seconds, 30 instances of
> that bot had visited the site from 30 different IP addresses.
>
> About 30% of all the HTTP requests in today's logfile (up until now) are
> from Slurp.
>
> This bot has gone completely bonkers ...
>
> I tried adding a Crawl-delay directive to the robots.txt weeks ago (as
> suggested by Yahoo's FAQ) but that didn't do anything. I guess it only
> slows down individual instances.
>
> I have now sent an enquiry via their website. Just wanted to check if
> anyone else had the same problem.
>
> And now you know why the site may appear slow at times ...
>
> bye, Dirk
>
>
> -- 
> http://www.geeklog.net/
> http://geeklog.info/
>
>
>
> ------------------------------
>
> Message: 8
> Date: Thu, 07 Jun 2007 09:28:11 -0400
> From: Tom Willett <tomw at pigstye.net>
> Subject: Re: [geeklog-devel] Yahoo bot becoming a pain
> To: Geeklog Development <geeklog-devel at lists.geeklog.net>
> Message-ID: <466807EB.9010503 at pigstye.net>
> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>
> Just looked and not a problem here.
>
> Out of 315,894 requests only 7,098 were slurp.
>
> On 6/7/2007 8:57 AM, Dirk Haun wrote:
> > Is anyone else seeing excessive visits from Yahoo's Slurp bot on their sites?
> >
> > I've been watching this on geeklog.net for a while now but never really
> > measured it until now.
> >
> > At one point today, of 44 active sessions, 30 were for instances of the
> > Slurp bot. Which means that within the last 120 seconds, 30 instances of
> > that bot had visited the site from 30 different IP addresses.
> >
> > About 30% of all the HTTP requests in today's logfile (up until now) are
> > from Slurp.
> >
> > This bot has gone completely bonkers ...
> >
> > I tried adding a Crawl-delay directive to the robots.txt weeks ago (as
> > suggested by Yahoo's FAQ) but that didn't do anything. I guess it only
> > slows down individual instances.
> >
> > I have now sent an enquiry via their website. Just wanted to check if
> > anyone else had the same problem.
> >
> > And now you know why the site may appear slow at times ...
> >
> > bye, Dirk
> >
> >
> >
>
> -- 
>
> Tom Willett
> tomw at pigstye.net
>
>
>
> ------------------------------
>
> _______________________________________________
> geeklog-devel mailing list
> geeklog-devel at lists.geeklog.net
> http://eight.pairlist.net/mailman/listinfo/geeklog-devel
>
>
> End of geeklog-devel Digest, Vol 3, Issue 6
> *******************************************
>
>