[geeklog-devel] Yahoo bot becoming a pain

[Tom Willett]

Just looked and not a problem here. 

Out of 315,894 requests only 7,098 were slurp.

On 6/7/2007 8:57 AM, Dirk Haun wrote:
> Is anyone else seeing excessive visits from Yahoo's Slurp bot on their sites?
>
> I've been watching this on geeklog.net for a while now but never really
> measured it until now.
>
> At one point today, of 44 active sessions, 30 were for instances of the
> Slurp bot. Which means that within the last 120 seconds, 30 instances of
> that bot had visited the site from 30 different IP addresses.
>
> About 30% of all the HTTP requests in today's logfile (up until now) are
> from Slurp.
>
> This bot has gone completely bonkers ...
>
> I tried adding a Crawl-delay directive to the robots.txt weeks ago (as
> suggested by Yahoo's FAQ) but that didn't do anything. I guess it only
> slows down individual instances.
>
> I have now sent an enquiry via their website. Just wanted to check if
> anyone else had the same problem.
>
> And now you know why the site may appear slow at times ...
>
> bye, Dirk
>
>
>   

-- 

Tom Willett
tomw at pigstye.net