[geeklog-devel] Yahoo bot becoming a pain

[Dirk Haun]

Is anyone else seeing excessive visits from Yahoo's Slurp bot on their sites?

I've been watching this on geeklog.net for a while now but never really
measured it until now.

At one point today, of 44 active sessions, 30 were for instances of the
Slurp bot. Which means that within the last 120 seconds, 30 instances of
that bot had visited the site from 30 different IP addresses.

About 30% of all the HTTP requests in today's logfile (up until now) are
from Slurp.

This bot has gone completely bonkers ...

I tried adding a Crawl-delay directive to the robots.txt weeks ago (as
suggested by Yahoo's FAQ) but that didn't do anything. I guess it only
slows down individual instances.

I have now sent an enquiry via their website. Just wanted to check if
anyone else had the same problem.

And now you know why the site may appear slow at times ...

bye, Dirk


-- 
http://www.geeklog.net/
http://geeklog.info/