Uploaded image for project: 'MetaBrainz Hosting'
  1. MetaBrainz Hosting
  2. MBH-461

Add rate limits for crawlers

XMLWordPrintable

      Also while looking at the logs that bitmap provided I see a lot of requests coming from:

      1. Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp
      2. bingbot/2.0; +http://www.bing.com/bingbot.htm

      Give them 1 request/sec while we sort our shit out.

            zas Zas
            rob Robert Kaye
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:

                Version Package