Attack of the Content-Sucking Robots!

Attack of the content-sucking robots

Apr 10, 2025 by Roadmaster
Tags: english geeky

I stood up a simple server with nepenthes. I put a link to it in an existing web page (I won’t say which one). In less than 2 weeks this server is handling about 1000 requests/hour, that’s about 20,000/day. This all comes from bots. By far the worst offender is Amazonbot.

          { "ag": "Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)", "hits": 1719 }
          { "ag": "Mozilla/5.0 (compatible; AwarioBot/1.0; +https://awario.com/bots.html)", "hits": 20596 }
          { "ag": "Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html)", "hits": 2912 }
          { "ag": "Mozilla/5.0 (Linux; Android 7.0; SM-G892A Build/NRD90M; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/60.0.3112.107 Mobile Safari/537.36", "hits": 11 }
          { "ag": "Mozilla/5.0 (compatible; MJ12bot/v1.4.8; http://mj12bot.com/)", "hits": 96187 }
          { "ag": "Mozilla/5.0 (compatible; MJ12bot/v2.0.0; http://mj12bot.com/)", "hits": 3103 }
          { "ag": "Mozilla/5.0 (compatible; DotBot/1.2; +https://opensiteexplorer.org/dotbot; help@moz.com)", "hits": 16 }
          { "ag": "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot) Chrome/119.0.6045.214 Safari/537.36", "hits": 174659 }
          { "ag": "Mozilla/5.0 (iPhone; CPU iPhone OS 13_2_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.3 Mobile/15E148 Safari/604.1", "hits": 89 }