woke up to load average through the roof again. some scraper farm is crawling every thumbnail on the site at like 40 req/s from a rotating pool of residential IPs. i dont want to just block by UA because half of them fake googlebot anyway. currently thinking token bucket at nginx per /24 but that feels crude. how are you all separating the actual crawlers you want from the junk?
verify googlebot by reverse dns, not UA. host the ip and it should resolve back to googlebot.com, then forward resolve matches. anything claiming to be google that fails that gets the bucket. same trick works for bing.
residential rotating is the annoying part. per /24 does basically nothing when theyre spread across thousands of ASNs. i ended up rate limiting by behavior instead, hits to thumbnail and listing paths with no referrer and no cookie get a much lower ceiling.
If the real pain is bandwidth and not CPU, put the thumbnails fully behind the CDN and let the origin breathe. The scrapers can hammer the edge all they like at that point and your box never sees it.
yeah thumbs are already on cdn, its the dynamic listing pages theyre chewing through. reverse dns verify is a good shout though, going to wire that in tonight and see how much of the traffic was fake google.
watch for the ones that crawl slow and polite specifically to stay under your thresholds. those are usually the ones building a full mirror to republish. i log anything that pulls the whole sitemap in order and flag it even if the rate looks fine.
did the rdns thing actually cut it down? curious how much before i bother setting it up on mine.
update for anyone who finds this, rdns verify plus the no referrer no cookie behavior ceiling knocked load back to normal overnight. the fake googlebots were most of the volume. should have done it months ago.
nice, saving this. i did something similar but added a cheap tarpit, the behavior ceiling triggers a slow 200 with a tiny delay instead of a hard block. the polite mirror scrapers hate it because it tanks their crawl rate without ever giving them a clean error to detect and route around.
one thing worth adding on the dynamic pages, cache them for even 30 to 60 seconds for anonymous hits. the scraper farm hammering the same listing pages is exactly the load a short micro cache eats for free, and a real user on a 30s old listing page wont notice. logged in or cookied requests bypass it. cheapest fix before you touch rate limiting at all.