ive got thousands of outbound affiliate and partner links across the site and programs die, get bought, change their url structure constantly. users hit a 404 on the other end and bounce, and worse google eventually notices the pattern. i do a manual spot check but its hopeless at this scale. whats your actual system, a cron that pings every outbound link weekly and flags non-200s into a queue, or something smarter? curious how people handle soft 404s too where it returns 200 but the page is now a parking page
cron plus head requests weekly, log anything not 200 or 301 into a table with a last_checked timestamp. the parking page problem you cant solve with status codes, i keep a little list of known parking domain fingerprints and string match the response body. ugly but catches most of them
I check more often than weekly for the money links and less often for the long tail. A dead link on a top ten page costs me real traffic, a dead link on page 400 can wait. Tier your checks by the page value instead of treating every link the same, it makes the whole thing cheaper to run.
dont ping them all from your origin ip on a schedule, some networks will see a weekly burst of head requests from one ip and get weird about it. spread it out or route through something. learned that one the annoying way when a program flagged my account for bot traffic
this is the bit i was missing, i was only checking status codes. do you share that fingerprint list anywhere or is everyone just building their own
the seo angle, wrap your outbound affiliate links through a redirect you control so when one dies you fix it in one place and every page pointing at it updates instantly. also lets you swap a dead program for a live competitor without editing 300 pages. the redirect layer is the real fix, the checker just tells you when to pull the lever
plus one the redirect layer. built mine so a dead link falls back to a category landing page instead of a 404 while i sort a replacement. user still lands somewhere useful and i dont bleed the click. took an afternoon and saved me countless 404s since
late but the soft 404 thing bit me hard last quarter. a whole network quietly turned their offers into parking and i kept sending traffic for weeks because the status was 200. now i alert on a sudden drop in clickthrough per link too, behaviour catches what status codes miss
dont just check status, check the response body for a size or a keyword match too. a dead offer that 200s with a parking page is usually tiny and has words like domain and buy this domain. a cheap content length threshold catches a lot of them before you even bother with fingerprints.
[quote]do you share that fingerprint list[/quote] happy to point you the right way rather than hand over a list that goes stale. the move is build your own, when you find a parked one grab a unique string from its html. the parking providers reuse the same template across thousands of domains, so one fingerprint catches a whole provider. a dozen of those covers most of the junk.
Store the http status, the final url after redirects, and a content hash on every check. The content hash is what catches the silent swap, when a live offer quietly becomes a different page at the same url and the status stays 200. If the hash changes and the page shrinks, flag it for a human, do not auto remove.
[quote]wrap your outbound affiliate links through a redirect you control[/quote] the redirect layer is the single best thing in this thread. we route everything through our own go domain and the checker only watches the mapping table, not ten thousand pages. one dead target, one row to fix, done. retrofitting it onto an old site is painful but worth every hour.
priority by revenue, not by position. i pull the click log and check the top earners daily, the middle weekly, the long tail monthly. checking a link that made you nothing last quarter on the same schedule as your top earner is just wasting crawl budget and your own attention.
i add also a simple alert, if a link that normally send clicks suddenly send zero for two days, something is broken even if the status look 200. the behaviour tell you faster than the checker sometimes. dead offer still 200 but nobody click through, that is the real signal.
[quote]a whole network quietly turned their offers into parking[/quote] this happened to me with a big network too and the worst part was they kept paying out a trickle so it looked alive in the dashboard. the traffic was going to a parking page but the stat said converting. now i eyeball the actual landing page of my top offers once a month, no tool caught that one.
put the broken link count on a dashboard you actually look at, not in a log you only open when something feels wrong. i have a single number on my morning check, dead links found overnight, and if it spikes i know a whole network died before a user ever emails me about a 404.
late to this but for a smaller site you dont even need cron infrastructure, a simple scheduled script and a sheet output works fine. dont let the fancy table and hashing advice scare you off starting, a weekly head check that emails you the non 200s is already miles better than finding out from an angry user.