so i had a nightly cron that regenerates my sitemap and clears a couple of caches, and it just quietly stopped running about a week ago. no error, no alert, nothing in my inbox. i only noticed because a page that shouldve updated didnt. turns out i renamed the script and forgot the crontab still pointed at the old path, so cron was firing into the void every night and failing silent. the part that bugs me is i had zero visibility that it stopped. how do you all make sure your scheduled jobs are actually running, not just assume they are because they used to?
the fix i put on everything after getting bitten the same way is a dead mans switch. the job pings a little monitor url at the end of a successful run, and if the monitor doesnt hear from it inside the expected window it pings me instead. you stop watching for the job to fail and start watching for it to go quiet, which is the failure mode you actually hit. cron failing loud is easy, cron silently not running at all is the one that gets you.
this is the exact same disease as untested backups. everyone has the job, almost nobody checks it actually did the thing. i learned it when a backup cron had been failing for eleven days and i found out the day i needed the backup. now every scheduled job has to prove it ran by leaving evidence i monitor, a row in a table, a touched file, a heartbeat, something. if the evidence is stale, i get woken up. the job existing is not the same as the job running.
this is the whole trick and almost nobody does it until they get burned once. i had a failover health check that itself died and i had no idea my safety net was gone until the primary went down and nothing caught it. monitor the monitor. i keep it dumb now, a cheap external pinger that yells if the heartbeat stops, living nowhere near the box it watches so one dead machine cant take the alarm down with it.
yeah that line is going on a sticky note. i think my mistake was treating the crontab as set and forget, like once it worked it would keep working forever. spent tonight adding a heartbeat row to a little table and a check that screams if it goes stale. feels dumb that i needed to get silently burned to do the obvious thing but here we are.
as the resident person who over builds everything before shipping, i will say the dead mans switch is the rare case where the simple version is genuinely enough. you dont need a whole observability stack for a handful of cron jobs. one heartbeat per job and one external thing watching the heartbeats covers like 90 percent of this. i wasted a weekend wiring up something fancy for three cron jobs and the cheap pinger would have done it.
same thing saved me on my one tube. fwiw you dont even need a service for it, a cron that writes a timestamp to a file and a second check that yells if the file goes stale is enough for a small setup. kept it that dumb for two years and it has caught two dead jobs i wouldnt have spotted for weeks otherwise.
the angle i would add is a silently dead job is a security problem too, not just an ops one. i had a cron that rotated some credentials quietly stop firing and didnt notice, so the old ones just kept living long past when they should have. the jobs i watch hardest now are the ones whose whole point is to expire or clean something up, because when those go quiet nothing looks broken, it just slowly rots in the background where you cant see it.
a cron that stops is usually one of three things, the script path moved, the binary isnt in crons stripped down env the way it is in your shell, or the box rebooted and something didnt come back up. that middle one bites people who tested the command by hand and assumed cron would run it identically. log stdout and stderr to a file from the crontab line itself and most of the mystery disappears the next time it happens.
i get this bug one time and the reason was exactly this, the box reboot and my cron user service not come back. now i do small thing, the job write the finish time into a file, and my status page read that file and show me when last it run. so if the number is old i see it in the morning without wait for something to break. is not fancy, just a timestamp i actually look at. the dead man switch is better but this one i build in ten minute.
same here, had a thumbnail cleanup cron die for a month and only noticed when the disk started filling. fwiw the thing that fixed it for me was dead simple, the job touches a file at the end and a separate tiny check screams if that file is older than it should be. a cron that watches the cron. ugly but i havent been surprised since.
This is the exact thing that made me build a ‘did it run’ dashboard. A cron that fails loud is fine, you see it. A cron that just stops is the dangerous one, nothing errors, the work quietly does not happen.
Cheapest fix that works for me: have the job touch a timestamp file or ping a dead-man URL on success, then alert when that timestamp goes stale. You stop trusting that it ran and start checking that it did.