got my invoice from the dc and the 95th percentile number is way up this month, enough that i actually noticed. problem is my own graphs dont show an obvious spike anywhere, mrtg is smooth as glass. im wondering if its the screenshot/thumbnail regen job hammering outbound at 3am when i wasnt looking, or a scraper. anyone got a clean way to line up the carrier 95th with your own netflow so you can actually blame something, right now im just guessing and that feels bad.
Ninety-fifth percentile is sampled every five minutes, so a single thirty minute burst at 3am is enough to set the whole month if it lands in the top five percent of samples. Pull your switch port counters at the same five minute granularity and overlay them. MRTG averaging will hide exactly the burst you are looking for.
yeah mrtg is averaging you into a lie. switch to per interface netflow and sort by top talker for that 3am window specifically. bet its your regen job pulling originals cross region.
cross region is a good shout, the regen worker does pull from the master box in the other dc. so im paying egress twice basically. ugh.
we moved the thumbnail worker onto the same box as the master storage for exactly this reason. egress between dcs is where the 95th lives for a lot of people and nobody looks because the job runs at night.
ask your dc if they can give you the raw sflow samples, some will hand them over. then you dont have to reconstruct the window, you just match timestamps and the culprit falls out.
didnt know theyd give out sflow, will ask. thanks all, at least i know where to point the flashlight now.
One more for next month. Cap the regen job concurrency and add a small sleep between fetches. You lose nothing at 3am and it flattens the burst right out of the billing window.
update, it was the regen job plus a half broken retry loop re pulling the same files over and over. capped concurrency and the overnight line is flat now. invoice will tell the real story next month but the graph already looks saner.