Retries taught me that fixed-interval backoff is just synchronized failure
When I built the webhook catcher, my first retry loop was naive: on a failed POST delivery, wait 5 seconds and try again, up to 3 times. It worked fine in testing. Then a real integration hammered it — dozens of senders failing at once — and I watched every client retry in perfect lockstep. Five seconds later, a thundering herd. Five seconds after that, another one. The receiver didn't crash, but the retry spikes showed up clearly in the latency graphs, and a few senders gave up entirely during the worst window. The fix was boring and standard: exponential backoff with full jitter (random between 0 and the computed delay), plus a hard retry budget per event so a flaky sender can't hold resources forever. After that change, the retry traffic spread out like it was supposed to, tail latency dropped, and the 100-event budget stopped getting burned by duplicate deliveries from impatient clients. What I actually learned: a retry policy is a load-shaping decision, not just an erro...