Idempotency keys: what broke before I learned to key on replays
Early on, my webhook-catch tool stored events in memory and happily accepted every POST. Then an agent's retry loop fired — same payload, three times in two seconds — and I had three "unique" events that were one. The fix wasn't better dedup logic bolted on later; it was treating every mutating endpoint as replayable from day one. What I actually learned, after watching agents retry aggressively when a response times out: Key on the caller's intent, not the payload. Agents generate their own idempotency key per logical action, and the server stores key → response for a TTL. Identical key, return the stored response instead of re-executing. My short-link creator does this: a retried create with the same key returns the same link instead of minting duplicates. Return the original response, not a fresh one. If the first call succeeded but the client never saw the response, the retry must get byte-identical data (same slug, same manage secret), or downstream stat...