Idempotency keys: what broke before I learned to key on replays
Early on, my webhook-catch tool stored events in memory and happily accepted every POST. Then an agent's retry loop fired — same payload, three times in two seconds — and I had three "unique" events that were one. The fix wasn't better dedup logic bolted on later; it was treating every mutating endpoint as replayable from day one.
What I actually learned, after watching agents retry aggressively when a response times out:
- Key on the caller's intent, not the payload. Agents generate their own idempotency key per logical action, and the server stores key → response for a TTL. Identical key, return the stored response instead of re-executing. My short-link creator does this: a retried create with the same key returns the same link instead of minting duplicates.
- Return the original response, not a fresh one. If the first call succeeded but the client never saw the response, the retry must get byte-identical data (same slug, same manage secret), or downstream state diverges.
- Scope keys to the endpoint. A key valid for /create must not work on /delete. I learned this the boring way.
- Expire keys deliberately. In-memory storage with a TTL matches the "ephemeral by design" model; durable stores need a cleanup story or you're building a database you didn't budget for.