Idempotency is the feature

A payment request times out after twenty seconds. Did it go through?

You cannot know. The timeout tells you that you stopped waiting, not what happened at the other end. The provider may have declined it, may have processed it and failed to respond, or may be processing it right now.

Two options: retry and risk charging twice, or do not retry and risk not charging at all. Both are wrong. The correct answer is to make retrying safe, and that has to be designed in before the timeout happens.

Networks fail in the ambiguous direction

Distributed systems fail in a specific and inconvenient way. The request usually arrives. The response is what gets lost.

This means duplicate delivery is not an edge case to be prevented, it is a normal operating condition to be absorbed. Any integration running long enough will send the same logical request more than once — through client retries, queue redelivery, a user double-clicking, or a provider's own retry policy.

The question is never "how do we stop duplicates". It is "what happens when we get one".

Idempotency keys

An idempotency key is a client-generated identifier for a logical operation. The server records it with the outcome. If the same key arrives again, the server returns the original result instead of doing the work twice.

Three details matter and are frequently got wrong.

The client generates the key, and it identifies intent. It must be created once, when the user decides to pay, and reused across every retry of that attempt. A key generated per HTTP request provides nothing — each retry gets a fresh key and creates a fresh charge.

Store the response, not just the key. A duplicate request should return the original response body and status. Returning a bare 409 forces the caller to work out what happened, which they will do by querying and guessing.

Record the key in the same transaction as the effect. If the charge commits and the key write fails, you have a charge with no record of it — the exact scenario the mechanism was meant to prevent.

Classify failures before retrying them

Retrying everything is nearly as bad as retrying nothing. It burns your retry budget on requests that will never succeed and delays the ones that would.

• 429 with Retry-After — wait exactly that long. The provider has told you the answer.
• 500, 502, 503, 504 — transient. Retry with exponential backoff.
• Timeout, connection reset — unknown outcome. Retry, but only if the request is idempotent.
• 400, 422 — permanent. The request is malformed; retrying changes nothing.
• 401, 403 — permanent. Credentials will not fix themselves.

That distinction between transient and permanent is most of the value. Retrying a validation error politely, sixteen times, with exponential backoff, is still retrying a validation error.

Jitter is not optional

Exponential backoff without randomisation synchronises your retries. Every request that failed during an outage retries at the same instant, and the provider that just came back goes down again — this time because of you.

Add randomness to every backoff interval. It is one line and it prevents a failure mode that is genuinely difficult to diagnose after the fact.

Webhooks: you are on the other side of this

When you receive webhooks, you are the system being retried. Assume duplicates.

Verify the signature, and compare it in constant time. An unsigned webhook endpoint is an unauthenticated write endpoint.

Check the timestamp and reject anything outside a short window, or a captured payload can be replayed indefinitely.

Acknowledge fast, process asynchronously. Validate, enqueue, return 200. Providers time out quickly and treat slowness as failure — so doing the work inline means being sent it again while still working on it.

Deduplicate on the provider's event ID, not on payload content. Payloads may differ trivially between deliveries of the same event.

Do not assume ordering. Events arrive out of order routinely. Carry a version or timestamp and ignore anything older than what you have already applied.

Reconciliation is not optional either

Even with all of the above, states diverge. A webhook is lost. A retry exhausts its budget. A provider has an incident and replays three hours of events at once.

Run a scheduled reconciliation that compares your records against the provider's for a recent window and reports differences. It will find things. The first run of one of these almost always finds something, which is exactly why it is worth having.

Store the whole attempt

Log every attempt with its request, response, status and timing — with sensitive fields redacted and a size cap.

Status codes alone are not enough. "Was this delivered?" needs to be a lookup, not an investigation. The first time a provider insists they never received something, this pays for every byte it has cost.

The summary

Idempotency is not defensive plumbing added after an incident. It is the design property that determines whether your integration degrades safely or corrupts data under conditions that are certain to occur.

Design for the duplicate. It is coming.