Retries & Exponential Backoff: Designing a Webhook Consumer That Survives

Retries Are Not Optional

Networks fail. Endpoints restart mid-deploy. A downstream database hits a lock and the request times out. None of these are bugs in your code, and all of them will happen to a webhook that runs long enough. The question is never whether a delivery fails. It's what happens on the next attempt.

A webhook consumer that survives is one built around a simple assumption: any given request might be a retry of one you already saw, and any given response might be a lie. Design for that and the failures stop mattering. Ignore it and you get duplicate charges, dropped events, and 3 a.m. pages.

This post covers both halves. First, how to make the receiver survive. Then, how a good relay handles the outbound retries so your endpoint mostly just has to be fast and idempotent.

Why a Failed Delivery Should Be Retried, Not Dropped

Most delivery failures are transient. A 503 during a rolling deploy clears in seconds. A connection reset from an overloaded load balancer clears when traffic ebbs. Drop the event on the first failure and you've turned a two-second blip into permanent data loss.

But not every failure is worth retrying:

• 5xx and connection errors are the retry sweet spot. The sender did nothing wrong; the receiver was momentarily unavailable.
• 408 and 429 mean "try again later" explicitly. Retry, but respect the timing (more on 429 below).
• 4xx (other than 408/429) usually means the request will never succeed. A malformed body or a rejected signature won't fix itself on attempt two. Retrying a 400 just burns your budget and delays the alert you actually need.

The rule: retry on failures that a later attempt could plausibly fix. Fail fast on the ones it can't.

Fixed Backoff Is a Trap

The naive retry loop waits a fixed interval — retry every 5 seconds, five times. It works in a demo and breaks in production for two reasons.

First, if the endpoint is down, a fixed interval hammers it with a steady stream of requests exactly when it's least able to cope. Second, when many senders retry on the same fixed schedule, they synchronize into a thundering herd — every failed request comes back at the same instant, the recovering endpoint gets a spike, falls over again, and the cycle repeats.

Exponential backoff fixes the first problem. Each attempt waits longer than the last, giving a struggling endpoint room to recover:

attempt 1 → wait 1s
attempt 2 → wait 2s
attempt 3 → wait 4s
attempt 4 → wait 8s
attempt 5 → wait 16s

You double the delay each time, capped at some ceiling so you don't wait an hour between attempts.

Add Jitter or You Haven't Fixed the Herd

Exponential backoff alone still synchronizes. If a thousand deliveries all fail at the same moment, they all wait 1s, then all wait 2s, then all wait 4s — the spikes just get further apart. Jitter is randomness added to each delay so the retries spread out instead of clumping:

function backoffWithJitter(attempt, { base = 1000, cap = 30000 } = {}) {
  const exponential = Math.min(cap, base * 2 ** attempt);
  // full jitter: pick a random point in [0, exponential]
  return Math.random() * exponential;
}

That's "full jitter." A more conservative variant keeps half the delay fixed and randomizes the rest (exponential/2 + random(exponential/2)) so you never retry too eagerly. Either way, the goal is the same: two deliveries that failed together should not retry together.

The single highest-leverage change most retry loops need is not more attempts — it's jitter. Backoff spaces retries out over time; jitter spaces them out across senders. You want both.

If you want to see how a specific attempt schedule plays out — total window, worst-case delay, attempt count — run the numbers through the retry calculator instead of guessing.

Retry Budgets and Knowing When to Quit

Retrying forever is its own failure mode. An endpoint that's been down for an hour isn't coming back in the next 200ms, and a delivery queue that never drains its failures will eventually drown the live traffic. You need a retry budget: a hard cap on attempts, a total time window, or both.

Pick the cap based on how long a realistic outage lasts for your consumer, not an arbitrary number. Five attempts with exponential backoff and a 30-second cap covers a couple of minutes of downtime — enough for a deploy or a transient blip. Surviving a multi-hour outage is a different problem, and the answer to that problem is durability (store the event, replay it later), not a longer retry loop.

When the budget is exhausted, the delivery is not silently dropped. It goes somewhere you can find it. That somewhere is the dead-letter queue.

Making Retries Safe: Idempotency and Dedup

Here's the uncomfortable truth about retries: a retry is only safe if the operation it repeats is safe to repeat. If your handler charges a card and then times out returning 200, the sender never sees the success — so it retries, and you charge the card again. The delivery "failed," but the work succeeded. Retries turn every non-idempotent side effect into a double-execution bug.

Two defenses, and you want both:

1. Idempotency keys. Derive a stable key for each event (the provider's event id, or a hash of provider:resource_id:event_type) and record it atomically before you act. If you've seen the key, you no-op. This is the mechanism that makes a retry a no-op instead of a duplicate — see the idempotency keys guide for the atomic check-and-store pattern that avoids the obvious race.
2. Deduplication at the edge. Catch duplicates before they reach your business logic at all. Webhook deduplication collapses repeated deliveries of the same event so your handler only runs once, no matter how many times the sender tries.

Idempotency is the guarantee; dedup is the optimization that keeps duplicates from ever touching your code.

Timeouts Are Retries in Disguise

The most common cause of a "failed" delivery isn't an error — it's a timeout. Your handler took too long, the sender gave up waiting, and it will retry an event your code is still processing. Now you have two invocations racing on the same event.

This is why the survival pattern is acknowledge fast, work later: verify the signature, persist the raw payload, return 200 in milliseconds, then do the real work off a queue you control. A handler that returns in 20ms almost never times out, which means it almost never gets retried for the wrong reason. If you're seeing mystery duplicate deliveries, the cause is usually a slow handler — why webhooks time out breaks down where the milliseconds go.

Respect Retry-After and 429

When an endpoint returns 429 Too Many Requests, it's telling you the exact thing your backoff is trying to guess: slow down. Many senders (and Hookbase) honor a Retry-After header on 429 and 503 responses, which can specify either seconds or an HTTP date.

If the server hands you a Retry-After, use it in place of your computed backoff — it's a real signal, not an estimate. If it's absent, fall back to exponential backoff with jitter. On the receiving side, returning 429 with a Retry-After during overload is far better than accepting requests you'll drop; it turns a cascade into an orderly wait.

Poison Messages and the Dead-Letter Queue

Some deliveries never succeed no matter how many times you retry — a payload your handler can't parse, an event referencing a record that was deleted, a bug that 500s on one specific shape. These are poison messages, and the danger is that a retry loop keeps them in circulation forever, wasting attempts and hiding real failures.

The dead-letter queue is where a message goes when it exhausts its retry budget. The DLQ does three jobs:

• Contains the damage. A poison message stops consuming retry capacity the moment it's parked.
• Preserves the event. Nothing is lost; it's held for inspection and later replay.
• Surfaces the problem. A growing DLQ is a signal — one you can alert on — instead of a silent drop.

A delivery pipeline without a DLQ has only two options for a message it can't deliver: retry it forever, or throw it away. Both are wrong.

Where Hookbase Fits

Everything above is work you'd otherwise build and maintain per consumer. On the outbound side, Hookbase does it for you. When Hookbase delivers to your destination, it retries automatically with exponential backoff, up to 5 attempts, and if all of them fail it routes the message to a dead-letter queue where you can inspect it, fix the cause, and replay — or replay with an edited payload — once the endpoint is healthy again.

That inverts the design problem. Your endpoint doesn't need a bespoke retry engine, a jitter function, or a poison-message parking lot. It needs to do two things well:

• Be fast. Acknowledge quickly so you don't time out and trigger avoidable retries.
• Be idempotent. Because retries will happen, treat a repeated event as a no-op.

Different senders behave very differently once you're the one receiving their retries — some give up after one failure, others hammer you for days. The retries by provider reference covers who does what, which is worth knowing even when a relay is absorbing the outbound half for you.

The Short Version

Retries are the price of running on an unreliable network, and the tools for paying it cheaply are well understood: exponential backoff so you don't hammer a struggling endpoint, jitter so retries don't synchronize, a bounded budget so you eventually stop, idempotency so repeats are harmless, and a dead-letter queue so nothing gets lost when you do give up. Build those into a receiver and it survives failures that take down naive consumers. Put a relay in front that already handles the outbound half, and your endpoint only has to guarantee the two things it actually controls: answer fast, and never do the same work twice.