Webhook Deduplication: Never Process the Same Event Twice

The Problem with Webhook Retries

If you've worked with webhooks, you've likely encountered this scenario: a network hiccup causes a provider to retry a webhook, and suddenly your system processes the same order twice, sends duplicate notifications, or creates duplicate records.

Webhook providers are designed to be reliable. When they don't receive a successful response, they retry. Stripe, for example, retries webhooks every 5 minutes for up to 3 days. This reliability is a feature, not a bug:but it creates a challenge for your backend.

Without deduplication, each retry looks like a new event:

• Duplicate charges processed for the same order
• Multiple notification emails sent for a single event
• Inventory counted twice for one purchase
• Analytics skewed by phantom events

The traditional solution? Write custom deduplication logic for every webhook source. Store event IDs, manage expiration, handle race conditions. It's tedious, error-prone, and different for every provider.

The Solution: Automatic Deduplication

Hookbase now handles all of this automatically. When you enable deduplication on a source, every incoming webhook is checked against recent events before processing. Duplicates are detected in sub-millisecond time and flagged:no deliveries are triggered, but a full audit trail is maintained.

Five Deduplication Strategies

Different use cases call for different approaches. Hookbase supports five strategies:

1. Auto (Default)

The smart choice for most setups. Auto mode uses the provider's event ID when available, falling back to a payload hash when it isn't.

{
  "dedupStrategy": "auto",
  "dedupWindowHours": 24
}

2. Provider ID

For providers with guaranteed unique event identifiers. Hookbase extracts these automatically:

| Provider | ID Source | |----------|-----------| | Stripe | id field (evt_xxx) | | GitHub | X-GitHub-Delivery header | | GitLab | X-GitLab-Delivery header | | Shopify | X-Shopify-Webhook-ID header | | Slack | envelope_id field | | Twilio | X-Twilio-Message-SID header | | SendGrid | X-Message-ID header | | Paddle | event_id field | | Linear | Linear-Delivery header | | Vercel | X-Vercel-ID header | | Discord | id field |

3. Payload Hash

Creates a SHA-256 hash of the entire webhook payload. Two identical payloads within the time window are considered duplicates.

Best for generic webhooks without unique identifiers.

4. Idempotency Key

Uses a custom header for deduplication. Clients can include an idempotency key to ensure exactly-once processing.

{
  "dedupStrategy": "idempotency_key",
  "dedupCustomHeader": "x-idempotency-key"
}

5. None

Disables deduplication entirely. Useful for testing or when you genuinely want to process every webhook.

Configurable Time Windows

The deduplication window determines how long Hookbase remembers event IDs. Configure anywhere from 1 hour to 7 days:

{
  "dedupEnabled": true,
  "dedupStrategy": "auto",
  "dedupWindowHours": 48
}

The default is 24 hours:long enough to catch most retry patterns.

How It Works Under the Hood

When a webhook arrives at Hookbase:

1. Key Generation: Based on your strategy, we generate a deduplication key
2. Cache Lookup: We check Cloudflare KV for the key (sub-millisecond)
3. Duplicate Handling: If found, the event is stored with is_duplicate: true and linked to the original. No deliveries are created.
4. New Event Processing: If not found, we record the key and process normally

Response for Duplicates

When a duplicate is detected, Hookbase returns a 202 response with details:

{
  "success": true,
  "eventId": "evt_new_abc123",
  "originalEventId": "evt_original_xyz789",
  "duplicate": true,
  "message": "Duplicate event detected and stored"
}

Your downstream systems can use originalEventId to trace back to the first occurrence.

Configuration via API

Enable Deduplication on a New Source

curl -X POST https://api.hookbase.app/api/organizations/{orgId}/sources \
  -H "Authorization: Bearer {token}" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Stripe Production",
    "slug": "stripe-prod",
    "provider": "stripe",
    "dedupEnabled": true,
    "dedupStrategy": "auto",
    "dedupWindowHours": 24
  }'

Update an Existing Source

curl -X PATCH https://api.hookbase.app/api/organizations/{orgId}/sources/{sourceId} \
  -H "Authorization: Bearer {token}" \
  -H "Content-Type: application/json" \
  -d '{
    "dedupEnabled": true,
    "dedupStrategy": "provider_id",
    "dedupWindowHours": 48
  }'

Benefits at a Glance

• Zero custom code : Enable with a single API call or UI toggle
• Sub-millisecond performance : Powered by Cloudflare's edge KV storage
• Full audit trail : Duplicates are stored and linked to originals
• 10+ providers supported : Native ID extraction for major webhook senders
• Flexible strategies : From automatic to manual control
• Cost savings : Don't waste compute on duplicate processing

Getting Started

1. Navigate to Sources in your Hookbase dashboard
2. Edit your source or create a new one
3. Enable Deduplication and select your strategy
4. Set the time window (1-168 hours)
5. Save : duplicates are now automatically detected

Already using Hookbase? Deduplication is available now on all plans. New to Hookbase? Sign up free and start receiving webhooks with built-in reliability.