Webhooks will be delivered twice. Design for it.

Every webhook provider worth using guarantees at-least-once delivery. Read that carefully. It does not say once.

If your handler is slow, they retry. If your response is a 500, they retry. If the response is lost after you processed it, they retry. Stripe, Shopify, Xero, GitHub — all of them, by design, because the alternative is losing events.

I learned this properly when a sync between a store and an accounting system produced 300 duplicate invoices over a weekend. The provider had a delivery incident and replayed several hours of events. The handler had no idea it had seen them before.

Deduplicate on the provider’s event id

Not on the order id. Not on a hash of the payload — payloads sometimes differ between the original and the retry. Use the event id the provider assigns, which is stable across retries.

create table webhook_events (
  provider    text        not null,
  event_id    text        not null,
  received_at timestamptz not null default now(),
  primary key (provider, event_id)
);

Then let the database enforce it:

const { rowCount } = await db.query(
  `insert into webhook_events (provider, event_id)
   values ($1, $2)
   on conflict do nothing`,
  ['xero', event.id]
);

if (rowCount === 0) {
  return res.status(200).end(); // already handled
}

The unique constraint does the work. Checking for existence and then inserting is a race — two concurrent deliveries both see nothing, both proceed. I have watched that happen in production. It is not theoretical.

Acknowledge fast, process after

Most providers time out at 5 or 10 seconds. If your handler creates an invoice, sends an email and updates a CRM inline, you will exceed that eventually, and the provider will retry an event you actually processed successfully.

Write the event down, return 200, process asynchronously. The handler does two things: verify the signature, and persist. Everything else happens on a queue.

This also gives you replay. When a downstream API is down for an hour, the events are already stored — you reprocess from your own table instead of asking the provider for a redelivery they may not offer.

Verify signatures against the raw body

A recurring bug. The signature is computed over the exact bytes sent. If your framework parses JSON before you verify, you are re-serialising, and key order or unicode escaping may differ. Verification then fails intermittently — which is worse than failing always, because it looks like a provider problem.

app.post('/webhooks/stripe',
  express.raw({ type: 'application/json' }),
  (req, res) => {
    const event = stripe.webhooks.constructEvent(
      req.body,                       // Buffer, not parsed object
      req.headers['stripe-signature'],
      process.env.STRIPE_WEBHOOK_SECRET
    );
  }
);

Use a constant-time comparison if you are verifying by hand. === on an HMAC leaks information through timing.

Events arrive out of order

Deduplication does not solve ordering. subscription.updated can land before subscription.created, especially after a retry storm.

Two options. Either fetch current state from the API when an event arrives and treat the webhook purely as a notification that something changed — more requests, always correct. Or compare a version field or timestamp on the payload and drop anything older than what you have stored.

I default to the first. It costs an API call and removes an entire category of bug.

Return 200 for things you will not process

If you get an event type you do not handle, return 200. A 404 or 500 tells the provider to retry forever, and eventually most of them disable the endpoint. I have seen an integration go dark because an unhandled event type was 500ing and the provider quietly switched it off after a week of failures.

Log it, return 200, move on.