When deliveries fail

Sarah's server is down for an hour on Tuesday morning. Three customers book while it's out.

She doesn't lose them. We keep trying. But there are three things her receiver has to handle for that to be true, and none of them are optional.

Anything 2xx means delivered

Reply 2xx and we're done. Anything else — a 4xx, a 5xx, a timeout, a TLS failure — counts as a failure and gets retried.

The timeout is 30 seconds. Don't spend it:

res.sendStatus(200);          // say thanks
await enqueue(event);         // then do the work, elsewhere

Process inline and take 25 seconds, and you're one slow query away from a timeout, a retry, and a duplicate booking.

What we retry

Eight attempts over roughly 44 hours. A deploy costs nothing. Sarah's one-hour outage costs nothing. A two-day outage loses events.

Every attempt is recorded — what we sent, what came back — under Deliveries on the endpoint. A failed one can be retried by hand from there, which is what you want after fixing a receiver bug rather than waiting for the next scheduled attempt.

You will get the same event twice

Not "might". A crash between your 200 and our bookkeeping means we send something you already handled.

Deduplicate on HyphaBase-Webhook-Id — the same value on every attempt of the same event:

async function handle(webhookId, event) {
  // A unique index on webhook_id makes this atomic.
  const fresh = await db.insertIgnore('handled_webhooks', {
    webhook_id: webhookId,
    received_at: new Date(),
  });

  if (!fresh) return;      // already done

  await apply(event);
}

Do the insert in the same transaction as the work where you can. A dedupe record that commits before the work leaves you skipping an event you never actually applied — which is worse than a duplicate, because it's silent.

They can arrive out of order

Retries mean a created can land after the updated that followed it. Sarah's Tuesday backlog arrives in whatever order the retries fall.

So:

Upsert, don't insert. Key on the id from data.object.

Use created to drop stale updates if you keep a timestamp:

if (existing && existing.updatedAt >= record.updated_at) return;

Re-read when it really matters. data.object is a snapshot from when the event happened, not necessarily current.

We'll switch you off eventually

After 25 consecutive failed attempts — roughly three events using up their full retry schedule — the endpoint is disabled and stops receiving anything.

That protects both sides: we stop hammering a URL that isn't answering, and you stop building a backlog you can't process.

The counter resets on any success, and on Enable once your receiver is healthy again.

The checklist

A receiver that survives contact with reality:

• Reads the raw body and checks the signature before anything else
• Returns 2xx before doing the work
• Returns 2xx for event types it doesn't recognise
• Deduplicates on HyphaBase-Webhook-Id
• Upserts rather than inserts, and assumes nothing about order
• Is safe to replay — running yesterday's deliveries again changes nothing
• Alerts you when the endpoint gets disabled. This is the one failure that's otherwise silent, because a disabled endpoint stops producing failed deliveries too.

Next