Billing Webhooks Done Right: The Checklist

Billing webhooks fail quietly. The charge succeeds, the event never lands, and the customer keeps using a plan they cancelled or loses access they paid for. Nobody notices until someone complains. This is the checklist that makes that class of bug structurally impossible.

Rule One: The Handler Does Almost Nothing

Verify the signature, insert the raw event into a table, return 200. That is the entire synchronous path, and it should complete in under 100 milliseconds. Processing happens in a separate consumer reading from that table.

The reason is timeouts. Providers retry on any non-2xx response and on slow responses, so a handler that also updates subscriptions, sends email, and provisions an account gives you five ways to time out after doing partial work. Split it and a slow email provider can no longer cause duplicate charges to be processed twice.

Rule Two: The Event Id Is the Idempotency Key

Your events table needs a unique constraint on the provider’s event id. Insert with an on-conflict-do-nothing clause. The provider will send the same event again, guaranteed, and this one constraint is the whole defence.

Do not build your own key by hashing the payload. Two legitimate identical events, such as two charges of the same amount one second apart, would collapse into one and you would lose a payment record.

Rule Three: Keep an Append-Only Ledger

Entitlement lives in a subscriptions table you overwrite. Money lives in a ledger table you never update or delete: one row per charge, refund, credit, and dispute, with amount in the smallest currency unit, currency, provider reference, and the event id that produced it.

The ledger is what lets you answer “what did this customer actually pay us” without paging through a provider dashboard, and it is the only way to catch a divergence between your database and the provider. Sum the ledger per customer nightly and compare it to the provider’s balance transactions. A mismatch is a bug you want to find on your own schedule.

Rule Four: Handle Events Arriving Out of Order

Webhooks are not ordered. A subscription updated event can arrive after the deleted event that supersedes it, and a naive handler will resurrect a cancelled plan. Store the provider’s event timestamp on the subscription row and ignore any event older than the one you last applied.

For the specific case of subscription state, the safest pattern is to treat the webhook as a signal rather than data: on any subscription event, fetch the current subscription object from the provider API and write that. One extra API call per event removes ordering from your problem list entirely.

Rule Five: Make Replay a One-Command Operation

Because raw events are stored, reprocessing is a loop over rows. Give yourself a script that replays a single event id or every unprocessed event since a timestamp, and mark rows with processed_at plus an attempt count. When you fix a bug in the consumer, you fix history too, which is not possible if you only kept the derived state.

Verify on Every Event

• Signature is verified against the raw request body, before any parsing, using the provider’s timestamp tolerance.
• Raw payload is persisted with a unique constraint on the provider event id and an on-conflict-do-nothing insert.
• Handler returns 200 in under 100 milliseconds and performs no business logic.
• Processing runs in a separate consumer with processed_at, attempts, and last_error columns.
• Money movements are appended to an immutable ledger with amount in minor units, currency, and the source event id.
• Subscription writes are guarded by the provider event timestamp, or refetched from the API on every event.
• Entitlement is read from your own table in the request path, never from a live provider call.
• A nightly job compares ledger totals against the provider and alerts on any difference.
• Failed events land in a dead-letter state that raises an alert when the count is above zero.
• You have replayed a real event in production at least once and confirmed the result was identical.