August 5, 2026 · #Payments #Distributed Systems #Backend #Idempotency #Razorpay #Node.js

Making Payments Bulletproof

A payment looks like one event. It isn't. It's three racing signals trying to confirm the same money, plus a customer who can cancel at the worst possible moment. Almost every "we charged them twice" or "I paid but my order vanished" bug lives in the gaps between those signals.

Here's how we found and closed those gaps in Cravo's checkout — and the handful of ideas that make any payment flow reliable.

The mental model: payments are a distributed-systems problem

When a customer pays via Razorpay, our order can be confirmed by any of three independent paths, in any order, any number of times:

Client verify — our app calls back after the payment SDK returns success.
Webhook — Razorpay's servers call us directly (this fires even if the app was killed).
Reconciliation job — every 10 minutes we ask Razorpay "what actually happened?" to catch anything the first two missed.

Three signals. They can arrive first, arrive twice, or never arrive. The moment you have that, you're not writing payment code — you're writing concurrency code. Every fix below is really a concurrency fix.

Three independent paths — client verify, Razorpay webhook, and a reconciliation job — all converge on one PENDING payment row; the first to transition it becomes PAID and places the order while the others no-op.

Lesson 1: Idempotency lives in the write, not in a read

The instinct for "don't process this twice" is a check:

if (alreadyProcessed(paymentId)) return;   // check
markProcessed(paymentId);                  // then act

This is a race. Two webhooks arrive together, both pass the check before either writes, both run the side effects. You clear the cart twice, credit a balance twice.

The fix is to let the database be the single arbiter, with one atomic conditional write:

// UPDATE payment SET status = 'PAID'
// WHERE id = ? AND status IN ('PENDING', 'PROCESSING')
// -> returns rowsAffected
const won = await updateStatusIfAllowed(id, 'PAID', ['PENDING', 'PROCESSING']);
if (won) {
  // Only ONE signal ever gets here. Safe to run side effects.
}

Only the first signal flips PENDING → PAID and gets rowsAffected = 1. Everyone else gets 0 and quietly no-ops. Idempotency isn't "have I seen this before?" — it's "did I win the transition?" The answer comes from the write itself.

Two concurrent webhooks send the same conditional UPDATE to the database; it returns rowsAffected = 1 to the winner, which runs the side effects, and rowsAffected = 0 to the loser, which no-ops.

Lesson 2: For money, the provider is the source of truth — never your own flag

The nastiest bug we had: a customer closes the app mid-payment, the app relaunches and auto-cancels the "abandoned" order — but the payment actually succeeded, and the webhook lands a few seconds later. If you check your local paymentStatus flag at cancel time, it still says PENDING, so you happily cancel an order the customer already paid for.

Two defenses, both asking Razorpay directly instead of trusting the local flag:

Cancel guard: before allowing a cancel, we live-query Razorpay. If the payment captured, we refuse the cancel and reconcile our database to match reality.
Resurrection: if a payment confirms after an order was cancelled, we put the order back.

The rule: any irreversible decision about money is made against the payment provider's truth, not a flag you set earlier and hoped stayed accurate.

Lesson 3: A confirmed webhook must guarantee the order is placed — instantly

We already resurrected paid-but-cancelled orders... but only via the 10-minute reconciliation job. Ten minutes is an eternity when a customer is staring at a "cancelled" screen for an order they just paid for.

So we made resurrection synchronous inside every confirmation path. The moment a payment is confirmed — by webhook, by client verify, by reconciliation — the same call stack places the order:

const RESURRECTABLE = new Set(['CREATED', 'CONFUSED_CUSTOMER', 'CANCELLED']);
if (RESURRECTABLE.has(order.status)) {
  // flip to CONFIRMED, write history, notify the kitchen — right now
}

We also added a backstop: even when a different signal won the payment transition, the webhook re-checks that the order actually got placed. "Payment confirmed" and "order placed" are now the same instant, not two events hoping to meet.

Order state machine: CREATED moves to CONFIRMED on payment, or to CONFUSED_CUSTOMER or CANCELLED; both of those resurrect to CONFIRMED if payment confirms afterward, while a deliberate REJECTED never resurrects and is refunded instead.

Lesson 4: Never trust an amount that came from the client

Our online flow already computed the charge server-side. But one code path stored the amount the client sent, with a ±₹5 "tolerance" for rounding. That tolerance was a quiet leak — a client could record ₹5 less than the real total.

The fix is a one-liner in spirit: the server computes the authoritative total from the database and stores that. The client's number is logged for observability and otherwise ignored. No tolerance, no leak — and, as a bonus, no false rejections of honest customers whose rounding differed by a rupee.

Lesson 5: Preventing the double charge

"Double charge" splits into two very different problems:

Double processing — one payment, our side runs the side effects twice. Solved by Lesson 1's atomic transition.
Double charge — the customer's money actually leaves twice. This is the scary one.

We verified against Razorpay's official documentation that one order can only ever have one successful payment — "Single successful payment bound to an order. Prevents multiple payments." So a genuine double charge is only possible if we accidentally create two payment orders for one cart. We found exactly two ways that could happen, and closed both:

The retry orphan. When a payment "failed," a retry minted a brand-new payment order and overwrote the old one — but the old one was still payable at the provider. If the "failure" was a false alarm (the payment actually captured), the customer could pay both. Now, before minting anything on retry, we live-check the old order; if it actually captured, we reconcile it to PAID and block the new one.
The concurrent tap. Two simultaneous "create order" requests could each mint a payable order. We added a short per-order lock so only one is ever created.

Testing the boring, critical paths

None of this is code you want to "hope" works. We added a suite covering the exact failure modes: duplicate webhooks, cancel-then-pay races, the resurrection matrix (which order states get revived and which must not), the retry-that-actually-paid, the concurrent-tap lock, and the server-authoritative amount. Money code without tests is a liability; these are the cheapest insurance you'll ever write.

Takeaways

A payment is three racing async signals. Design for that, not for the happy path.
Idempotency is a property of your write (atomic compare-and-set), not a prior read.
For anything irreversible, treat the payment provider as the source of truth.
"Payment confirmed" should mean "order placed" — in the same instant.
Never trust a client-supplied amount.
Double charges hide in duplicate orders, not duplicate captures — guard the order-creation path.

The fixes were small. The bugs they killed were the kind that erode customer trust at the one moment it matters most: paying you.

Making Payments Bulletproof: Idempotency, Race Conditions, and Never Charging Twice