Payment and Transaction Processing Systems Questions

Direct answer

Split the integration into a thin, fast receiver and a separate processor. The receiver verifies the provider's signature on the raw body, rejects stale timestamps (replay protection), stores the event once keyed by the provider's event ID, and returns 2xx within milliseconds. The processor applies events idempotently (applying the same event twice has the same effect as applying it once) and treats each one as a hint: it moves our records only forward, and for anything that matters it re-reads the object from the provider's API, so duplicates and out-of-order deliveries cannot corrupt state. Retries are the provider's job for delivery and ours for processing (backoff, then a dead-letter queue), and a reconciliation job closes whatever webhooks miss.

What the provider actually guarantees (and what it does not)

A webhook is an HTTP POST the provider sends to our endpoint when something happens (a payment succeeds, a dispute opens). Taking Stripe as the concrete example (other providers are similar, check each one's documentation):

• Delivery is at-least-once: the same event can arrive more than once, and occasionally two different event objects describe the same change.
• Order is not guaranteed: invoice.paid can arrive before invoice.created, and event timestamps have one-second resolution, so they cannot be used to order events.
• Undelivered events are retried for up to three days with exponential backoff in live mode. A non-2xx response or a timeout counts as a failure.
• Each delivery carries a Stripe-Signature header: t=<unix timestamp>,v1=<hex HMAC-SHA256> computed over the string timestamp + "." + raw body with the endpoint's signing secret. HMAC is a keyed hash: only someone holding the secret can produce a matching value. The official libraries reject timestamps more than 5 minutes old by default.

So the design has to supply deduplication, ordering tolerance and replay protection itself.

Architecture

flowchart LR
    PR[Provider] -->|POST event| LB[TLS load balancer]
    LB --> RX[Receiver: verify, store, 2xx]
    RX --> INBOX[(Inbox table: unique event_id)]
    INBOX --> Q[Queue]
    Q --> W[Processor workers]
    W -->|re-read object| API[Provider API]
    W --> DB[(Payments + ledger)]
    W -->|after N failures| DLQ[Dead-letter queue]
    RECON[Reconciler: list events and objects] --> DB

1. Receiver: authenticate, persist, acknowledge

1. Read the raw bytes of the body before any JSON parsing (re-serialised JSON changes whitespace and breaks the signature).
2. Verify the v1 signature with a constant-time comparison (so response timing does not leak how many characters matched), ignoring any other scheme, and check the timestamp is within tolerance.
3. INSERT INTO webhook_inbox(event_id, type, object_id, payload, received_at) ON CONFLICT (event_id) DO NOTHING.
4. Return 200. No business logic here: slow handlers time out, timeouts trigger provider retries, and retries multiply load exactly when the system is struggling.

Signature failures return 400 and are counted; they should be near zero, and a spike means a rotated secret or an attack.

2. Deduplication

• Event-level: the unique event_id in the inbox.
• Effect-level: every side effect has its own guard. Marking an order paid is UPDATE ... WHERE status <> 'paid'; posting a ledger entry has a unique key (payment_id, entry_type); sending a receipt goes through an outbox keyed by (payment_id, 'receipt'). This catches the case of two distinct event IDs for the same change, which event-ID dedupe alone misses.

3. Out-of-order events

Pick one of two strategies per resource type, not a global one:

Re-fetching costs one API call per event (about 93 extra reads/second at the load below), which is cheap insurance. Serialise processing per object ID (partition the queue by object_id) so two workers never apply two events for the same payment concurrently.

4. Replay protection

• The signed timestamp bounds how long a captured request stays usable: an attacker who records a valid delivery cannot resend it an hour later. Keep server clocks on NTP (Network Time Protocol) so a skewed clock does not reject real events.
• Within the tolerance window, a replay is just a duplicate and the inbox's unique key absorbs it.
• Optionally allow-list the provider's published source IP ranges at the load balancer as a second layer; never as the only one.

5. Retry behaviour on our side

• Delivery retries belong to the provider; we just must return 2xx once the event is safely stored, even if processing later fails.
• Processing retries: exponential backoff with jitter (random spread so failures do not retry in lockstep), for example 1 s, 2 s, 4 s ... capped at 5 minutes, up to 8 attempts, then move the event to a DLQ (dead-letter queue: a holding area for messages that repeatedly failed, for a human or a fixed version of the code to replay).
• Poison events (a type we do not understand, a payload that fails validation) go to the DLQ immediately rather than burning retries.

A runnable check of the core rules

The program below signs events the same way, delivers three state changes for one payment shuffled with one duplicate, then tries a tampered body and a one-hour-old replay. Standard library only; the shuffle seed is pinned.

import hashlib, hmac, json, random

SECRET = b"whsec_test_secret"
TOLERANCE_S = 300

def sign(body: bytes, ts: int) -> str:
    mac = hmac.new(SECRET, f"{ts}.".encode() + body, hashlib.sha256).hexdigest()
    return f"t={ts},v1={mac}"

def verify(body: bytes, header: str, now: int) -> bool:
    parts = dict(p.split("=", 1) for p in header.split(","))
    ts = int(parts["t"])
    expected = hmac.new(SECRET, f"{ts}.".encode() + body, hashlib.sha256).hexdigest()
    if not hmac.compare_digest(expected, parts.get("v1", "")):
        return False
    return abs(now - ts) <= TOLERANCE_S          # replay window

# Payment state machine: a status may only move forward.
RANK = {"requires_capture": 1, "succeeded": 2, "refunded": 3}

seen_event_ids = set()      # in production: a unique-keyed table, written in the same transaction
payments = {}               # payment_id -> status

def handle(body: bytes, header: str, now: int) -> str:
    if not verify(body, header, now):
        return "rejected"
    ev = json.loads(body)
    if ev["id"] in seen_event_ids:
        return "duplicate"
    seen_event_ids.add(ev["id"])
    pid, new = ev["payment_id"], ev["status"]
    old = payments.get(pid)
    if old is not None and RANK[new] <= RANK[old]:
        return f"stale ({new} after {old})"
    payments[pid] = new
    return f"applied -> {new}"

events = [{"id": f"evt_{i}", "payment_id": "pay_1", "status": s}
          for i, s in enumerate(["requires_capture", "succeeded", "refunded"], 1)]
rng = random.Random(3)
deliveries = events + [events[1]]            # one duplicate delivery
rng.shuffle(deliveries)                      # out of order
now = 1_700_000_000
for ev in deliveries:
    body = json.dumps(ev).encode()
    print(ev["id"], ev["status"], "=>", handle(body, sign(body, now - 10), now))

body = json.dumps(events[0]).encode()
print("tampered   =>", handle(body.replace(b"pay_1", b"pay_9"), sign(body, now), now))
print("replayed 1h later =>", handle(body, sign(body, now - 3600), now))
print("final state:", payments)

Output:

evt_2 succeeded => applied -> succeeded
evt_1 requires_capture => stale (requires_capture after succeeded)
evt_3 refunded => applied -> refunded
evt_2 succeeded => duplicate
tampered   => rejected
replayed 1h later => rejected
final state: {'pay_1': 'refunded'}

The late requires_capture event cannot drag the payment backward, the duplicate is dropped, and both the tampered and the stale deliveries fail verification.

Load and sizing (assumed numbers)

2,000,000 payments/day x 4 events each = 8,000,000 events/day, which is 8,000,000 / 86,400 = about 93 events/second on average. Subscription renewals at the start of a month can spike 20x, to about 1,850/second. The receiver does one signature check and one indexed insert per event, so a few stateless instances behind the load balancer absorb that; the queue lets processors drain the spike at their own pace instead of the spike hitting the database.

How to test it

1. Unit tests: signature verification against known vectors (valid, wrong secret, modified body, timestamp just inside and just outside tolerance, header with several v1 values during a secret rotation, extra unknown schemes); state-transition rules for every pair of states.
2. Property-style tests: generate a random sequence of valid lifecycle events, deliver them in every permutation with random duplicates, and assert the final state equals the state after in-order delivery. The small program above is one instance of that property.
3. Integration tests with the provider's tooling: Stripe's CLI can forward test-mode events to a local endpoint (stripe listen --forward-to) and trigger specific event types (stripe trigger payment_intent.succeeded); use them to exercise real payloads and real signatures.
4. Failure injection in staging: make the processor fail for 10 minutes and confirm events pile up in the queue and drain afterwards; return 500 from the receiver and confirm the provider's retries are absorbed as duplicates; kill a worker mid-transaction and confirm nothing is half-applied.
5. Contract drift: pin the provider API version your endpoint receives, and test a version upgrade against recorded payloads before switching.

How to operate it in production

• SLOs (service-level objectives): for example 99.9% of deliveries acknowledged within 500 ms, and 99% of events processed within 60 s of receipt.
• Dashboards and alerts: receive rate by type (a drop to zero is itself an incident: a firewall change or a disabled endpoint), signature-failure rate, inbox-to-processed lag, age of the oldest unprocessed event, DLQ depth, and the provider's own delivery-failure view.
• Reconciliation as the safety net: every hour, list recent events from the provider's events API and compare against the inbox; daily, compare payment objects and settlement reports with the ledger. Webhooks are the fast path, not the only path.
• Secret rotation: Stripe lets the old signing secret stay valid for up to 24 hours after rolling, sending one signature per active secret; deploy the new secret to the verifier during that overlap.
• Runbooks: replaying from the DLQ after a fix, backfilling a window of missed events, and disabling a single event type's processing without dropping the rest.

Trade-offs and pitfalls

• Processing inline in the HTTP handler is the most common failure: it couples provider retries to your database latency.
• Deduping only on event ID misses duplicate changes carried by different events; guard effects too.
• Trusting payload snapshots for money decisions lets a stale event win; re-read the object.
• Using a tolerance of zero in some libraries disables the timestamp check entirely rather than making it strict.
• What would change the design: at very low volume, a single table plus a cron worker is enough (no queue); if the provider offers direct delivery into a cloud event bus, you can drop the public receiver and inherit the bus's retry and dead-letter features, while keeping the same idempotent processor.

Direct answer

Plan for three things in order: keep taking payments by routing new authorizations to a pre-integrated backup processor (and degrading payment methods that cannot move), never lose or double-count money by freezing every in-flight transaction with the primary as "unknown" until the primary can be asked what happened, and settle and reconcile per processor afterwards, because each processor pays out, charges fees and reports in its own way. Communication runs on a pre-agreed clock: merchants and customers through status pages and API signals within minutes, regulators and card-scheme partners according to their incident-reporting obligations.

Key terms

• Processor / PSP (payment service provider): the company that sends card authorizations to the card networks and later pays out the money.
• Authorization: the issuing bank approves and places a hold on funds. Capture: the merchant claims the held funds. Settlement: the processor actually transfers the money, usually a day or more later, minus fees.
• Reconciliation: matching what your own records say should have happened against what processors and banks report did happen.
• Circuit breaker: logic that stops sending traffic to a failing dependency once errors cross a threshold, then probes it periodically.
• MID (merchant ID): the identifier a processor or acquirer assigns to a merchant account; the backup needs its own.

What must exist before the outage

A DR (disaster recovery) plan for payments is mostly work done in advance:

1. A second processor, integrated and warm (fully set up and already carrying a slice of live traffic, not sitting idle). Signed contract, underwriting completed (the processor's own review and approval of your business before it will process for you), MIDs issued for every merchant you intend to fail over, and a small share of live traffic (for example 2 to 5%) routed there permanently, so you know it works today, not in theory.
2. Portable card credentials. Tokens issued by the primary processor do not work at the backup. Stored cards (subscriptions, one-click checkout) can only fail over if you hold network tokens (issued by the card networks, usable across processors) or run your own vault that can send card data to either processor. Without that, only customers typing a card fresh can pay through the backup.
3. A mapping layer. A table per processor translating your internal payment methods, currencies, merchant category codes (MCCs, the standard 4-digit codes that classify what kind of business a merchant is), statement descriptors (the text that shows up on the cardholder's statement), decline codes and 3-D Secure (the card-holder authentication step) settings into that processor's vocabulary. Divergence here (for example a merchant category code the backup does not recognise, or a decline code mapped to the wrong retry behavior) is what turns a failover into a flood of declines.
4. A routing service with per-processor health, a circuit breaker, and a manual override switch.

During the outage

flowchart TD
  A[New payment] --> B{Primary healthy?}
  B -- yes --> C[Primary processor]
  B -- no --> D{Card usable at backup?}
  D -- yes --> E[Backup processor]
  D -- no --> F[Offer other method or retry later]
  G[In-flight at primary when it failed] --> H[State: unknown, no retry elsewhere]
  H --> I[Resolve via primary status query or settlement report]

• Route new authorizations to the backup once the breaker opens. Keep the idempotency key (the client's unique ID for this payment attempt) as your internal reference so each processor call is traceable to one payment.
• Do not replay in-flight payments at the backup. A request that timed out at the primary may have been approved there. Retrying it at the backup risks charging the customer twice. Mark it unknown, tell the merchant the outcome is pending, and resolve it once the primary answers status queries or its settlement file arrives.
• Captures, refunds and voids follow the original processor. An authorization made at the primary can only be captured or refunded through the primary. Queue these operations durably and drain them when the primary returns; if an authorization would expire first (online card authorizations are typically valid for about 7 days), decide per merchant whether to re-authorize at the backup.
• Degrade gracefully. Recurring charges on non-portable tokens are deferred and retried later rather than failed; merchants see a clear "processing delayed" status rather than hard declines. For card-present terminals, offline "store and forward" approvals (the terminal approves the sale itself on the spot, with no live authorization call, then submits it for real authorization once connectivity returns) under a low per-transaction limit are possible but the merchant carries the risk of later declines; enable it only where merchants opted in.
• Protect the backup. Ramp traffic gradually and watch approval rates; a sudden jump in volume can trigger the backup's own fraud controls.

After the outage: settlement and reconciliation

• Reconcile per processor. Each processor's settlement report is matched against your ledger (your internal, append-only record of money movements) for the transactions it handled. The routing decision is recorded on every payment, so each payment belongs to exactly one processor's reconciliation.
• Resolve every unknown. For each, the primary either reports it approved (keep it, capture as normal) or never received or declined it (mark failed, and if the customer retried and succeeded at the backup, there is no double charge to unwind). Any real duplicate found is refunded proactively.
• Different payout timing. The backup may pay out a day later, to a different bank account, with different fee deductions. Merchants' payout projections and your treasury forecast (the internal projection of cash moving in and out of the business) must show this explicitly.
• Chargebacks and disputes (a customer disputing a charge through their bank) for backup-processed payments arrive from the backup for months afterwards; the dispute system must know to look there.

Fees and who pays them

Fees differ between processors, so decide in advance whether the platform absorbs the difference or passes it on. Illustrative numbers (assumed contract rates, not any real vendor's price list):

• A merchant processes 2,000,000 USD per day, spread evenly, at an average ticket of 50 USD. A 6-hour outage moves 6/24 of that, 500,000 USD, or 10,000 payments, to the backup.
• Primary rate 2.2% + 0.10 USD: 11,000 + 1,000 = 12,000 USD.
• Backup rate 2.9% + 0.30 USD: 14,500 + 3,000 = 17,500 USD.
• Extra cost of the outage: 5,500 USD for this merchant.

Most platforms absorb this for short outages (it is small against lost sales and merchant trust) and write the rule into their merchant terms so it is not negotiated during an incident.

Communication

Trade-offs and pitfalls

• Cost of warmth. Keeping a second processor live costs contract minimums and engineering on two integrations, but a cold backup is almost always broken when needed.
• Automatic vs manual failover. Automatic failover reacts in seconds but can flap (switch back and forth rapidly as health checks briefly pass, then fail, then pass again) on a partial outage and send traffic to the backup that the primary would have approved. A common compromise: automatic for hard failures (connection refused, sustained 5xx), human decision for soft degradation (elevated latency, falling approval rate).
• The dangerous mistake is converting timeouts into failures and retrying them elsewhere. That produces the double charges that customers remember long after the outage.
• Approval rates will drop on the backup (new MIDs, no transaction history, different issuer relationships). Plan and tell merchants; do not treat it as a new incident.

Direct answer

Never put a slow, variable dependency on the checkout path without a deadline you own. Call the fraud vendor with a hard per-request deadline sized to your checkout budget (a few hundred milliseconds, not the vendor's seconds), and when the deadline passes, decide with a local fallback: a recent cached score for the same card and device, an in-house rules/model score, or a policy default by risk band. Anything decided without the vendor is authorized but not yet captured, and the vendor score is fetched asynchronously afterwards; a bad late score triggers a void (cancel the authorization, no money moved) before capture, or a refund and review after it.

Why a fixed timeout is the core decision

Checkout has a total latency budget (say 1.5 seconds end to end, of which the card authorization itself takes a large share). The fraud step can have perhaps 300 ms of it. A vendor whose tail (the slow end of its response-time distribution: the rare calls that take far longer than typical) reaches several seconds will blow the budget unless you cut it off.

To make the trade-off concrete, here is a latency model of such a vendor (the mixture is an assumption for illustration, not a measurement of any real vendor) and how many calls each deadline would send to the fallback:

import random, math
random.seed(42)
# Vendor latency model (illustrative mixture, not a vendor measurement)
def sample():
    r = random.random()
    if r < 0.80:   median = 60     # fast path
    elif r < 0.95: median = 400    # vendor calling its own slow sources
    else:          median = 2500   # vendor degraded
    # log-normal spread: a random multiplier centered on 1x (never negative, occasionally
    # much larger) so latencies vary within each mode instead of repeating exactly
    return median * math.exp(random.gauss(0, 0.35))
lat = sorted(sample() for _ in range(100_000))
def pct(p): return lat[int(p/100*len(lat))-1]
print("p50 %.0f ms  p95 %.0f ms  p99 %.0f ms" % (pct(50), pct(95), pct(99)))
for d in (150, 300, 500, 1000):
    over = sum(1 for x in lat if x > d) / len(lat)
    print("deadline %4d ms -> %.1f%% of calls fall back" % (d, 100*over))

Output:

p50 67 ms  p95 1131 ms  p99 3379 ms
deadline  150 ms -> 20.3% of calls fall back
deadline  300 ms -> 16.9% of calls fall back
deadline  500 ms -> 8.9% of calls fall back
deadline 1000 ms -> 5.1% of calls fall back

Reading it: p50/p95/p99 mean the 50th, 95th and 99th percentile latency (half, 95% and 99% of calls finish faster). With a bimodal vendor (one that behaves like two or more distinct speed regimes, fast almost all the time and then sharply slower for a minority of calls, rather than one smooth curve), moving the deadline from 300 ms to 500 ms halves fallbacks (16.9% to 8.9%) for 200 ms more worst-case checkout time, while going to 1 second buys only 3.8 more points. That knee (the point where paying for a longer deadline stops buying much reduction in fallbacks) is where you set it, and you re-measure with your real vendor's histogram rather than trusting this model.

The design

flowchart LR
  C[Checkout] --> D[Risk decision service]
  D --> K[(Score cache)]
  D --> V[Vendor adapter with deadline]
  D --> L[Local rules and model]
  D --> P[Policy table by risk band]
  D --> A[Authorize]
  D -. timed out or low confidence .-> Q[(Post-auth scoring queue)]
  Q --> W[Async scorer]
  W --> R[Remediation: void, hold, review, refund]

1. Real-time scoring where required. A policy table decides, per merchant category and amount band, whether the vendor score is mandatory (wait up to the deadline, and if it is missed treat the payment as high risk), preferred (use it if it arrives, fall back otherwise) or skipped (low-risk, low-value; score only afterwards). Mandatory is for irreversible, high-value goods: gift cards, electronics shipped the same hour, crypto top-ups.

2. Graceful degradation. The vendor adapter has:

• a deadline per call (the budget above), and no automatic retries inside the checkout path (a retry doubles the latency you just decided you cannot afford);
• a circuit breaker (after, say, 50% of calls in a 30-second window time out, stop calling for a cool-down period and go straight to fallback, so you are not paying 300 ms per checkout to learn the vendor is down);
• a bulkhead (a separate, bounded connection pool, so a slow vendor cannot exhaust the threads that serve other checkout calls).

Fallback order, first match wins:

1. Cached vendor score for this card fingerprint (a stable identifier derived from the card's number and attributes, kept instead of the number itself) plus device, under 10 minutes old, and the amount no more than the cached one was scored at.
2. In-house model or rules (velocity: count of attempts per card, device and IP in the last hour; mismatch between billing country and IP country; account age).
3. Policy default for the band: approve-and-review for low value, step-up (send the shopper through a 3-D Secure authentication challenge) for medium, decline or hold for high.

3. Cache of recent scores. Key it by the entity, not the transaction: (card fingerprint, device id, merchant), with the score, the amount scored, the vendor model version and a timestamp. Short TTL (time-to-live) of minutes, because risk changes fast (a stolen card is used in bursts). The cache also absorbs retries: a shopper who resubmits after a decline should not trigger a second paid vendor call.

4. Asynchronous post-authorization scoring and remediation. Any payment decided without a fresh vendor score is published to a queue with its full context. A worker calls the vendor with a relaxed timeout (seconds, with retries and backoff) and acts on the answer:

This is why the design deliberately delays capture for fallback-decided orders by the few seconds or minutes the async score needs: the cheap remediation (void) is only available before capture.

Worked example

A marketplace does 200 checkouts per second at peak. With a 300 ms deadline and the model above, 16.9% (about 34 per second) fall back, but every checkout still triggers a synchronous vendor call attempt whether or not it beats the deadline: a call that arrives late still lands at the vendor and still counts against its rate limit, even though the checkout path stopped waiting for it. So the vendor's synchronous load is the full 200 per second, not 166.

Of the 34 per second that fall back, suppose 40% (about 14 per second) are decided from a fresh cached score, which is itself a recent vendor opinion, so those are not requeued. The remaining 60% (about 20 per second) have no usable cache and are decided by the in-house model alone; only these go to the async queue for a supplemental vendor check before capture. Total vendor load is therefore about 200 (synchronous attempts) + 20 (async rescoring) = 220 per second, a bit above the raw checkout rate, which is what the vendor contract should actually be sized against.

Size the contract with explicit headroom for outage recovery: say 300 per second, 50% above the raw 200 per second checkout rate. That leaves 300 − 220 = 80 per second of spare capacity even during ordinary operation. If the vendor goes fully down, the circuit opens and all 200 per second flow to fallback (cache where fresh enough, in-house model otherwise); a 15-minute outage leaves 200 × 900 = 180,000 payments to rescore once it recovers. Once the vendor is back, live traffic returns to its usual ≈220 per second demand, leaving the same 80 per second free to drain the backlog: 180,000 ÷ 80 ≈ 2,250 seconds, about 38 minutes to fully catch up on top of the outage itself. That is why capture stays held on fallback orders well past the moment the vendor comes back, not just for the outage's own duration.

Trade-offs and pitfalls

• Fail open versus fail closed. Declining everything when the vendor is slow protects against fraud and destroys revenue; approving everything invites attackers who learn your vendor is down. The risk-band policy is the answer, not a global switch.
• Retrying the vendor synchronously turns a 2-second tail into a 4-second one.
• Caching per transaction gives no hits; caching per card forever misses bursts of fraud. Entity key plus short TTL.
• Hedged requests (sending a second copy of a slow call after the p95 latency) cut tail latency but double vendor cost for those calls and only help if slowness is random per request rather than a vendor-wide incident.
• Measure the fallback's quality: track the fraud and chargeback rate of payments approved by fallback separately, or you will not notice the fallback is the hole.

Direct answer

Make every merchant single-home: each merchant's balance and idempotency records are owned by exactly one region (its home region, chosen by data-residency rules), so there is only ever one writer for the facts that must be exactly right. This is active-active in the sense the client sees, not in the sense of every region independently accepting writes for the same data: all three regions are active (each is home to a third of merchants and accepts traffic for all of them, so there is no single "primary site" a client must reach), but a write for a merchant homed elsewhere is forwarded to the home region instead of being accepted locally and merged later, which is what keeps balances correct. Balances are derived from an append-only ledger (the authoritative, append-only record of every money movement) of immutable entries, so cross-region replication copies facts rather than resolving conflicting numbers. Processor outages are handled by multiple processors per region behind a router; region outages by an in-jurisdiction standby (an already-running, up-to-date replica in the same legal jurisdiction, ready to take over); and a continuous reconciliation job compares our ledger with processor settlement files.

Requirements and the numbers they imply

• 10,000 TPS (transactions per second) peak globally, split across 3 residency domains (see Data residency below), each running its own home region plus an in-jurisdiction standby: about 3,333 TPS per domain at peak. Because a domain's failover target is always its own standby, never one of the other two domains' regions (residency forbids that), no region ever has to absorb another domain's load. Size every region, home and standby alike, for its own domain's full peak of about 3,333 TPS plus headroom, not for a fraction of the global total.
• P95 under 200 ms (95% of API calls finish within 200 ms). A cross-region round trip between continents is on the order of 70 to 150 ms (estimate; depends on the region pair), so no synchronous cross-continent step can sit on the common path. I read the 200 ms as our platform's own latency; the processor's authorization round trip to the card network is outside our control, so the API either includes it with a separate budget or returns pending and completes by webhook. That split is stated to the product owner, not assumed.
• Write amplification: a payment writes roughly 6 rows (idempotency record, payment, 2 to 3 ledger entries, outbox event), so 10,000 × 6 = 60,000 row writes per second at peak, about 20,000 per region: shard the ledger.
• Storage: if average load is 30% of peak (assumption), that is 10,000 × 0.3 × 86,400 = 259.2 million payments per day; at about 2 KiB of rows each (2,048 bytes), 259.2 M × 2,048 bytes ≈ 531 GB per day before indexes and replicas.
• Replication bandwidth at peak: 10,000 × 2,048 bytes ≈ 20.5 MB/s globally, which is modest.

Service decomposition

flowchart LR
  C[Client] --> E[Regional edge gateway]
  E --> PA[Payment API]
  PA --> TV[(Regional token vault)]
  PA --> IDM[(Idempotency store)]
  PA --> RR[Home-region router]
  RR --> LD[(Ledger shards)]
  PA --> PR[Processor router]
  PR --> P1[Processor 1]
  PR --> P2[Processor 2]
  PA --> OB[(Outbox)] --> Q[Event stream]
  Q --> ST[Settlement and payouts]
  Q --> RC[Reconciliation]

Data flow for one payment (merchant homed in this region)

1. Edge tokenizes the card; the API receives the token.
2. Idempotency: insert (merchant_id, idempotency_key) with a unique constraint in the merchant's home shard. If it exists, return the stored result. Because the key lives only in the home region, a client retrying against another region still hits the same record (after forwarding), so there is no cross-region double charge.
3. Call a processor (with its own idempotency key) through the processor router.
4. In one local transaction on the merchant's shard: update payment state, post ledger entries, write outbox event. Replicated synchronously across availability zones (separate data centres within the region) with a quorum (a majority of replicas acknowledging the write before it counts), so a zone loss loses nothing; this costs a few milliseconds, not a cross-continent round trip.
5. Respond. The outbox relay publishes the event for async consumers.

For a merchant homed elsewhere, step 2 onwards runs in the home region after one forward; that adds one inter-region round trip, which is why merchants are homed near their customers and why the forward path is measured separately.

Data replication and conflict resolution

• Within a region: synchronous quorum replication across three zones (for example a Raft-based distributed SQL database, or Postgres with synchronous standbys (replicas that must confirm a write before it counts as committed)). Raft is a consensus protocol in which a majority of replicas must acknowledge a write before it counts.
• Across regions: asynchronous, one-way from home to a standby. The standby lags by seconds.
• Conflict resolution by design: with a single writer per merchant there are no write-write conflicts (two writers changing the same record at the same time, one silently overwriting the other) on balances or idempotency keys. Ledger entries are immutable facts with globally unique IDs, so merging replicas is a set union. The rejected alternative, multi-writer balances merged later (last-writer-wins or CRDT counters, conflict-free replicated data types that merge automatically), cannot enforce "balance never goes below zero" or "charge exactly once", because two regions can each approve a spend the other has not seen. (The rejected alternative's "last-writer-wins" means: when two conflicting writes arrive, keep whichever has the later timestamp and silently discard the other, which is exactly how a second spend can vanish from the losing region's view.)

Data residency

The home region is chosen by residency: an EU merchant's personal and cardholder data is homed in the EU region and its standby must also be in the EU (a second EU region, or at minimum separate zones), never the US region. Cross-region, only what is allowed leaves: aggregated or pseudonymized (personal identifiers replaced with a substitute so the record can't be traced straight back to a person) events for global reporting and fraud models. The token vault is per region. This means "three regions" is really three residency domains, each with its own failover pair.

Failover

• Processor outage: each region has at least two processors. The breaker opens on error or timeout rate; new payments go to the backup. A timed-out call is not resent elsewhere until the first processor's status is queried, to avoid double authorization.
• Zone outage: absorbed by quorum replication, no action.
• Region outage: merchants homed there are promoted to their standby (in-jurisdiction) after the lag has drained or with a documented loss window. Promotion is a deliberate, fenced operation: the old home is marked read-only by incrementing a fencing epoch (a number the ledger checks on every write, so a partitioned (cut off from the rest of the cluster by a network split, but still running and unaware it has been replaced) old primary's writes are rejected). Until promotion, payments for those merchants are refused or queued as pending, rather than accepted by a second writer.

Reconciliation

Three loops: (1) intra-system, ledger debits equal credits per shard every few minutes; (2) standby versus home, entry counts and hashes per merchant per hour; (3) external, processor settlement files matched to payments by processor reference daily, with every unmatched item opened as a case. Merchant balances reported to finance come from the reconciled ledger.

Trade-offs and pitfalls

• Home-region forwarding costs latency for travelling merchants' customers; the alternative (multi-writer) costs correctness. For money, pay the latency.
• Asynchronous standby means a region disaster can lose the last seconds of committed payments. Mitigation: the processors hold the truth for authorized payments, and reconciliation replays them. If zero loss is required, the standby must be synchronous and nearby, which costs latency on every write.
• Idempotency keys stored in a global cache that replicates asynchronously is the classic double-charge bug.
• Hot merchants: one merchant producing 2,000 TPS overloads a single shard; split its ledger by sub-account and aggregate balances.

Direct answer

The design hinges on separating two situations that look identical from the outside: a payment request that was never sent to the processor, and one that was sent but the answer never came back. Never-sent requests are safe to queue durably and retry later, even on a different processor. Sent-but-unknown requests must only ever be retried with the same idempotency key (a unique ID attached to one payment attempt, so a repeated send carrying it is recognised as that same attempt rather than charged again) against the same processor, or resolved by asking that processor for the payment's status, because the customer may already have been charged. Around that core: persist every payment attempt and its outbound message in one database transaction (a transactional outbox: a table in the same database that records what still needs to be sent, written in the same transaction as the payment row so the two can never disagree), have workers send with exponential backoff and jitter behind a circuit breaker, tell the customer honestly that payment is pending, and drain the backlog at a controlled rate when the processor comes back.

Architecture

flowchart TD
    CO[Checkout] --> API[Payment service]
    API -->|one DB transaction| DB[(Payments + outbox)]
    DB --> W[Submitter workers]
    W --> CB{Circuit breaker}
    CB -->|closed| PSP[Processor]
    CB -->|open| WAIT[Stay queued]
    PSP -->|timeout| UNK[Status: unknown]
    UNK --> INQ[Status inquiry job]
    INQ --> PSP
    W -->|attempts exhausted| DLQ[(Dead-letter queue)]
    DLQ --> OPS[Ops console]
    API --> NOTIFY[Customer notifications]

1. Durable intake: the transactional outbox

When the customer clicks Pay, the payment service writes, in one database transaction, the payment row (status = pending_submission, amount, currency, order) and an outbox row ("submit payment 881 with processor key pay_881_a1"). If the service crashes a millisecond later, both exist or neither does. A separate worker reads the outbox and makes the external call. This is what "data is not lost" means concretely: the source of truth is our database, not an in-memory queue and not the processor.

The processor idempotency key is generated once per attempt and stored on the payment row, before anything is sent. Every retry of that attempt reuses it.

2. Payment states

The unknown state is the heart of duplicate prevention. It is never converted into failed by a timer, and never re-sent with a new key or to a second processor, because that is exactly the path that creates a second charge.

3. Retry and backoff

Transient faults are retried with exponential backoff (each wait double the last) capped at a maximum, with full jitter (each wait is a random value between zero and the computed delay) so that thousands of queued payments do not all retry in the same second when the processor recovers.

With a 1-second base, doubling, and a 5-minute cap, the maximum delays before each of the first 12 retries are 1, 2, 4, 8, 16, 32, 64, 128, 256, 300, 300, 300 seconds. The cumulative worst case after the 12th retry is 1,411 seconds, about 23.5 minutes. I cap automatic retries at that point for a customer-present checkout (the customer is watching the page in real time, as opposed to a payment being retried later by a background job with nobody waiting on it); after it the payment moves to needs_attention and the customer is told.

A circuit breaker sits in front of the processor. Its two main states borrow their names from an electrical circuit, which reads backwards to most people: closed means current (here, traffic) flows normally, and open means the circuit is broken and nothing gets through. So when the error rate over a sliding window (the failure rate measured over just the last minute or so, not since the service started, so an old problem cannot keep tripping the breaker forever) crosses a threshold, the breaker trips open and stops sending for a cooldown period, then lets a few probe requests through (a small number of real requests, sent on purpose to test whether the processor has actually recovered before trusting it with the full backlog). While open, work stays in the outbox instead of burning retries against a dead endpoint.

Retry only what is retryable: timeouts, connection errors, 5xx responses and explicit "try again" declines. A hard decline (insufficient funds, stolen card) is a definitive answer, not an outage.

4. User messaging

A customer cannot be told "paid" until there is an authorization. The pattern I use:

• For a short blip, the checkout waits a few seconds (a synchronous retry within the page's own timeout) and most customers never notice.
• Beyond that, the order is accepted as "Payment processing": inventory is reserved for a bounded time (say 30 minutes), the confirmation page says the order is received and that an email will confirm payment, and the Pay button is disabled for that order so the customer cannot start a second attempt.
• When the payment resolves, the customer gets a confirmation, or a link to try another payment method if it failed. If it never resolves within the reservation window, the reservation is released and the customer is told.

For high-risk or instantly delivered goods (gift cards, digital downloads) I do not accept the order pending payment; the customer sees "payment provider temporarily unavailable, please try again shortly", because delivering first and failing to collect is a direct loss.

5. Recovery and draining the backlog

When the circuit closes again, that is, the breaker judges the processor healthy and lets traffic flow normally once more, the backlog built up while it was open must drain without tripping the processor's rate limits or causing a second outage.

Worked example: a 30-minute outage at 50 payment attempts per second queues 50 × 1,800 = 90,000 payments. If the processor contract allows 200 TPS (transactions per second) and live traffic is 50 TPS, the drain can use 150 TPS: 90,000 ÷ 150 = 600 seconds, so the backlog clears in about 10 minutes. The drain order is: resolve every unknown first by status inquiry (those customers may already have been charged), then oldest pending_submission first, with live traffic always taking priority over backlog.

Before sending a queued payment, the worker re-checks that it is still wanted: the order was not cancelled, the customer did not pay with another method in the meantime, and the reservation has not expired.

6. Surfacing failures

Payments that exhaust retries, return unexpected responses or fail status inquiry land in a dead-letter queue (DLQ: a holding queue for messages that could not be processed automatically) and appear in an ops console with the full attempt history, processor responses and the order. Ops can re-drive them (same key), cancel them, or trigger customer outreach. Metrics and alerts: queue depth, oldest pending age, count in unknown, DLQ arrivals per minute.

7. Failover to a second processor

Failover is valuable for availability, but only pending_submission payments may move to a backup processor. An unknown payment on processor A that is resent to processor B can be charged by both, and no idempotency key spans two companies.

Trade-offs and pitfalls

• Queueing in memory (or in a broker without the database write, a broker being a standalone message-queue service such as SQS or RabbitMQ that the payment service writes to but that is not itself the source of truth) loses payments on a crash; the outbox costs one extra insert per payment and removes that class of loss.
• Treating timeouts as declines is the single most common source of double charges in this scenario.
• No jitter turns recovery into a synchronised retry storm that knocks the processor back over.
• Accepting orders pending payment trades conversion for credit risk; the reservation window and the product categories it applies to are business decisions to make explicitly.
• Unbounded retries mean a customer is charged hours later for an order they assumed failed and bought elsewhere; cap them and communicate.