Payment and Transaction Processing Systems Questions

Requirements & constraints:

• Support multi-currency pricing, payments, payouts, and reporting while preserving legal/tax needs per locale.
• Low-latency UX, strong auditability, and PCI scope minimization.

Architecture / technical considerations:

• Central Payments Service with clear responsibilities: settle, convert, tax calculation, ledger entries. Keep idempotent APIs and event-sourced ledger for auditability.
• Money type: use integer minor units (cents) + currency code; avoid floats.
• Exchange rates: source from trusted providers, timestamped and versioned; store rate id with each transaction for reproducibility. Support both “spot” and provider-marked rates and configurable spread.
• Rounding: define currency-specific rounding rules (banker’s rounding vs half-up) and per-operation rounding points (display vs settlement). Apply rounding consistently in ledger flows.
• Taxes/regulatory: plugin tax rules engine per jurisdiction (VAT, GST, withholding) with tax IDs, receipts, and withholding calculations. Allow per-listing tax overrides and legal hold flags.
• PCI & security: minimize PCI scope with tokenization, use certified PSPs, vault tokens, strong encryption for sensitive fields, strict logging controls, and quarterly SAQ/ROC processes.
• Reconciliation & failures: asynchronous retry queues, compensating transactions, reconcile batch vs ledger daily, surface payment exceptions to ops dashboard.

UX considerations:

• Show prices in guest-local currency with clear note of payout currency for hosts; show conversion rate, fees, and estimated payout date.
• Offer explicit UX for multi-currency hosts to choose payout currency and preview net payout.
• Surface taxes/fees line items and provide receipts including tax IDs.

Testing & rollout:

• Unit tests for money math and rounding, integration tests with mocked PSPs and rate providers, end-to-end test environment with deterministic rates.
• Property-based tests for rounding invariants and reconciliation tests for ledger balance.
• Staged rollout: feature flags → internal dogfood → country-by-country canary (low volume) → full rollout; monitor revenue reconciliation, chargeback rates, latency, and user support tickets. Maintain kill-switch to revert PSP or feature flag quickly.

Trade-offs:

• Centralized conversion simplifies consistency but adds latency; consider client-side display conversion for UX with server authoritative settlement.
• Onshore PSPs reduce regulatory friction but increase integration surface.

This design prioritizes correctness, auditability, and clear UX while allowing iterative rollout and safe operations.

Requirements:

• Strong atomicity: either ledger debit/credit, anti-fraud decision, notification and external bank transfer all succeed, or the user-visible result is rolled back or compensated.
• Low latency for user-facing success; high reliability; external bank API is flaky/uncontrollable.
• High throughput and auditability.

High-level architecture:

• Orchestrator service (or external coordinator) that drives the payment workflow.
• Per-service local transaction boundaries: Ledger service, Anti-fraud service, Notification service, and an External Bank Adapter.
• Durable event store / saga log that records each step and its idempotent command IDs.

Approaches, trade-offs:

1. Two-Phase Commit (2PC)

• How: Use XA-like distributed transaction across services and bank (prepare/commit).
• Pros: Synchronous atomicity, simple mental model.
• Cons: Blocks resources (locks), poor availability if coordinator or bank hangs, most external banks don’t support prepare/commit, limited scalability. Not practical in microservices + external APIs.

2. Sagas with compensations (recommended)

• How: Orchestrated saga pattern: orchestrator issues steps: reserve ledger funds (ledger: hold), call anti-fraud (approve/decline), call bank transfer (execute), finalize ledger (capture), notify customer.
• If a later step fails (e.g., bank transfer), run compensating actions: release ledger hold, record reversal entries, send failure notification. Persist saga state in event store for retry.
• Pros: Highly available, scalable, tolerant of partial failures, works with external HTTP APIs.
• Cons: Only eventual consistency; requires careful compensation logic and idempotent ops; business complexity (e.g., partial money movement at external bank may be irreversible — must detect and reconcile).

3. External Transaction Coordinator / Hybrid

• How: External durable coordinator (workflow engine — e.g., Temporal, Cadence) orchestrates saga steps and provides retry, timeouts, and long-running workflows.
• Pros: Simplifies orchestration, reliable retries, visibility, built-in idempotency primitives and dead-letter handling.
• Cons: Adds dependency and operational complexity.

Idempotency & retries:

• All external-facing endpoints accept an idempotency key (UUID tied to payment attempt). Operations must be idempotent: ledger holds/captures use idempotency keys; bank adapter de-duplicates requests and stores external reference IDs.
• Retry policies: exponential backoff with jitter for transient failures; circuit breaker for downstream instability; configured max attempts then escalate to manual reconciliation.

Eventual consistency & reconciliations:

• Maintain audit log and periodic reconciliation job between ledger and bank statements. For irreversible bank transfers that partially complete, use reconciliation to create compensation transactions and mark sagas as “requires manual review” if automated compensation cannot guarantee correctness.
• Use compensating transactions that are ledger-native (reversals) and customer-facing notifications marking state clearly.

Example flow (orchestrated saga):

1. Create Payment (idempotency_key) -> ledger.create_hold(amount, idempotency_key)
2. antiFraud.check(hold_id)
3. bankAdapter.transfer(hold_id, idempotency_key) — store external_tx_id
4. ledger.capture(hold_id, external_tx_id)
5. notify.success(user)

On bank failure:

• bankAdapter fails -> orchestrator -> ledger.release_hold(hold_id) (compensate) -> notify.failure(user) -> persist failure & schedule reconciliation.

Key operational practices:

• Strong observability: traces, durable saga state, alerting for stuck sagas.
• Short holds and TTLs to limit exposure.
• Clear SLAs for manual reconciliation and dispute handling.

Recommendation:
Use an orchestrated saga implemented with a durable workflow engine (Temporal/Cadence) + strict idempotency keys, compensating transactions, and robust reconciliation for edge cases where external bank behavior is non-atomic. This balances availability, scalability, and practicality for real-world bank integrations.

Requirements & constraints:

• Functional: ingest bank confirmations (async), match to internal ledger entries, mark settled/failed, expose ops dashboard, searchable audit trail.
• Non-functional: eventual consistency, high throughput (kTPS), idempotency, tamper-evident audit, regulatory retention.

High-level architecture:
Bank feeds → Ingest API / Message Queue (Kafka) → Validation & Normalization service → Dedup/Idempotency store → Reconciliation engine → Ledger write (intent & finalization) + Audit store → Notification & Dashboard services

Key components and responsibilities:

1. Ingest layer: accept webhooks/files; immediately ack receipt (HTTP 202) to banks. Push raw events to Kafka topic partitioned by payment id to preserve order.
2. Validation/Normalization: canonicalize formats, enrich with metadata, compute a deterministic event-id (hash of bank_id, message_id, amount, timestamp).
3. Deduplication/Idempotency: use a strongly-consistent store (e.g., Redis with write-through or relational DB) to store event-id with TTL and processed-state. If event-id exists, skip further processing.
4. Reconciliation engine: for each normalized event, attempt match against ledger by payment reference, amount, and time-window. Use a state machine: UNMATCHED → PENDING_CONFIRMATION → MATCHED → SETTLED/EXCEPTION. Persist reconciliation records in transactional DB.
5. Ledger writes: separate two-step write—(a) record bank confirmation in reconciliation table (append-only), (b) idempotent ledger update via unique idempotency key. Use database transactions or sagas to ensure either both reconciliation record and ledger update are applied or compensating action recorded.
6. Audit trail: append-only immutable store (WORM-enabled DB or object store + signed events). Store original message, normalization diff, decision, operator overrides, timestamps, and user id. Include cryptographic signing or hashes for tamper evidence.
7. Dashboard & Ops: materialized views for finance: match rate, latency, exceptions, unpaired items. Provide drill-downs and replay controls (reprocess, mark manually reconciled) with RBAC and audit logging.
8. Double-processing protection: idempotent ledger API keyed by reconciliation-id; ledger rejects duplicate reconciliation-ids. Use at-least-once processing but ensure idempotency to prevent side effects.

Eventual consistency and correctness:

• Accept that bank confirmations may arrive late/duplicate/incorrect. Use optimistic matching with periodic re-run (windowed joins) and background reconciliation jobs to resolve stale UNMATCHED items.
• Use Kafka for durable event stream and tombstone events (cancellations). Reconciliation engine subscribes and can rehydrate state from event stream to recover.

Operational concerns & observability:

• Metrics: inbound rate, dedupe rate, match latency, exceptions per minute.
• Traces & logs: distributed tracing for each payment-id, structured logs for decisions.
• Alerts: threshold-based alerts for rising exception rates or backlog.
• Playbooks & manual tools: allow finance to tag, escalate, or force-settle with two-person approval; every manual action written to audit trail.

Trade-offs:

• Strong consistency (single DB) simplifies correctness but limits scale; chosen design uses idempotent operations + event sourcing to scale while preserving correctness.
• Dedup store TTL must balance storage and regulatory replay needs; keep raw events indefinitely in immutable audit store.

This design ensures eventual consistency via durable streams and background reconciliation, deduplicates by deterministic event-ids + idempotent ledger writes, maintains tamper-evident audit trails, offers finance-friendly dashboards and manual controls, and prevents double-processing through idempotency keys and transactional safeguards.

Requirements & constraints:

• Strong consistency for balance-updating operations (debits/credits/settlements).
• Low-latency, highly available reads across regions.
• Global scale, tolerance to regional failures, accurate fraud checks.

High-level approach:

• Partition by accountId (range or hashed) so each account’s updates map to a single partition (shard). Each shard is replicated across multiple regions (N replicas) and has a single logical leader for serializing updates.

Consensus & write path:

• Use Raft/Paxos per-shard (leader + followers). All balance-updating requests go to the shard leader; leader appends to an append-only ledger (durable WAL), replicates to followers and commits via quorum. This gives linearizable writes and exact ordering (CP behavior for writes).
• To tolerate leader region failure, automatic leader election among replicas. Use write quorum across majority of replicas spanning regions to remain correct.

Read routing & availability:

• Serve reads from local follower replicas for low latency (AP-style), but attach a monotonic version (log index or vector-clock) to each committed write.
• Provide two read modes:
  1. Strong read: route to leader (or use quorum reads) when client requests up-to-date balance (e.g., just after a payment).
  2. Fast read: serve from local follower with last-known committed index. Clients can accept slightly stale reads for UX (balances view), or use cache-invalidation/refresh to get strong read on demand.

Caching:

• Local read caches / CDN with short TTL and version-aware entries (store committed_log_index with cached value). Read-through cache consults follower; if client demands freshness and cache log_index < required index, redirect to leader/quorum.
• Use per-account cache keys to avoid cross-account inconsistency.

Reconciliation & network partitions:

• Writes always require quorum; if a partition prevents quorum, leader refuses writes (reject/queue). To preserve correctness, reject local-only writes rather than accept divergent updates.
• Use an append-only ledger per shard as source-of-truth. After partition heals, leader election ensures single leader; followers catch up by applying the WAL, producing deterministic state.
• For out-of-band or retries, make all operations idempotent (operation IDs) and use a deterministic commutative compensating transaction model for business-level reconciliation (e.g., settlement adjustments) where absolute correction is needed. Maintain an immutable transaction history for audits.

Fraud checks:

• Two-tier fraud model:
  1. Synchronous, deterministic rules in the write path on the shard leader (velocity limits, balance checks, blacklists). These must run inline before commit — low-latency, rule-based.
  2. Asynchronous/near-real-time ML scoring: stream events (transaction, context) to a central risk service and feature store; apply heavier ML models and, if high-risk, trigger immediate mitigation: reverse transaction via compensating entry or freeze account. To ensure no missed fraud on leader rejection, the leader can call a lightweight risk API synchronously that uses cached features and deterministic heuristics; if ML offline flags, follow up with compensating action.
• Keep risk datastore replicated and consistent enough for reads in leader region; for critical checks consult quorum if necessary (trade-off with latency).

Operational considerations & trade-offs:

• Latency vs strong consistency: writes incur cross-region quorum latency; mitigate by placing a majority of replicas in nearby regions or using geo-aware quorums (fast quorums).
• Availability: By refusing writes during minority partitions we preserve correctness; provide clear UX messages and queued retry semantics.
• Observability: audit trails, immutable ledger, transaction IDs, metrics for replication lag, and reconciliation reports.
• Security: end-to-end encryption, strict authentication/authorization, HSMs for sensitive keys, and rate-limiting to avoid fraud attack amplification.

Why this works:

• Per-shard consensus gives strict ordering and linearizability for updates.
• Local reads + versioning provide low-latency availability while allowing clients to request strong consistency when required.
• Append-only ledger + idempotent ops enable deterministic reconciliation post-partition and full auditability.
• Inline deterministic fraud checks prevent clear fraud patterns immediately; heavier ML can catch complex patterns asynchronously with compensating actions.

Requirements & constraints:

• Functional: authorize, capture, refund, reconciliation, idempotency, audit logs.
• Non-functional: Availability 99.999% (~5.26 min downtime/year), P99 latency <100ms for auth, strict PCI-DSS constraints, regional compliance.

High-level architecture:

• API Gateway (edge) → Authenticator & Rate Limiter → Payment Orchestrator → Connectors (card networks, acquirers) → Ledger + Audit Store
• Deploy across 3 AZs per region, active-active across 2 regions for disaster recovery.

Redundancy & deployment:

• Microservices packaged in immutable containers, deployed via Kubernetes with pod anti-affinity across AZs.
• Stateful components:
  • Ledger: primary-replica distributed DB (e.g., CockroachDB or Spanner) with multi-region replication and synchronous replication within region for consistency.
  • Message bus: Kafka with 3+ brokers per cluster, cross-region MirrorMaker for DR.
• Use blue-green/rolling canary deployments; automated traffic shifting with health gate.

Health checks & observability:

• Liveness + readiness endpoints; readiness gates block traffic until caches warmed and DB connections healthy.
• Synthetic transactions (end-to-end test payments) every 30s from multiple locations.
• Metrics: request latencies, error rates, queue lengths, connector latencies; traces with 100% sampling for payment flows; alerts based on SLO burn rates and error budget.
• Log immutable audit trail (WORM) stored separately.

Failover strategies:

• Connector failure: circuit breaker + retry with exponential backoff + failover to secondary acquirer per routing rules. Fail open for non-critical failures, fail closed for suspected fraud.
• Region outage: automated DNS failover + state reconciliation via CDC and idempotent replay. Use leader election for scheduled tasks.
• Database node failure handled via automated failover; application uses optimistic retries with idempotency keys.

Incident runbook (short):

• Trigger: SLO alert or synthetic failure.
• Pager -> On-call engineer triages using runbook play:
  1. Check SLO dashboards, synthetic test results, connector statuses.
  2. If single connector, route to backup and disable bad connector in routing table.
  3. If DB lagging, enable read-only degrade, throttle non-essential traffic, escalate to DB SME.
  4. For region outage, trigger DR runbook: shift DNS, validate ledger reconciliation, run consistency checks.
  5. Post-incident: blameless postmortem within 72 hours, capture RCA, action items with owners.

On-call & testing budget:

• On-call rotations: 2-week primary, 2-week secondary; small team with SRE and payments SME. Budget 1.5 FTE SRE + 0.5 FTE payments SME per global region for 99.999%.
• Testing: invest in continuous chaos engineering (fault injection of connectors, AZ/region kill tests) and daily synthetic end-to-end tests. Allocate 20% of engineering time for observability/DR drills and SLO maintenance.
• Runbook drills quarterly and full DR rehearsals annually; include business stakeholders for escalation paths.

Trade-offs:

• Cost vs availability: multi-region synchronous replication increases cost and write latency; choose sync in-region + async cross-region and rely on idempotent replay for DR.
• Complexity vs safety: circuit breakers and retries must be conservative to avoid cascading failures.

This architecture emphasizes active-active redundancy, automated failover, extensive observability, and disciplined operational processes to meet 99.999% availability and strict latency SLOs.