Payment Processing and Fraud Prevention Architecture Questions
Architectural patterns for building scalable payment processing systems, including integration with payment gateways and processor APIs, ensuring idempotent transactions, reliable ordering and delivery, fraud detection and risk scoring integration, security and compliance (PCI), data protection, and multi-region resilience within a distributed systems context.
HardSystem Design
50 practiced
Design a ledger service that guarantees correct merchant balances (preventing double-spend) under concurrent payments and refunds while allowing eventual consistency for non-financial services such as reporting. Explain your approach to concurrency control, partitioning/sharding, idempotency, reconciliation, and how to scale the ledger for millions of merchants.
Sample Answer
Requirements & constraints:- Strong correctness for merchant balances (no double-spend), support concurrent payments/refunds.- Allow eventual consistency for non-financial reads (reporting, analytics).- Support millions of merchants, high throughput, HA, and operational reconciliation.High-level architecture:- Front-end API + gateway → Ingest/validation → Partitioned Ledger Service → Durable append-only journal (WAL / Kafka) → Balance store (per-shard authoritative state) → Read replicas / materialized views for reporting → Reconciliation & background processors.Partitioning / sharding:- Shard by merchant_id (or merchant_id % N) so each merchant’s financial state lives on a single shard/leader to ensure serializable updates for that merchant. Range or hash-based consistent hashing; support dynamic resharding via shard manager + rebalances with state transfer using snapshot + WAL replay.Concurrency control & preventing double-spend:- Use leader-based single-writer-per-shard: the shard leader serializes all financial operations for merchants it owns. Within a shard: - Operations are applied as idempotent, ordered commands persisted to the WAL (Kafka topic partition per shard). - Apply optimistic concurrency tokens for cross-shard operations (if needed) but prefer modeling transfers as two-phase ledger entries (debit on source, credit on destination) coordinated by a coordinator with prepared/commit phases to maintain atomicity. - For single-merchant ops (payments/refunds), leader serializes and checks available balance before committing; failure rejects.Idempotency:- All client requests carry a client-generated idempotency key (UUID). Shard maintains recent idempotency map (persisted/compacted) keyed by key → result. Incoming request first checks map to return previous result; otherwise persists entry and publishes operation.Durability & ordering:- Write: validate → append to shard WAL (Kafka partition) → apply to in-memory state and persistent local DB (e.g., RocksDB/Postgres) in same transaction where possible → ack.- Use Kafka exactly-once semantics (producer transactions) or broker-side guarantees; local DB updates occur after WAL commit to allow replay on leader failover.Eventual-consistent reads / reporting:- Materialize event streams to OLAP stores (ClickHouse/BigQuery) asynchronously; these read models are eventually consistent and used for analytics, dashboards.Reconciliation & auditing:- Maintain immutable transaction journal (append-only) per merchant. Regular background reconciliation compares computed balances from journal to authoritative balance store; mismatch triggers alerts and automatic repair workflows (replay journal, corrective entries) with human review gates for large diffs.- Provide proof structures (Merkle trees per-shard) for efficient integrity checks and to support external audits.Scaling to millions of merchants:- Horizontal shard scaling: Increase shard count and reassign merchants. Use tenant grouping (hot merchants separated) and autoscaling for shard leaders.- Stateless front-end and horizontally scaled ingestion.- Partition hot merchants into dedicated shards; use caching for read-heavy merchants (redis read-through).- Use multi-region replicas with shard leaders elected per region for geographic locality; cross-region replication via async WAL replication.Failure modes & trade-offs:- Single-writer-per-shard simplifies correctness but creates hotspots; mitigate via re-sharding and splitting hot tenants.- Cross-shard atomic operations introduce complexity (2PC) — avoid by routing related accounts to same shard when possible or use distributed transactions with idempotent resolve/retry.- Latency vs global consistency: Strong per-merchant consistency, eventual across reporting systems.Metrics & operational tooling:- Monitor WAL lag, idempotency map size, reconciliation diffs, per-shard QPS, latency, and error rates. Provide admin tools for replays, resharding, and manual corrections.Why this meets requirements:- Single-writer shards, durable ordered journal, and idempotent operations prevent double-spend under concurrency.- Eventual read models decouple reporting to allow scale.- Reconciliation + immutable journal ensures correctness, auditability, and recovery for millions of merchants.
EasyTechnical
74 practiced
Compare common integration patterns with payment gateways: hosted checkout pages, iframe/hosted fields, direct API tokenization, redirect flows, and PSP SDKs. For each pattern describe PCI scope implications, UX trade-offs, integration complexity, and typical failure modes a Solutions Architect should plan for.
Sample Answer
Hosted checkout pages- PCI scope: Minimal for merchant (SAQ A) — sensitive data handled entirely by PSP.- UX trade-offs: Fast to deploy, secure, but brand/flow control limited; may feel disjointed.- Integration complexity: Low — configure via dashboard or simple redirect parameters.- Typical failures to plan: Third-party outages, regional blocking of PSP domain, inconsistent styling, returning POST/callback failures; plan retries, webhooks validation, and clear fallbacks.Iframe / Hosted fields- PCI scope: Reduced (SAQ A-EP or SAQ A depending on implementation) — merchant page hosts iframe but card fields isolated.- UX trade-offs: Better branding and seamless UX while keeping card data out of merchant DOM.- Integration complexity: Medium — requires JS integration, CORS and CSP handling, accessibility work.- Failures: JS load failures, CSP blocking, iframe sandboxing issues, slow tokenization; plan graceful degradation and client-side monitoring.Direct API tokenization (merchant sends card to PSP via backend)- PCI scope: High (SAQ D) unless using direct client-to-PSP tokenization; merchant handles card data so full compliance required.- UX trade-offs: Full control of UX and flow, supports advanced features, but heavy compliance burden.- Integration complexity: High — backend, encryption, secure storage, auditability.- Failures: Network errors, PCI audit gaps, improper key management, failed token exchanges; plan secure logging, retries, idempotency, and outage modes.Redirect flows (e.g., bank auth / 3DS redirects)- PCI scope: Minimal for merchant when PSP handles payment pages; merchant still responsible for redirect integrity.- UX trade-offs: External flow interrupting checkout; often required for regulatory auth (3DS) or local methods.- Integration complexity: Low–medium — handle return URLs, state validation.- Failures: User abandons during redirect, failed signature/nonce checks, callback lost; plan timeout handling, UX messaging, and server-side reconciliation.PSP SDKs (mobile/web SDKs)- PCI scope: Typically minimal (SAQ A/A-EP) because SDKs handle sensitive fields; confirm SDK attestation.- UX trade-offs: Native, high-quality UX and features (biometrics, token lifecycle) with moderate vendor lock-in.- Integration complexity: Medium — embed SDK, handle lifecycle, update management.- Failures: SDK bugs, version incompatibility, app store delays, platform permissions; plan SDK health monitoring, feature flags, and fallback to web checkout.Cross-cutting recommendations- Always use tokenization and idempotency keys, enforce TLS, validate webhooks, and implement monitoring/alerting.- Design for graceful degradation: clear error states, retry policies, reconciliation jobs, and an agreed SLA with the PSP.- Choose pattern based on risk appetite, required UX control, compliance cost, and expected failure modes.
HardSystem Design
36 practiced
Design a global active-active payment processing architecture to support 10,000 TPS with P95 API latency under 200ms across 3 regions. Requirements: idempotent transactions, correct merchant balances, failover for processor outages, and handling per-region data residency constraints. Describe service decomposition, data replication approach, conflict resolution, routing, and reconciliation.
Sample Answer
Requirements clarification:- 10k TPS global, P95 <200ms across 3 regions, active-active- Idempotent transactions, correct merchant balances (no double-spend), processor failover, per-region data residencyHigh-level approach (goal: low-latency local acceptance, strong per-merchant correctness, eventual global convergence):- Each region exposes identical API endpoints behind a geo-LB (Anycast + region-aware DNS). Clients hit nearest region.- Merchants are assigned a “home-region” based on residency rules. Writes for that merchant prefer home-region; reads served locally when allowed.- Use local fast path for authorization + reservation; durable global ledger for final settlement.Service decomposition1. API Gateway / Ingress (rate limiting, auth, idempotency key check)2. Transaction Router / Local Orchestrator (per-request routing, local validations)3. Local Ledger Service (append-only, per-merchant ordered log, implements idempotency and reservation)4. Global Settlement / Replicator (async cross-region replication, ensures global ordering per merchant)5. Balance Service (per-merchant strong-serializable view via single-writer shard; offers reservation/commit/rollback)6. Processor Adapters (connect to acquirers/processors; circuit breaker, retry, failover pool)7. Reconciliation & Anti-Entropy (ledger compare, compensations, alerts)8. Monitoring/Tracing/Config (SLOs, dynamic routing)Data model & sharding- Merchant-centric sharding: merchant_id → shard using consistent hashing. Each shard has a logical leader (home-region preference). This keeps all balance-affecting operations for a merchant serialized.- Per-merchant ledger: append-only sequence numbers (incremental). Each transaction record: txn_id (client-provided idempotency key), seq_no, amount, type, status, origin_region, timestamp, signature.Idempotency- API requires client-supplied idempotency key. Gateway checks local idempotency cache + persists idempotency entries in the local ledger atomically with the transaction append. Replays return same result.- For cross-region duplicate requests (client retried to different region), idempotency keys replicate with the ledger; local node consults replicated index (eventual) and uses sequence acceptance rules to avoid double-processing.Balance correctness & conflict resolution- Strong per-merchant correctness via single-writer logical lease: use a lightweight Paxos/Raft per-shard or a leader-election mechanism with failure detection and short leader lease. Leader may be in any region but prefer home-region to satisfy residency.- Writes (authorize/reserve/settle) are sent to the current shard leader. To meet latency, allow non-leader region to accept request as a proxy: it creates a reservation locally and forwards to leader asynchronously, returning accepted to client when local durability is achieved AND a local reserved balance check passes. To avoid double-spend, local reservation consumes from a local “available reserve pool” which is refreshed by applying leader-ordered entries; proxies maintain conservative limits.- Conflict resolution: globally, the leader-ordered per-merchant seq_no determines canonical order. If two regions accepted conflicting reservations due to partition, reconciliation uses seq_no and timestamps; later conflicting acceptance is rolled back or compensated.Replication approach- Per-shard log replication: leader appends, then ships to followers in other regions via change-propagation (RAFT log shipping or append-only WAL replication). For performance, use: - Sync replication to a quorum including at least one other region when merchant residency requires stronger guarantees; otherwise leader-local + async followers for latency. - Use change-batches over gRPC with compression and idempotent apply.- Snapshot & checkpoint for new followers.Routing & failover- Client -> nearest region. Local orchestrator looks up merchant shard leader: - If local is leader: handle directly. - If not: forward to leader (fast RPC) or accept as proxy with limited local-only reservation (fast path) and forward.- Processor adapters: pool per-region with health checks; if a processor is down, adapter retries with alternate processor and marks failed processors, circuit-breaker plus backoff. Outages cause transactions to be queued in durable retry queues; business SLA defines retry window.- Regional outage: leader-lease failover elects new leader in another region. To avoid split-brain, use lease + quorum and monotonic epoch numbers.Latency trade-offs- Fast-path accepts local reservation and returns quickly (sub-200ms) — final settlement may be async. For strict consistency merchants can opt-in for sync commit (higher latency).- Use client-observable statuses: PENDING → COMMITTED → SETTLED.Reconciliation & anti-entropy- Continuous ledger tail-checks and background reconciliation: compare per-merchant sequence logs across regions; detect missing/inconsistent entries, generate compensating transactions or repair operations.- Daily/real-time double-entry reconciliation against processor statements. Provide audit trails and proofs (signed ledger entries).- Dispute resolution flow: attach immutable transaction receipts (hashes) signed by service.Security & compliance (data residency)- For merchants constrained to region: all PII and primary ledger must remain in home-region. Cross-region replication transfers only pseudonymized or encrypted tokens; final settlement messages omit PII. Optionally, store shadow global indices with pointers only.- Encrypt in transit and at rest. Maintain region-specific key management.Observability & SLOs- Instrument per-request tracing, tail-latency dashboards, and per-shard KPIs.- Automated alerting on replication lag, leader flaps, and processor error rates.Failure scenarios (short)- Region network partition: leader lease prevents split-brain; proxies accept local reservations but mark them PENDING; when partition heals, leader orders and reconciliation either commits or compensates.- Processor outage: adapter retries alternate processors, durable retry queue ensures eventual dispatch, reconciliation adjusts merchant balances after settlement.- Duplicate client retries: idempotency keys prevent double-processing; if cross-region duplicates arrive before replication, reconciliation uses seq_no to pick canonical and compensates duplicates.Why this design- Per-merchant sharding + single-writer leader gives strong correctness with bounded coordination.- Local fast-path + proxy model meets latency while keeping global correctness through leader ordering and reconciliation.- Configurable sync vs async replication allows trade-off between strictness and latency to match merchant SLAs.- Clear reconciliation and audit ensure correctness in long tail and outages.Key metrics to validate during implementation: end-to-end P95 latency, replication lag percentiles, leader failover time, TPS per shard, reconciliation drift rate, and transaction loss (zero).
HardTechnical
49 practiced
As a Solutions Architect advising a fintech, evaluate building an in-house payment processing capability versus integrating multiple third-party payment service providers. Consider costs, compliance overhead (PCI and regulatory), time-to-market, routing control, fraud detection, operational complexity, and vendor lock-in. Provide a recommendation for a startup targeting rapid scale to 1B USD ARR.
Sample Answer
Situation: A fintech targeting rapid scale to $1B ARR must decide between building an in-house payment processor (acquiring, rails, settlement, risk stack) or integrating multiple third‑party PSPs.Evaluation (by criterion):- Costs: Building is high CapEx and ongoing OpEx — engineering (payments rails, settlement, reconciliation), legal, compliance, and dedicated ops teams. Third‑party PSPs are OpEx (transaction fees, monthly minimums) and much lower upfront spend. At early scale, PSP fees are likely cheaper than the total cost of building and operating secure, reliable rails.- Compliance (PCI & regulatory): In‑house forces full PCI DSS scope (likely SAQ D + external audits), card scheme registrations, money transmitter licensing, AML/KYC engineering and compliance teams. PSPs reduce PCI scope via tokenization and hosted fields (SAQ A/B); they handle scheme relationships and many regulatory obligations. Regulatory complexity (PSD2, local licensing) is nontrivial if you own settlement.- Time-to-market: PSP integrations (orchestrated) enable launch in weeks/months. Building settlement, acquiring relationships, and compliance takes 12–24+ months.- Routing control: In‑house offers maximum control over routing, dynamic optimization, and margin capture. Multiple PSPs with an orchestration layer can approximate this (smart routing, A/B failover) without building full acquiring.- Fraud detection: PSPs provide baseline fraud tools; advanced, bespoke fraud models are often a competitive differentiator and easier to own. Hybrid approach: use PSP fraud tools plus your own risk scoring/ML layer.- Operational complexity: Building multiplies complexity — settlement reconciliation, chargebacks, disputes, scheme reporting. PSPs centralize many responsibilities.- Vendor lock‑in: Single PSP = lock‑in risk. Multiple PSPs + abstraction/orchestration minimize lock‑in and allow price/performance arbitrage.Recommendation (for a startup targeting $1B ARR):- Do not build full acquiring/settlement in year 0. Start with multiple vetted PSP integrations behind a payments orchestration layer you control (API + routing rules, metrics, retry logic, token vault, idempotency). This gives fast time‑to‑market, routing control, resilience, and mitigates single‑vendor risk.- Invest early in: (1) payments orchestration (microservice/API), (2) unified reconciliation and observability, (3) in‑house fraud/risk scoring and chargeback management, (4) strong legal/compliance advisory to map licensing requirements as you scale.- Track unit economics per volume band. When volume & margins reach thresholds where acquiring fees + cost of capital justify creating your own acquiring / settlement entity (typically tens to hundreds of millions in annual TPV), re‑evaluate building or purchasing an acquirer. At that point, migrate incrementally for specific corridors/products rather than a Big‑Bang replacement.Actionable next steps:1. Shortlist 3–4 PSPs covering target geographies/rails.2. Build a minimal orchestration API with smart routing and telemetry.3. Implement PCI scope reduction patterns (hosted fields, tokenization).4. Launch with PSP primary + backup routing; build fraud ML pipeline in parallel.5. Reassess building acquiring when predictable TPV, margins, and regulatory readiness justify the investment.This hybrid path balances speed, cost, compliance risk, and routing control while preserving future optionality to bring more of the stack in‑house once economics and scale warrant it.
HardTechnical
39 practiced
For a payment that requires authorization, settlement, ledger update, and downstream fulfillment, analyze trade-offs between using two-phase commit/distributed transactions (XA) versus sagas with compensating transactions. Propose a robust saga-based architecture that minimizes inconsistent states and handles failures such as partial refunds and late-arriving chargebacks.
Sample Answer
Two-phase commit/XA vs Sagas — trade-offs- XA (distributed transactions) provides strong atomicity across services but requires all participants to support XA, creates tight coupling, long-held locks, reduced availability, and poor scalability — risky in payment flows with external PSPs and fulfillment systems.- Sagas (long-running, compensating transactions) favor availability and loose coupling. They accept eventual consistency and require well-designed compensations. For payments, sagas map better to heterogeneous systems (PSP, ledger, fulfillment) and allow partial progress with compensations for failures.Recommended robust saga-based architecture1. Pattern: Orchestrated saga with durable orchestration service + local transactional outbox in each service. - Orchestrator coordinates steps: Authorization → Capture/Settlement → Ledger update → Fulfillment initiation. - Each participant performs its local action inside a local DB transaction that writes an outbox event (transactional outbox). A reliable delivery mechanism publishes events to the orchestrator.2. Idempotency & correlation: All operations carry global saga_id, step_id, and idempotency keys so retries are safe.3. Compensations: Implement explicit compensating actions per step: - For Authorization: void auth (if not captured). - For Capture/Settlement: issue refund (partial or full) via PSP and create compensating ledger entries. - For Ledger update: write reversing ledger entries (not delete). - For Fulfillment: cancel shipment/recall and create financial adjustment. Compensations are themselves sagas when they involve multiple systems (e.g., refund requires PSP + ledger).4. Failure modes handling - Partial refunds: Model money flows as immutable journal entries. Refunds produce negative entries linked to original payment_id; reconciliation service enforces ledger invariants (balance = sum(entries)). Orchestrator tracks intended settlement amount; if partial refund occurs, continuing fulfillment depends on business rules (allow partial fulfillment or wait). - Late-arriving chargebacks: Ingest chargeback events via a dedicated webhook consumer that starts a remediation saga: mark transaction disputed, create dispute ledger entries, attempt to reverse settled funds if possible, notify fraud/ops for manual review when automated compensation cannot complete. Maintain dispute state machine (open, contested, resolved).5. Safety mechanisms - Exactly-once processing: outbox + deduplication store; consumers detect processed saga_id/step_id. - Timeouts and compensating triggers: Steps set deadlines; orphaned sagas after timeout are automatically compensated or routed to manual review. - Reconciliation job: Periodic end-to-end reconciliation between PSP, ledger, and fulfillment; mismatches create corrective sagas. - Circuit breakers and backoff for external PSP failures; escalate to manual queues.6. Observability & ops - Central saga monitoring UI with live state, history, and manual intervene (retry/compensate/skip). - Audit logs (immutable) for financial/regulatory traceability. - Metrics & alerts: stuck sagas, compensations rate, reconciliation drift.Why this works- Keeps services decoupled and scalable, avoids distributed locks, and models money as immutable entries so compensations are safe and auditable. Combining orchestrated sagas, transactional outbox, idempotency, timeouts, and reconciliation minimizes inconsistent states and provides deterministic handling for partial refunds and late chargebacks while preserving operational controls for exceptions.
Unlock Full Question Bank
Get access to hundreds of Payment Processing and Fraud Prevention Architecture interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.