Payment and Transaction Processing Systems Questions

Situation: A fintech targeting rapid scale to $1B ARR must decide between building an in-house payment processor (acquiring, rails, settlement, risk stack) or integrating multiple third‑party PSPs.

Evaluation (by criterion):

• Costs: Building is high CapEx and ongoing OpEx — engineering (payments rails, settlement, reconciliation), legal, compliance, and dedicated ops teams. Third‑party PSPs are OpEx (transaction fees, monthly minimums) and much lower upfront spend. At early scale, PSP fees are likely cheaper than the total cost of building and operating secure, reliable rails.
• Compliance (PCI & regulatory): In‑house forces full PCI DSS scope (likely SAQ D + external audits), card scheme registrations, money transmitter licensing, AML/KYC engineering and compliance teams. PSPs reduce PCI scope via tokenization and hosted fields (SAQ A/B); they handle scheme relationships and many regulatory obligations. Regulatory complexity (PSD2, local licensing) is nontrivial if you own settlement.
• Time-to-market: PSP integrations (orchestrated) enable launch in weeks/months. Building settlement, acquiring relationships, and compliance takes 12–24+ months.
• Routing control: In‑house offers maximum control over routing, dynamic optimization, and margin capture. Multiple PSPs with an orchestration layer can approximate this (smart routing, A/B failover) without building full acquiring.
• Fraud detection: PSPs provide baseline fraud tools; advanced, bespoke fraud models are often a competitive differentiator and easier to own. Hybrid approach: use PSP fraud tools plus your own risk scoring/ML layer.
• Operational complexity: Building multiplies complexity — settlement reconciliation, chargebacks, disputes, scheme reporting. PSPs centralize many responsibilities.
• Vendor lock‑in: Single PSP = lock‑in risk. Multiple PSPs + abstraction/orchestration minimize lock‑in and allow price/performance arbitrage.

Recommendation (for a startup targeting $1B ARR):

• Do not build full acquiring/settlement in year 0. Start with multiple vetted PSP integrations behind a payments orchestration layer you control (API + routing rules, metrics, retry logic, token vault, idempotency). This gives fast time‑to‑market, routing control, resilience, and mitigates single‑vendor risk.
• Invest early in: (1) payments orchestration (microservice/API), (2) unified reconciliation and observability, (3) in‑house fraud/risk scoring and chargeback management, (4) strong legal/compliance advisory to map licensing requirements as you scale.
• Track unit economics per volume band. When volume & margins reach thresholds where acquiring fees + cost of capital justify creating your own acquiring / settlement entity (typically tens to hundreds of millions in annual TPV), re‑evaluate building or purchasing an acquirer. At that point, migrate incrementally for specific corridors/products rather than a Big‑Bang replacement.

Actionable next steps:

1. Shortlist 3–4 PSPs covering target geographies/rails.
2. Build a minimal orchestration API with smart routing and telemetry.
3. Implement PCI scope reduction patterns (hosted fields, tokenization).
4. Launch with PSP primary + backup routing; build fraud ML pipeline in parallel.
5. Reassess building acquiring when predictable TPV, margins, and regulatory readiness justify the investment.

This hybrid path balances speed, cost, compliance risk, and routing control while preserving future optionality to bring more of the stack in‑house once economics and scale warrant it.

Requirements (assumptions): keep checkout available during short external-processor outages; never lose payment intents or customer data; prevent duplicate charges; provide clear UX; scale to peak traffic; auditability for reconciliation.

High-level architecture:

• Frontend → Payments API → Orchestrator/Payments Service → Durable Queue (persistent store) → Processor Worker(s) → External Processor.
• Persistent store: transactional DB (e.g., Postgres) for payment intent state + append-only event store/log for auditing.
• Durable queue: Kafka, AWS SQS with DLQ, or Redis Streams (replicated) to persist pending requests.

Flow:

1. Customer submits checkout; Payments API creates a PaymentIntent record (state=CREATED) and writes an idempotency key.
2. If external processor reachable, attempt synchronous authorization. If success → update state=CONFIRMED.
3. If unreachable or receives transient error, write message to durable queue and set state=AWAITING_PROCESSOR with attempt_count=0 and next_retry timestamp.

Retry & backoff:

• Worker consumes queue, performs processor call with exponential backoff + jitter (e.g., base 2s, cap 5 min) using attempt_count to compute delay.
• Implement circuit breaker per-processor: when failure rate > threshold, stop immediate retries and switch to delayed scheduled retries.
• Use visibility timeouts (SQS) or consumer offsets/leasing to avoid double-processing.

Duplicate prevention:

• Always send idempotency-key (payment_intent_id + merchant_id) to processor; store processor transaction id in PaymentIntent when received.
• Workers check PaymentIntent state and processor tx id before calling; if an in-flight message reappears, worker will detect completed state and ack without re-submitting.

User messaging / UX:

• Immediate feedback: show “Payment pending — we’ll finalize when connection is restored” for AWATING_PROCESSOR; allow user to navigate away and return.
• Provide email/SMS webhook on final success/failure.
• Offer explicit retry button for user-initiated retry (creates new intent or re-uses existing idempotency key depending on business rule).

Failure surfacing & reconciliation:

• DLQ for messages that exceed max attempts; move to “FAILED_NEEDS_MANUAL_REVIEW” state and notify ops/finance with full audit log.
• Admin dashboard: failed queue items, attempt history, processor responses, ability to requeue or mark reconciled.
• Once connectivity restored, workers automatically pick up queued messages and continue retries. For long outages, run bulk reconciliation: compare local PaymentIntents without processor_tx_id against processor transaction history (using idempotency keys) to avoid duplicates.

Observability & safety:

• Metrics: queue depth, retry counts, error rates, circuit-breaker status.
• Tracing: correlate payment_intent_id across services.
• Alerts: high queue depth, DLQ spikes, recon failures.

Trade-offs:

• Synchronous UX vs safety: immediate declined authorizations are best synchronously; fallback to queued async for availability.
• Choice of durable queue affects ordering and complexity (Kafka = strong ordering and replay; SQS = simpler, DLQ semantics).

This design guarantees durability (DB + persistent queue), safe retries with exponential backoff and circuit breaker, duplicate prevention via idempotency and state checks, clear user messaging for pending payments, and operational tooling to surface and resolve failed payments after connectivity is restored.

Benefits:

• Loose coupling: services (auth, capture, settlement, fraud, notifications) evolve independently and scale separately.
• Asynchrony and elasticity: absorb bursts (Black Friday) via durable event queues and auto-scaling consumers.
• Observability and auditability: event log is an immutable trail for reconciliation and forensics.
• Replayability: rebuild state or reprocess failed flows without re-running transactions.

Trade-offs:

• Complexity: distributed consistency, debugging, and operational overhead increase.
• Latency: eventual consistency can add end-to-end delay versus synchronous RPC.
• Testing and error-handling complexity (sagas, compensations).
• Operational cost: messaging infra, retention, monitoring.

How to use events (high-level flow):

• Authorization: PaymentService publishes PaymentAuthorized event (includes idempotency key, amount, auth_code, status). Downstream fraud scorer subscribes; notification service can inform customer of pending charge.
• Fraud scoring: FraudService consumes Authorization events, emits FraudScoreUpdated or FraudRejected. If high risk, emit HoldPayment / CancelAuthorization events consumed by Capture service.
• Capture: On business trigger or settlement window, CaptureService consumes PaymentAuthorized & FraudScoreUpdated, performs capture, emits PaymentCaptured (with capture_id).
• Settlement: SettlementService batches captures, emits SettlementInitiated events to bank connectors; on success emits SettlementCompleted, on failure emits SettlementFailed for retry/compensation.
• Reconciliation: ReconciliationService consumes SettlementCompleted and external bank webhooks (BankSettlementConfirmed) and produces ReconciliationResult events; mismatches create InvestigationRequired events.
• Notification: NotificationService subscribes to user-facing events (Authorized, Captured, Settled, Failed, Reconciled) to send emails/SMS.

Safe ordering and at-least-once delivery:

• Design for at-least-once: assume duplicates; make consumers idempotent using dedupe keys (payment_id + event_sequence or event_id stored in a lightweight idempotency store).
• Event schema: include event_id (UUID), causal metadata (parent_event_id), version, timestamps, and logical sequence numbers where ordering matters (e.g., authorization_sequence).
• Partitioning: route related events (same payment_id or merchant) to same partition/stream key to preserve order at partition level.
• Use transactional outbox pattern in services that change DB + publish events to avoid lost events.
• Sagas/Orchestrator vs. Choreography: for multi-step money flows prefer a saga coordinator for complex compensations (e.g., refund on failed capture), but keep simple flows choreographed to reduce coupling.
• Retries and DLQ: implement exponential backoff, poison message handling into DLQ + alerting, with human-in-loop for reconciliation edge-cases.
• Monitoring & metrics: track consumer lag, duplicate rate, processing latency, and reconciliation delta.

Example safety patterns:

• Idempotent handlers: store last_processed_event_id per aggregate.
• Compensating transactions: emit RefundRequested if capture fails after authorization.
• Reconciliation jobs: periodic full-replay or snapshot comparisons to detect missed events and trigger corrections.

This approach balances scalability and resilience while ensuring financial correctness through idempotency, partitioned ordering, transactional outbox, and clear compensation/saga logic.

Requirements & constraints:

• Functional: accept payments, refunds, settlements, reconciliation, dispute handling, audit trail.
• Non-functional: grow from 10K/day (~0.12 TPS avg) to 10M/day (~116 TPS avg) in 3–5 years; high availability (99.99%), PCI scope minimization, low-latency authorization, strong auditability.

High-level architecture (components):

• API Layer: API Gateway (rate limiting, auth, TLS termination, routing).
• Ingress services: Auth Service (OAuth/JWT), Client Validation Service.
• Core microservices (bounded contexts): Payment Orchestrator, Authorization Service, Risk/Anti-Fraud, Ledger (immutable transaction store), Settlement Service, Refunds, Disputes, Reconciliation, Notification.
• Event Backbone: Apache Kafka (durable, partitioned) for async flows and integration.
• Data stores: OLTP (sharded RDBMS or distributed SQL like CockroachDB/Postgres Citus) for ledger & strong consistency; NoSQL (e.g., DynamoDB/Cassandra) for idempotency, tokens, caching.
• Idempotency & dedupe service (idempotency keys).
• Integration layer: connector microservices for banks, card networks, PSPs (each has retry/backoff).
• Observability: metrics (Prometheus), tracing (Jaeger/OpenTelemetry), logging (ELK), SLO dashboards.
• Security & compliance: HSM/Key Manager, tokenization, PCI DSS scoping, audit logs.

Scaling patterns & capacity planning:

• Start single-region, design for multi-region. Use partitioning by merchant/account ID to shard Kafka topics and DB shards.
• Autoscale stateless services horizontally (K8s), right-size instances for peak-hours (estimate peaks ~3–5x average).
• For the ledger, use partitioned DB (range/hash) per merchant to scale writes; consider event-sourced ledger (append-only Kafka topic + materialized views) for audit and replayability.
• Use CQRS: write path -> synchronous validation + produce event to Kafka; read path -> materialized views for low-latency queries.
• Backpressure: circuit breakers, bulkheads, token buckets to protect downstream PSPs.
• Batch settlements & reconciliation to reduce external API calls.

Eventing choices:

• Kafka as central event bus (exactly-once semantics using idempotency + transactional producers where available). Topics partitioned by merchant/region.
• Use compacted topics for latest-state (accounts), append-only for audit ledger.
• Use schema registry (Avro/Protobuf) to enforce contracts and enable consumer evolution.

Maintainability & incremental growth strategies:

• Start with clear bounded contexts and API/event contracts. Use semantic versioning and schema registry to evolve without breaking consumers.
• Feature flags & canary deployments for risky changes.
• Backward-compatible event evolution: additive fields, default values.
• Encapsulate PCI-sensitive operations behind a single service and tokenization layer to limit scope later.
• Observability-first: instrument everything early; define SLOs and alert thresholds.
• Automated chaos/DR drills and load-testing that emulate 10M/day patterns before hitting them.
• Documentation, runbooks, and ownership (service per team) to reduce coupling.

Trade-offs:

• Event sourcing/append-only ledger increases complexity but gives replayability and audit — valuable for payments.
• Distributed SQL eases transactional guarantees but needs careful schema design for sharding.

Summary:
Design for partitioned scaling (merchant/account), async event-driven flows with Kafka + CQRS for throughput and auditability, strong idempotency and anti-fraud upstream, and rigorous contract/versioning to allow incremental growth without big rewrites.

Start with the PCI DSS control areas most relevant for a cloud payments platform:

• Network segmentation and firewalling (isolate cardholder data environment — CDE)
• Data protection (encryption at rest/in transit, key management)
• Access control and MFA (least privilege, strong auth, logging)
• Logging, monitoring and incident response (SIEM, retention)
• Secure software development and change control (SAST/DAST, patching)
• Vulnerability management and testing (ASV scans, penetration testing)
• Physical and cloud provider responsibilities (shared responsibility model)

Practical architecture approaches to reduce merchant PCI scope

• Redirect (hosted payment page): Merchant redirects customers to provider’s hosted checkout (or an iframe-hosted full-page). Provider handles CDE; merchant has minimal scope. Trade-off: UX flow and brand control reduced; integration simpler.
• Hosted fields / iFrame snippets: Provider serves secure iframes for card inputs embedded in merchant page; card data posts directly to provider. Trade-off: better UX, lower scope for merchant DOM, but merchant must ensure parent page integrity (no XSS).
• Tokenization: Provider exchanges PAN for non-reversible token. Merchant stores tokens instead of PANs, reducing scope for storage and processing. Trade-off: reliance on provider for vault availability and cross-provider portability.
• Client-side encryption (CSE): Card data encrypted in browser with provider public key before transmission to merchant; merchant only sees ciphertext. Trade-off: complexity (key distribution, crypto lifecycle), still must protect JS integrity and key endpoints.
  Combined pattern: use hosted fields + tokenization for best balance (good UX, minimal merchant CDE).

Residual merchant responsibilities after scope reduction

• Ensure page/application integrity (no XSS, protect supply chain, CSP, SRI)
• Securely handle tokens and merchant-side systems (access control, logging)
• Maintain secure integration (TLS, validate redirects, webhook security)
• Document and enforce vendor SLA, key management and incident procedures
• Demonstrate required controls during assessment (segmentation proof, scanning, policies)

Key trade-offs summary:

• UX vs scope: redirects minimize scope but reduce UX/branding; hosted fields improve UX with slight residual scope.
• Complexity vs control: CSE gives control over crypto but increases operational burden.
• Vendor reliance vs portability: tokenization reduces merchant risk but creates dependency on provider.

Conclude with best practice: pick the minimal-scope integration that satisfies customer UX and resiliency needs, enforce strong front-end integrity controls, and formalize shared-responsibility and incident handling with the payment provider.