Requirements & constraints:
- Strong linearizable consistency for account balances (no double-spend).
- Global low-latency reads for customers.
- High availability where possible, but correctness (financial safety) is priority.
- Multi-region deployment, per-account scale.
High-level architecture:
- Partition accounts by account-id hash into shards. Each shard is a replicated state machine (append-only ledger + current balance).
- Deploy each shard as a Raft cluster (3-5 nodes) with members placed across regions (leader preferably co-located with a primary write region; followers in other regions).
- Global router + local edge proxies in each region for client connections.
Replication & consensus:
- Use leader-based consensus (Raft/Paxos) per shard to ensure linearizability. All writes go through the shard leader which appends to its WAL and replicates to followers synchronously to a commit quorum (majority).
- Persisted, append-only ledger with monotonically increasing term/index. Use fencing tokens on leader change.
Read routing & low-latency reads:
- Primary approach: linearizable reads served by leader. To avoid cross-region latency for reads, use:
- Read Index (Raft ReadIndex) or leader lease: local read replicas can serve linearizable reads if they hold a valid lease or use ReadIndex to ensure no leader change — enabling local-region reads without contacting leader for every read.
- If lease/read-index cannot guarantee, offer two read modes:
- Strong reads: forwarded to leader (global latency).
- Bounded-staleness local reads: served from closest follower with metadata (last-applied index) and a staleness SLA (e.g., <=250ms). UI/clients choose preference.
- Cache hot read-only fields (non-critical) in regional caches with short TTL and validation via ledger index for consistency-sensitive operations.
Synchronous vs asynchronous replication trade-offs:
- Synchronous (majority commit) guarantees durability and linearizability on commit → higher write latency (cross-region RTTs if leader/followers remote) but required for correctness.
- Asynchronous replication to distant replicas improves availability/performance but risks data loss on leader failure and stale reads — unacceptable for balance writes. Use asynchronous replication only for analytics/backup replicas (eventual consistency), not for primary ledger commits.
Handling network partitions without violating correctness:
- Favor CP (consistency + partition tolerance) for balances: require majority quorum for commits. If leader cannot reach majority, stop accepting writes — fail closed.
- During partition:
- If local region retains leader and quorum -> normal reads/writes.
- If isolated and cannot form quorum -> demote to read-only mode; return explicit error for writes (or queue locally but do not apply to ledger until quorum).
- Prevent split-brain via quorum rules + fencing tokens; new leader only becomes active if it has majority and higher term.
- For reads during partitions, serve bounded-staleness reads locally with clear client indicators; disallow any operation that requires strict linearizability (e.g., transfers) until quorum restored.
Cross-shard / multi-account transactions:
- Use two-phase commit across shard leaders with coordinator; but because 2PC blocks, prefer application-level compensation or avoid cross-shard strong transactions where possible. If necessary, use Raft log-based distributed transactions with global ordering (slower).
Operational concerns (SRE-focused):
- Extensive observability: per-shard metrics (replication lag, commit index, leader term), alerts for follower lag and quorum loss.
- Automated failover: leader election via Raft; rolling upgrades with leader relocation.
- Chaos testing and partition drills; clear SLOs: write availability may be lower than read availability, error budget budgeting for write rejections during partitions.
- Auditing/immutability: cryptographic linking of ledger entries, offline reconciliation procedures, and safe replay for disaster recovery.
Summary of trade-offs:
- Choose CP for correctness: synchronous majority commits + leader-based consensus.
- Optimize read latency via leases/read-index and bounded-staleness local reads.
- Accept higher write latencies globally and write unavailability during quorum loss in exchange for no incorrect balance states.
