Platform Architecture Fundamentals Questions
Foundational concepts for designing large-scale platform architectures and distributed systems, including service boundaries, microservices patterns, API design, data consistency models, caching strategies, messaging and event-driven patterns, scalability, fault tolerance, observability, and deployment considerations.
HardTechnical
76 practiced
Design a disaster recovery plan for a payments platform deployed across AWS and GCP with RPO 5 minutes and RTO 10 minutes. Describe cross-cloud replication approach, orchestration of failover, DNS cutover and TTL considerations, secret and key management across clouds, testing cadence, and stakeholder coordination with legal and regulators.
Sample Answer
Requirements clarification:- RPO = 5 minutes (data loss ≤5m), RTO = 10 minutes (service restored ≤10m).- Multi-cloud: primary on AWS, secondary on GCP (or active-passive).High-level approach:- Active-passive, near-synchronous cross-cloud replication for transactional data; automated orchestration to failover within 10m; pre-provisioned infra in secondary to meet RTO.Cross-cloud replication:- Use change-data-capture (CDC) streaming: primary writes to durable log (e.g., AWS Kinesis / DynamoDB Streams or RDS binlog) → CDC pipeline (Debezium on ECS/Fargate) → replicate to GCP Pub/Sub and to secondary DB (Cloud SQL / Spanner). For low latency, use cross-region VPN/Interconnect or AWS Direct Connect + Cloud Interconnect via partner to reduce RTT. Ensure idempotent writes and strict ordering; keep a replication lag metric with alerting. For object storage, use multi-cloud replication tool (rclone/Cloud Storage Transfer or custom process) with versioning; ensure eventual consistency <5m.Orchestration of failover:- Centralized orchestrator (Lambda / Cloud Functions + Step Functions / Workflows) that: - Validates health probes and replication lag <RPO. - Promotes read-replica in GCP to primary role (or flips DNS to AWS endpoints pointing to GCP load balancer). - Runs data reconciliation jobs (final log apply). - Runs smoke tests and health checks; only then mark as healthy.- Include manual abort/approve gates for regulator-required approvals, but keep automated hot-path for low-severity scenarios.DNS cutover and TTL:- Use short TTLs (30–60s) for service CNAMEs. For customer-facing domain, plan staged DNS: keep a fast-cutover domain (api-fast.example.com) with 30s TTL used by clients, and a long-lived root alias with higher TTL. Use Global Anycast DNS (Route 53 + Cloud DNS secondary via Traffic Director or via external DNS provider that supports weighted failover). Pre-create records for both clouds; orchestrator switches weight or updates records atomically. Account for DNS caching — keep client SDK retry/backoff and implement health-check-based client routing.Secrets and key management:- Mirror secrets securely: use AWS Secrets Manager / KMS with automatic export to a secure vault in GCP (HashiCorp Vault with replication, or use Cloud KMS + external key wrap). Store encryption keys in HSM-backed KMS in both clouds with key rotation synced. Use envelope encryption: private master key never leaves HSM; replicate wrapped keys across clouds. Ensure access policies and audit logs exist in both clouds. Rotations performed centrally with careful key versioning to allow rollback.Testing cadence and validation:- Continuous DR testing: weekly automated failover rehearsals in a staging replica; monthly partial failovers (non-production traffic); quarterly full failover drills involving production data in a maintenance window. Each test follows runbook, measures RPO/RTO, collects metrics, and produces postmortem. Maintain synthetic transactions and chaos tests (network partition, process kill).Stakeholder coordination, legal & regulators:- Maintain a DR runbook and RACI matrix; designate DR lead, infra, security, legal, compliance, and PM. For regulated payments, pre-notify regulators per required SLAs and keep audit trail of tests. Ensure data residency and cross-border transfer policies reviewed with legal; implement data classification and consent model. Provide regularly scheduled reports (quarterly) to regulators, and immediate notification channels for actual failovers. Run tabletop exercises with legal/compliance to validate notification/contractual obligations.Trade-offs & controls:- Active-active reduces failover time but increases complexity (consensus/dupe-suppression). Our active-passive with near-sync CDC meets RPO/RTO with simpler correctness guarantees. Monitor costs vs RTO; warm standby in GCP minimizes cost while meeting RTO.Key metrics to monitor:- Replication lag, end-to-end transactional latency, failover time (orchestration), DNS propagation metrics, secret sync success, and smoke-test pass rates.This plan emphasizes automated, auditable orchestration, secure replicated keys, short DNS TTLs with client resilience, regular testing, and clear regulatory coordination to meet 5‑minute RPO and 10‑minute RTO.
HardTechnical
76 practiced
Evaluate event sourcing versus CRDTs for a collaborative real-time document editing platform that must operate active-active across regions. Discuss conflict resolution, consistency convergence, latency implications, storage and replay costs, operational complexity, and trade-offs between deterministic merges and per-client operational transforms.
Sample Answer
Situation: We need an active‑active, multi‑region collaborative editor. Two broad architectures to compare: append‑only event sourcing (with operational transforms or server-side transforms) vs CRDTs (commutative data types replicated peer-to-peer).Conflict resolution & convergence:- Event sourcing + deterministic server merge: events are authoritative and ordered; conflict resolution depends on your merge logic (e.g., OT, last‑write‑wins, intent‑preserving transforms). Deterministic merges can give strong, repeatable outcomes but require careful global ordering or vector clocks to avoid divergence.- CRDTs: designed for automatic, monotonic convergence without central ordering. For text, sequence CRDTs (RGA, Logoot, Yjs/Automerge variants) ensure eventual consistency by construction. CRDTs remove the need for manual conflict resolution logic.Latency implications:- Event sourcing with centralized ordering (or quorum) adds cross‑region latency for strong ordering. You can mitigate with local leaders + causal delivery but risk increased complexity.- CRDTs allow local, low‑latency updates; replicas apply ops locally and sync asynchronously, so user experience is best for real‑time typing.Storage & replay costs:- Event sourcing stores the full operation log; replaying reconstructs state and produces auditability. However logs grow indefinitely — need compaction/snapshots and garbage collection.- CRDTs also store tombstones or identifiers; some CRDTs accrue metadata bloat (identifier growth, tombstones). They require compaction (garbage collection/snapshots/merge compaction) to control storage.Operational complexity:- Event sourcing: simpler to reason about audit trail, time‑travel, and compliance. But building robust global ordering/causality and scalable streaming (cross‑region replication, snapshotting, retention) is complex.- CRDTs: more complex data structure design and debugging (invariants, metadata growth), fewer operational coordination points. Tooling exists (Yjs, Automerge, delta CRDTs) but integrating with persistence, access control, and complex domain ops adds work.Trade-offs: deterministic merges vs per‑client OT- Deterministic server merges (centralized or consistent replicated log) give repeatability and easier auditing; OT requires careful intent preservation and transform functions compatible across clients. OT historically delivers compact operations but is brittle at scale (transform explosion, hard to prove convergence).- Per‑client OT (clients transform incoming ops) offers low latency but needs strict causal delivery and consistent transformation rules; scaling multi‑region increases failure modes.- CRDTs represent a middle ground: they give per‑client low latency and formal convergence guarantees without OT transforms, but at cost of metadata and non‑trivial CRDT design to implement rich editor semantics (cursors, rich text, embedded objects).Recommendation (Solutions‑Architect view):- For the lowest latency active‑active UX and simpler multi‑region writes, choose a battle‑tested CRDT library (delta CRDTs to reduce bandwidth) plus snapshotting and periodic compaction; front it with regionally deployed sync servers for presence/awareness, and central services for persistence, analytics, and access control.- If auditability, deterministic replay, or complex business rules requiring a single canonical sequence are dominant, use event sourcing with a hybrid: local buffering + causal ordering (Lamport/vector clocks) and periodic global reconciliation; add OT only if you need compact op representation and can tolerate higher operational complexity.- Hybrid approach: use CRDTs for real‑time collaboration (low latency, availability), and materialize a canonical event stream for auditing/analytics by converting CRDT deltas into an append‑only event store with snapshots.Key operational mitigations:- Implement snapshot + GC for logs/tombstones- Use delta‑CRDTs and compression to control bandwidth- Build deterministic conflict resolution hooks for semantics (e.g., rich text annotations)- Invest in end‑to‑end testing and chaos testing for multi‑region divergence scenariosSummary: CRDTs favor availability and latency; event sourcing favors auditability and deterministic replay. Choose based on which non‑functional requirement (real‑time UX vs canonical audit/ordering) is primary, or combine both to get the best of each.
MediumTechnical
66 practiced
Propose an API versioning strategy for a public platform used by many third-party clients. Address URI vs header versioning, semantic versioning concepts applied to APIs, deprecation policies and windows, client migration plans, SDKs, and how to run multiple API versions concurrently in production with minimal overhead.
Sample Answer
Situation: Designing a robust API-versioning strategy for a public platform used by many third-party clients.Proposal (high-level):- Version identifier: use explicit versioning (vN) in the URI for major versions and use headers for non-breaking evolution and negotiation. URI: /v1/orders/... = stable public contract; Accept: application/vnd.company.v1+json or custom header X-API-Version for clients that prefer header-based negotiation.- Semantic-style rules: adopt semantic versioning principles mapped to API changes: - MAJOR (vN in URI) = breaking changes to request/response shapes, auth model, or semantics. - MINOR = additive, backward-compatible features (new fields, endpoints). Expose via header negotiation or feature flags. - PATCH = bug fixes, non-functional changes.Deprecation policy and windows:- Public deprecation lifecycle: Announce deprecation 90 days before end-of-life (EOL) for MINOR, 12 months for MAJOR versions (adjustable by impact). Steps: Announcement → Grace period with warnings in responses (Deprecation and Sunset headers) → Migration assistance → Final EOL with 30-day final notice.- Provide machine-readable deprecation metadata: GET /.well-known/api-versions returns versions, deprecation dates, migration docs.Client migration plan & SDKs:- Publish clear migration guides and sample code for each breaking change.- Maintain official SDKs per language tied to API major versions; SDKs implement negotiation, retries, and fallbacks. Backport critical patches to older SDK branches during deprecation window.- Offer migration tooling: linters, contract-diff utilities, and compatibility test suites clients can run.Running multiple versions concurrently (low overhead):- Single codebase, router-based versioning: route /vN/* to versioned controllers where business logic shares common services. Use adapter/wrapper layers for response shaping.- Use feature flags and toggles to gate new behavior before promoting to a new major.- Shared core services (auth, rate-limiting, data layer) remain unchanged; only thin compatibility layers differ.- Automate CI/CD to run integration tests per active version; containerize versioned adapters to scale independently.- Use API gateway for routing, rate-limiting, and injecting Deprecation headers; gateway handles header/URI negotiation and can route traffic to appropriate backend flavor.Operational/compatibility controls:- Strong contract testing (consumer-driven contract tests), schema validation, and extensive monitoring (versioned usage metrics, error rates).- Fail-safe defaults: when Accept header ambiguous, default to latest non-breaking minor of requested major.- Rate-limit and throttle migration traffic with staged rollout and partner-specific feature flags.Trade-offs:- URI major-versioning is explicit and cache-friendly; header negotiation is cleaner for non-breaking evolution but less visible. Combining both gives flexibility.- Longer deprecation windows reduce disruption but increase maintenance cost; mitigate with shared core and adapters.Result: Clear, predictable versioning that minimizes client disruption, simplifies migration, and keeps operational overhead low through shared services, adapters, and automation.
MediumSystem Design
58 practiced
Design a user profile microservice that must support 1,000,000 reads/sec and 50,000 writes/sec. Requirements: read latency under 50ms, eventual-consistency acceptable across follower regions but strong write consistency per user, multi-region read replicas, and safe cache invalidation. Describe high-level architecture components including API layer, caching, partitioning strategy, replication approach, and trade-offs.
Sample Answer
Requirements clarification:- 1,000,000 reads/s, 50,000 writes/s, read <50ms, eventual cross-region consistency acceptable, but strong per-user write consistency, multi-region read replicas, safe cache invalidation.High-level architecture:- API layer: stateless API gateways (regional) behind LB. Read API routes to nearest read replica; write API routes to that user's write leader (see partitioning). Gateways handle auth, rate-limiting, request validation, and optimistic concurrency headers (ETag/version).- Partitioning: consistent-hash on user_id → shard. Each shard has a single primary (leader) for writes and multiple read replicas. Leader assignment is per-shard and can be placed in a preferred region (sticky to user’s home region) to guarantee single-writer strong consistency per user.- Primary data store: distributed KV/row store (e.g., CockroachDB with affinity or a partitioned NoSQL like Cassandra with a leader layer). Writes go to shard leader; leader applies and assigns monotonic version numbers.- Replication approach: leader asynchronously replicates to regional follower replicas via log shipping/CDC. Followers serve reads; replication is async across regions (eventual). For local strong reads, gateway can forward reads to leader when needed.- Caching: global edge cache (CDN/Redis clusters per region) for reads. Cache stores profile snapshots keyed by user_id + version. Use short TTL (e.g., 30s) + conditional fetch with ETag to keep latency low while bounding staleness.- Safe cache invalidation: on successful write, leader publishes invalidation event to a global pub/sub (e.g., Kafka/Cloud PubSub). Regional caches subscribe and either invalidate or update (push-update) the cached entry. Include version in events so caches only accept newer updates. For race-safety, use versioned keys (user:version) and store current_version pointer; clients use ETag to detect stale reads.- Read path: edge cache → regional replica (if miss) → optionally leader for strongly consistent read. Use read-through pattern for cache misses.- Write path: gateway → shard leader (single-writer) → persist synchronously on leader → ack to client → enqueue replication + publish invalidation event.Scalability and performance:- Scale shards horizontally; target shard write capacity so 50k writes/s spreads across N leaders. Reads served from regional caches and followers to meet 1M/s.- Use multi-level caches (edge CDN for public fields, regional Redis for full profile) to reduce DB load and meet <50ms.Trade-offs:- Strong per-user writes via leader gives simple consistency but requires leader placement and failover complexity. Failover must transfer leadership and ensure no split-writes.- Async cross-region replication keeps low write latency but allows short-term staleness.- Cache invalidation via pub/sub is reliable but adds operational complexity; versioning mitigates race conditions.- Using leader-per-shard limits write concurrency per shard but matches requirement (50k writes/s manageable with shard count).- Alternative: quorum reads/writes (e.g., quorum-based DB) would provide different consistency/latency trade-offs but adds cross-region latency for writes.Operational concerns:- Automated shard rebalancing, leader failover with coordinated state transfer, monitoring for replication lag, circuit-breakers for heavy read-to-leader fallbacks.- Backups, GDPR data locality, and schema migration strategies.This design balances low-latency reads at scale with per-user strong writes, safe cache invalidation via versioned events, and multi-region read capacity with acceptable eventual consistency.
MediumTechnical
61 practiced
Discuss strategies to scale stateful services such as session stores, WebSocket servers, or game servers. Cover partitioning strategies (sticky sessions vs externalized state), session affinity, consistent hashing, state externalization patterns, and the trade-offs between latency, complexity, and consistency.
Sample Answer
Requirements and constraints:- Low-latency real-time interactions (WebSockets/game servers), high availability, multi-region, user-affinity, and eventual or strong consistency depending on game logic.High-level strategies:1. Sticky sessions (session affinity)- Route a client to the same server (load balancer cookie or reverse-proxy). Simple, minimal latency, server keeps state in-memory.- Good when working set per server fits memory and failover tolerates reconnects.- Trade-offs: poor elasticity (hard to rebalance), single-server blast radius, harder multi-region routing.2. Externalized state- Move session/state to a shared store (Redis, DynamoDB, etc.) or stateful service (stateful microservice).- Enables stateless frontends, easy scaling, failover and rebalancing.- Trade-offs: added network hops and serialization cost → higher latency; need caching and locality to mitigate.3. Partitioning / sharding- Hash-based partitioning (consistent hashing): assign client/session IDs to shards so that rebalances move few keys. Works well for many servers and stateful caches.- Range partitioning: better for queries by range but less flexible.- Example: game rooms partitioned by room ID; chat channels sharded by channel ID.4. Hybrid approaches- Sticky + external store (local cache + write-through): low-latency reads from local memory, writes replicated to central store asynchronously or with consensus for critical state. On failover, restore from external store.- Rendezvous hashing for multi-tier routing to reduce coordination.Consistency, latency, complexity trade-offs:- Strong consistency (consensus/primary-replica) increases write latency and complexity but required for authoritative game state.- Eventual consistency with CRDTs or optimistic updates reduces latency but needs conflict resolution and careful game invariants.- Local caches reduce latency but complicate invalidation and increase stale-read risk.Operational patterns:- Autoscale with graceful drain: use health checks + connection draining to move new connections away before rebalancing.- State transfer: move active sessions via handoff protocols or migrate state snapshots during rebalancing.- Multi-region: keep authoritative state per region and use replication lanes; route clients to nearest region with fallback.Recommendations:- For low-latency, small-state real-time apps: sticky sessions with local cache + external persistent backing for recovery.- For large-scale, multi-tenant systems: externalize state with consistent hashing sharding, local caching, and careful choice of consistency model per state type.- Measure and tune: latency budgets, failover RTO, and scale tests; instrument hot shards and rebalancing effects.This balanced approach clarifies trade-offs and lets stakeholders choose based on latency, complexity, and consistency needs.
Unlock Full Question Bank
Get access to hundreds of Platform Architecture Fundamentals interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.