State Management and Data Flow Architecture Questions
Design and reasoning about where and how data is stored, moved, synchronized, and represented across the full application stack and in distributed systems. Topics include data persistence strategies in databases and services, application programming interface shape and schema design to minimize client complexity, validation and security at each layer, pagination and lazy loading patterns, caching strategies and cache invalidation, approaches to asynchronous fetching and loading states, real time updates and synchronization techniques, offline support and conflict resolution, optimistic updates and reconciliation, eventual consistency models, and deciding what data lives on the client versus the server. Coverage also includes separation between user interface state and persistent data state, local component state versus global state stores including lifted state and context patterns, frontend caching strategies, data flow and event propagation patterns, normalization and denormalization trade offs, unidirectional versus bidirectional flow, and operational concerns such as scalability, failure modes, monitoring, testing, and observability. Candidates should be able to reason about trade offs between latency, consistency, complexity, and developer ergonomics and propose monitoring and testing strategies for these systems.
HardTechnical
52 practiced
Describe the primary partial-failure modes in data flow (e.g., downstream write failure, message queue backlog, cache pump failures). For each, propose immediate mitigation steps and long-term engineering changes an SRE should drive to harden the system.
Sample Answer
Below I list common partial-failure modes in data flow, immediate mitigations to stabilize service during an incident, and longer-term engineering work SRE should drive to harden the system.1) Downstream write failure (DB or external API errors)- Immediate mitigations: - Switch to a degraded write mode: reject non-critical writes or return cached/stale reads; enable fail-open/fail-closed policy depending on correctness needs. - Throttle incoming traffic (rate-limit/slowdown) and return clear 503s to upstream callers. - Wake on-call owners for the downstream; engage a circuit breaker to avoid cascading retries.- Long-term hardening: - Implement idempotent writes, retry with exponential backoff and jitter, and a durable retry queue. - Add bulkheads and circuit breakers; define SLOs for downstream dependencies and fallbacks. - Capacity planning and runbooks; synthetic tests and contract tests for downstream APIs.2) Message queue backlog (consumer lag)- Immediate mitigations: - Temporarily increase consumer parallelism (scale replicas or partitions) and prioritize critical topics. - Throttle producers or shed low-priority traffic; enable consumer fast-forward/skip where safe. - Monitor lag and progress; if poisoned messages block, move them to DLQ for inspection.- Long-term hardening: - Autoscaling consumers, partitioning strategies, backpressure-aware producers, and idempotent consumers. - Implement DLQs, replay tooling, and visibility (per-message processing time histograms). - Define retention, capacity forecasts, and chaos tests for surge scenarios.3) Cache pump failures (cache warming/refresh fails or stampede)- Immediate mitigations: - Serve stale-but-valid cache entries (stale-while-revalidate) and disable aggressive invalidations. - Throttle client refresh requests; re-enable golden master or read-through fallback to DB. - If stampede is occurring, apply temporary locking / request coalescing.- Long-term hardening: - Use distributed locking or request coalescing libraries, TTLs with jitter, and cache warming strategies. - Implement multi-tier caches, circuit breakers, and monitoring on hit-rate, eviction, and origin load. - Define fallback content policies and automated warmers for deployment events.4) Partial data corruption or schema drift in pipeline- Immediate mitigations: - Stop pipeline ingestion, switch to read-only consumers, quarantine affected partitions. - Reprocess from safe checkpoints; notify stakeholders and enable alert for consumers.- Long-term hardening: - Schema registry with compatibility checks, end-to-end data validation, automated canary replays, and provenance metadata. - Implement replayable storage and versioned schemas.Cross-cutting SRE actions:- Runbooks with precise playbooks, automated mitigations (autoscaling, circuit breakers), robust observability (SLO-based alerts, end-to-end traces, per-component dashboards), chaos testing, and regular dependency reviews and SLAs.
MediumSystem Design
39 practiced
Design a caching strategy for a read-heavy service serving 100k RPS globally with 1M writes/day. Requirements: <50ms median read latency, eventual consistency acceptable (stale reads <5% within 30s), and cost-conscious. Describe cache layer placement, invalidation approach, TTL strategy, and how you would scale and monitor the caches.
Sample Answer
Requirements clarification:- 100k RPS read-heavy, 1M writes/day (~11.6 writes/sec), <50ms median read latency, eventual consistency OK with <=5% stale reads within 30s, cost-sensitive.High-level design:- Multi-tier cache: CDN/edge for public static or cacheable HTTP responses → regional in-memory cache (Redis cluster or Memcached) per region → local per-app LRU in-process cache for microsecond access to very hot keys.- Flow: Client → Edge CDN (TTL + stale-while-revalidate) → Regional Redis read-replica clusters → App local cache fallback → Origin DB.Invalidation & Consistency:- Accept eventual consistency: use asynchronous, event-driven invalidation. - On write, application publishes change events to a durable pub/sub (e.g., Kafka or Redis Streams). - Regionally subscribed cache workers process events and evict/update keys in Redis and optionally push invalidation to CDN via purge API or use cache-key versioning. - For safety: write-through small critical keys if strict freshness needed; otherwise rely on invalidation + TTL. - Use monotonic version numbers / cache key versioning to avoid race conditions (key:vN).TTL strategy:- Global default short TTL = 30s to meet staleness SLA, plus: - Hot-key adaptive TTLs: observe access frequency and extend TTL (up to e.g., 5 minutes) using LFU metrics to reduce origin load and cost. - Use stale-while-revalidate at edge: serve slightly stale content while async refresh happens. - Negative-caching for missing keys (~60s) to reduce repetitive DB hits.Scaling:- Regional Redis clusters sized by RPS and working set; use clustering and partitioning (hash slots) to scale horizontally.- Autoscale read replicas and cache node groups based on metrics (throughput, CPU, memory, network).- Use client-side sharding or consistent hashing to distribute load.- App local caches bounded by memory; LRU eviction.Monitoring & SLOs:- Key metrics: read latency (P50, P95), cache hit ratio (edge, regional, app), origin QPS, write/event processing lag, eviction rate, memory, CPU, network.- Alerts: - Hit ratio drop > X% (configurable) - Redis slow commands / latency > threshold - Event consumer lag > 5s - Eviction spikes or OOM.- Dashboards per region and global, with synthetic tests for TTL/invalidation correctness.- Runbooks: rollback purge storms, backpressure on writes, and fail-open (serve stale content) to preserve latency.Trade-offs and cost considerations:- CDN + regional Redis reduces global latency and origin cost; adding strong consistency (sync invalidation) would raise cost/latency. Adaptive TTLs balance cost and staleness.- Use versioned keys to avoid expensive global cache purges; only purge CDN for high-impact changes.This design meets <50ms median reads (edge + regional caching), keeps stale reads <5% within 30s via 30s TTL and rapid event-driven invalidations, and scales cost-effectively with autoscaled regional caches and adaptive TTLs.
MediumSystem Design
53 practiced
Design an event-driven architecture (commands → events → read models) for a service where clients require denormalized read views for low-latency access. Explain guarantees (ordering, at-least-once delivery), pipeline components, and SRE responsibilities around monitoring, reprocessing, and ensuring eventual convergence of read models.
Sample Answer
Requirements:- Low-latency denormalized read views for clients, strong ordering per-aggregate, at-least-once delivery of events, eventual consistency acceptable for reads, ability to reprocess events to rebuild views.High-level architecture:Commands → Command Handler (validates, applies business rules) → Event Store (append-only log, durability, source-of-truth) → Event Broker/Stream (Kafka) → Event Processor(s) → Read-model writers (materialized views: Redis, ElasticSearch, or denormalized SQL tables) → Clients (serve reads via cache/edge).Pipeline components and responsibilities:- Command Handler: synchronous validation, idempotency keys, write to Event Store in a transaction.- Event Store: immutable, ordered by aggregate id; exposes changelog to broker (log-compaction for retention).- Event Broker: partitioned by aggregate id to guarantee ordering; provides at-least-once delivery.- Event Processors: consumer groups that project events into read models; implement idempotent handlers and track consumer offsets + processed-event IDs.- Read-model stores: optimized for queries; updated by processors; support bulk rebuild.Guarantees and how to achieve them:- Ordering: Kafka partitions keyed by aggregate id ensure per-aggregate ordering; processors must process partition sequentially.- At-least-once delivery: broker may redeliver; processors must be idempotent and use exactly-once-like patterns where feasible (consume-process-commit with transactional sinks or dedup tables).- Durability: event store + replicated broker.SRE responsibilities:- Monitoring & alerting: track lag (consumer lag), error rates, retry queues, read-model staleness (age of last applied event), SLOs for read latency/consistency. Dashboards: broker health, partition distribution, GC/pause, processing throughput.- Reprocessing & recovery: provide tooling to rewind/rehydrate (reset consumer offsets, replay topics or restore from event store), run black-box replays in staging, support partial rebuilds (per-aggregate ranges), validate via checksums between source and read-model.- Automation & safety: support safe rollbacks (consumer versioning, dual-write or shadow mode), automated canary replays, rate-limited replays.- Ensuring eventual convergence: enforce idempotency, implement dead-letter queues for poison messages with alerting and manual triage, continuous background validators that scan event-store vs read-model and surface discrepancies for automated or manual reconciliation.- Capacity & ops: capacity plan for broker storage and consumer throughput, runbooks for partition rebalance, node failures, and reprocessing steps; rehearsed incident playbooks and postmortems.Key trade-offs:- Strong per-aggregate ordering vs cross-aggregate transactions (not supported); accept eventual consistency for cross-entity views.- Exactly-once requires more complexity (transactions, two-phase commits); prefer idempotent processors + monitoring for practical reliability.
HardTechnical
53 practiced
You manage a distributed cache tier that occasionally experiences corrupted entries after a rollback. Propose detection, quarantine, and remediation procedures to ensure corrupted cache data is flushed or repaired without causing large-scale performance degradation.
Sample Answer
Clarify scope & constraints:- Cache type (e.g., Redis cluster, memcached), consistency model (eventual/strong), TTL behavior, traffic patterns, and acceptable outage/latency SLOs. Assume distributed in-memory cache with replicas and rolling rollbacks cause sporadic corrupted values.Detection:- Add checksum/version metadata per cached value (e.g., value + 32-bit checksum + schema version). Compute checksum on write and verify on read.- Monitor read-check failures rate; create alert when failures > X per minute or >Y% of reads.- Instrument application to log and tag “checksum_mismatch” with key, node, timestamp, request id. Aggregate in metrics (Prometheus) and traces.Quarantine:- When checksum fails, mark key as quarantined in a small side-index (Bloom filter or hash set) per node to avoid repeated expensive validations.- Serve a cache-miss to application (fallback to origin) and increment quarantine counter. Optionally return stale data with degraded flag for non-critical reads.- Prevent propagation: reject replication of quarantined entries during syncs; mark source node as potentially tainted if many quarantines.Remediation:- Background repair worker scans quarantined keys: - Try re-fetching authoritative value from origin (DB or service) and rewrite with fresh checksum/version. - If origin unavailable, delete key and allow lazy repopulation. - For policies where repair from replicas is valid, attempt cross-replica reconcile using majority/version.- If quarantine rate spikes → run targeted rolling restart of affected cache node(s) after draining connections; use graceful eviction, rejoin protocols.Operational runbook (incident steps):1. Detect alert → identify scope (keys, clients, nodes).2. Increase logging and enable read-verify mode if not already on.3. Activate quarantine; switch non-critical traffic to read-through mode.4. Start repair workers with rate-limiting to avoid origin overload.5. If repair fails or node shows hardware corruption → cordon and replace node (rolling for cluster).6. Post-incident: analyze root cause (rollback artifact, serialization mismatch), add pre-rollback validation, and add canary toggles for future rollbacks.Performance considerations:- Checksum verification on read is cheap (add sampling mode: verify 1% of reads in steady state; full verify during or after rollout).- Repair workers must be rate-limited and backoff-aware to protect origin.- Use progressive throttling and feature flags to avoid cascading load.Metrics & SLAs:- Track quarantine_count, repair_success_rate, origin_pressure (requests/s), latency P95 changes.- Define thresholds for automated remediation vs human escalation.Prevention & hardening:- Enforce schema/version compatibility during deployments.- Add pre-rollback sanity checks (test rehydrate flows), and use canaries with read-verify enabled before full rollout.This approach isolates corrupted data fast, prevents spread, repairs from authoritative sources, and balances degraded correctness with system performance.
HardTechnical
47 practiced
Explain CRDTs and how they'd be used for conflict resolution in a distributed offline-first application. Compare CRDTs to operational transform (OT) and last-writer-wins. From an SRE perspective, what operational challenges do CRDTs introduce (storage, debugging, observability)?
Sample Answer
CRDTs (Conflict-free Replicated Data Types) are data structures that guarantee eventual consistency without coordination by making operations commutative, associative, and idempotent. Each replica can apply updates locally (offline) and merge divergent states deterministically using a mathematically-defined join (e.g., G-Counter, PN-Counter, LWW-Element-Set, RGA for sequences). In an offline-first app, CRDTs let clients apply edits locally and later sync by exchanging state or deltas; merges never require manual conflict resolution and preserve intent (e.g., concurrent inserts in a CRDT sequence both appear).Comparison:- CRDT vs Operational Transform (OT): OT transforms operations against concurrent edits, often requiring complex transformation functions and central coordination/or consensus for correctness at scale. OT is powerful for low-latency collaborative editing but harder to reason about and implement correctly in peer-to-peer topologies. CRDTs avoid transforms by design, favoring state or operation-based merge rules; they scale better in decentralized settings but can produce different UX semantics (e.g., tombstones, position arbitration).- CRDT vs Last-Writer-Wins (LWW): LWW is simple—choose the update with latest timestamp—but it loses concurrent updates (data loss) and depends on synchronized clocks or causality. CRDTs preserve concurrent effects rather than discarding them.SRE operational challenges and mitigations:- Storage bloat: Many CRDTs keep metadata (timestamps, tombstones, per-replica counters). Mitigation: compaction/GC protocols (causal stable point), periodic tombstone reclamation, delta-based sync to reduce persisted deltas.- Debugging and correctness: Deterministic merges hide bugs until rare concurrency patterns appear. Mitigation: retain immutable logs for replay, deterministic test harnesses, property-based tests (convergence, monotonicity), and canary rollouts.- Observability: Hard to surface “why” a value exists (merged history). Instrumentation should expose provenance: deltas, vector clocks, replica IDs. Provide dashboards for tombstone counts, metadata growth, merge conflicts resolved by CRDT rules, and metrics for sync latency and anti-entropy frequency.- Operational cost: Increased network and CPU for anti-entropy/delta propagation. Mitigation: rate-limit sync, prioritize deltas, use batching, and autoscale background sync workers.- Recovery and upgrades: Schema/CRDT versioning needed. Plan migration strategies (dual-write compatibility, transformation tools) and test cross-version merges.In short: CRDTs are excellent for decentralized offline-first reliability but require deliberate operational tooling—compaction, provenance logging, observability, testing, and migration plans—to keep storage, debugging, and runtime costs manageable.
Unlock Full Question Bank
Get access to hundreds of State Management and Data Flow Architecture interview questions and detailed answers.
