Covers the spectrum of data consistency models used in distributed systems and the operational practices for detecting and recovering from inconsistency. Topics include strong consistency guarantees provided by atomicity, consistency, isolation, and durability style transactions and synchronous replication, and weaker models such as eventual consistency and causal consistency along with their read guarantees like read your writes and monotonic reads. Explain the trade offs between consistency, availability, and latency and how those trade offs influence architecture decisions, user experience, and cost. Discuss replication strategies including synchronous replication, asynchronous replication, and read replicas, and how replication modes affect staleness and failure behavior. Include coordination and consensus mechanisms for achieving stronger guarantees, for example leader based replication and consensus protocols, and distributed transaction approaches such as two phase commit. Cover operational concerns: how consistency choices change testing, deployment, monitoring, and incident response. Describe detection and recovery techniques for inconsistency such as validation checks, reconciliation and anti entropy processes, tombstones and conflict resolution strategies, use of vector clocks or conflict free replicated data types to resolve concurrent updates, point in time recovery and backups, and procedures for partial repairs, rollbacks, and replays. At senior levels also address how consistency decisions shape runbooks, alerting, and post incident analysis.
HardTechnical
81 practiced
CRDTs provide strong convergence guarantees, but they are not a silver bullet. Explain limitations of CRDTs: operations that cannot be made commutative and application semantics that CRDTs cannot express. Provide examples of data types or business invariants where consensus or linearizability is required instead.
Sample Answer
CRDTs guarantee eventual convergence without coordination by making operations commutative, associative, and idempotent. That buys availability and partition tolerance, but has important limits.Fundamental limitations- Non-commutative operations: Any operation whose outcome depends on real-time ordering (e.g., bank transfer debits that must not allow negative balance) cannot be safely reduced to commutative merges without weakening semantics.- Global invariants: Constraints spanning multiple keys or objects (e.g., total capacity, quotas, serial numbers, unique usernames) often require coordination to prevent violations.- Real-world side effects: External actions (sending an email, billing) must not be executed multiple times; CRDTs alone don’t provide exactly-once semantics.Concrete examples where consensus/linearizability is required- Financial ledgers: Strong serializability for balances and transfers to enforce no-double-spend.- Leader election and configuration changes: Cluster membership or primary-failover decisions need consensus (Paxos/Raft) to avoid split-brain.- Unique ID allocation: Monotonic strictly increasing counters or globally unique sequence numbers require a linearizable counter or coordination service.- Quota enforcement: Enforcing "sum of allocations ≤ capacity" across replicas needs atomic reservation (compare-and-set or transactions).SRE implications and guidance- Use CRDTs for high-availability, commutative use cases (presence, counters, collaborative text with relaxed semantics).- For strong invariants, combine CRDTs with coordinated components: run a consensus-backed service for reservations/financial ops; or perform a two-step protocol (local CRDT update + centralized validation) with compensating actions.- Instrument and monitor divergence windows, reconciliation rates, and compensating rollback frequency to maintain reliability.
HardTechnical
103 practiced
Senior leadership: How should consistency choices (strong vs eventual, sync vs async replication, CRDTs vs consensus) shape runbooks, alerting, SLOs, post-incident analysis, and cross-team responsibilities? Provide concrete examples of runbook entries and escalation paths that differ by consistency model.
Sample Answer
High-level principle: the chosen consistency model directly changes what can fail, how fast it must be repaired, who owns correctness, and what customers expect — so runbooks, alerts, SLOs, postmortems, and team boundaries must reflect those differences.Runbook & alerting patterns (concrete examples)1) Strong consistency (consensus, sync replication)- Runbook entry: "Primary election failure / stalled leader" - Symptoms: >30s quorum loss, write latency >500ms, write errors return 503 - Immediate steps: 1. Verify quorum: runctl cluster status 2. If quorum < majority, do NOT accept writes. Notify On‑Call DB SRE (pager + channel) and Product Owner (for possible read‑only window) 3. Attempt graceful leader re-election; if fails in 2min, trigger automated controlled failover playbook - Escalation: On‑call DB SRE -> Senior SRE (15m) -> Engineering Manager (30m) -> Exec if >2hr- SLOs: tight write latency and correctness SLOs (e.g., 99.9% writes acknowledged within 300ms); error budget tied to automated rollback.- Post‑incident focus: root cause of quorum split, test of failover deterministic behavior, and improvements to election timeouts.2) Eventual consistency (async replication, CRDTs)- Runbook entry: "Replication lag / data divergence" - Symptoms: tail latency spikes on background replication > X minutes; client reports stale reads - Immediate steps: 1. Identify affected region/replica; enable read routing to up‑to‑date replica if available 2. If CRDT merge conflicts detected, run reconciliation script (idempotent merge) and alert Data Integrity team - Escalation: App SRE -> Data Integrity SME (30m) -> Product PO (1hr) if user‑facing inconsistency >N users- SLOs: surface consistency windows (e.g., Prob(stale read >5s) <0.1%); emphasize availability/latency SLOs over absolute immediate correctness.- Post‑incident focus: quantify consistency window, improve monitoring of divergence, add compensation logic or read-after-write hints.3) Hybrid (read-optimized eventual + critical strong paths)- Runbook distinguishes paths: "Critical canonical writes" vs "best-effort sync" - Escalation routes include both DB SRE and owning service team; require joint RCA and contract on SLAs.Cross-team responsibilities- Ownership matrix: system owning data model = correctness owner (decides compensation, CRDT merges), infra SRE = operational owner (replication, leader election), service SRE = customer-facing remediation (traffic routing, feature flags).- Runbooks must include contact roles, authority to freeze writes, and decision thresholds for rollback vs continued degraded operation.Post‑incident analysis & SLOs- For strong consistency failures, RCAs must include timeline to consistency restoration and test coverage for leader election; SLO remediation often requires rollback or config patch.- For eventual/CRDT incidents, RCAs must quantify user impact by measuring staleness and assess whether business invariants were violated; focus on improving observability (per‑key divergence metrics) and application-level compensation.Example alert mappings (summary)- Leader/quorum lost -> P0 to DB SRE (strong)- Replication lag above window -> P1 to SRE + Data Integrity (eventual)- Merge conflicts beyond automatic resolver -> P1 with Product escalation (CRDT)Why this matters: aligning runbooks, alerts, SLOs, and escalation to the consistency contract prevents incorrect operational decisions (e.g., forcing writes during split‑brain) and clarifies who can take what action under what risk profile.
HardSystem Design
93 practiced
Design a rollback and replay mechanism using a write-ahead log (WAL) that allows partial repair across replicas after a corruption event. Explain how to ensure idempotency, handle gaps in the WAL, determine canonical ordering, and coordinate replay without further corrupting other replicas.
Sample Answer
Requirements:- Allow partial repair: replay WAL entries to a subset of replicas after corruption without harming others.- Ensure idempotent application, detect/handle gaps, pick a canonical ordering, and coordinate safe replay.High-level design:1. WAL format: append-only, entry = {seq, shard_id, op_id (UUID), timestamp, payload, checksum, causal_tokens} - seq: global increasing log position (monotonic per-cluster or per-shard with cluster-wide tie-breaker) - op_id: unique operation id for idempotency - causal_tokens: vector clock or parent seq pointers for causal ordering2. Idempotency: - Every replica keeps an applied-op index: map(op_id -> last_applied_seq) and per-shard last_seq_applied. - Apply is guarded: if op_id already applied at >= seq, skip; if applied < seq, ensure operation is safe to reapply (use commutative/mergeable ops or include compensating logic). - Use operation tombstones/undo metadata for reversible operations.3. Handling gaps: - Replay uses sequence ranges. If a gap detected (missing seq), mark range as "pending" and attempt to fetch missing entries from other replicas or object store snapshot. - If gaps persist beyond threshold, pause dependent replays and surface to operator; allow partial replays only if operations are commutative and independent.4. Canonical ordering: - Use seq as primary canonical order. For concurrent writes generating same causal level, tie-break by (origin_node_id, op_id). - Maintain causality checks using causal_tokens; reject replays that would violate causal dependencies.5. Coordination and safety: - Recovery coordinator service orchestrates rollback/replay: - Phase 1: Quiesce writes for affected shard (lease or global coordination) to avoid new conflicting mutations. - Phase 2: Collect WAL fragments, compute missing ranges, verify checksums. - Phase 3: Validate replay plan against each target replica’s applied-op index and dependency graph. - Phase 4: Execute replay under transactional apply with two-phase commit (prepare/apply) per replica or using per-replica staging area. If a replica fails prepare, abort and roll back staging. - Use fencing tokens (epoch/leader lease) to prevent other nodes from accepting concurrent writes during repair.6. Observability and safety nets: - Strong monitoring: metrics for gaps, checksum mismatches, replay progress. - Automated dry-run: simulate replay on snapshot to detect potential invariant violations. - Rate-limit and canary replays: apply to one replica, verify health/smoke tests, then roll out.Trade-offs:- Strong quiesce and two-phase commit increases availability impact during repair but prevents further corruption.- Allowing partial repairs with commutative ops improves availability but requires more careful operation design.This design ensures deterministic, idempotent replays, explicit gap handling, canonical ordering, and coordinated application that avoids propagating corruption.
EasyTechnical
71 practiced
Define the read guarantees 'read-your-writes' and 'monotonic reads' in distributed storage systems. Provide a short example for each showing client actions and server responses, and explain how an SRE might instrument or enforce these guarantees at the client or middleware layer.
Sample Answer
Read-your-writes (RYW): A client that successfully writes a value is guaranteed to see that write on subsequent reads in the same session. Example:- Client: PUT /kv/foo = "A" → Server (leader) responds 201, version=42- Client: GET /kv/foo → If RYW holds, server returns "A", version=42 (not an older value)Monotonic reads (MR): Once a client observes a value at a certain version, all later reads in the same session will return that version or a later one (never move backward).Example:- Client: GET /kv/bar → Server returns "X", version=10- Client: GET /kv/bar → MR guarantees response version >=10 (e.g., "Y", version=12), never version 9How an SRE might instrument or enforce these at client/middleware layer:- Enforce via session tokens/metadata: have clients attach a session ID + last-seen version/timestamp. Middleware routes reads to replicas that have applied >= that version (sticky leader routing or replica selection by version).- Client-side strategies: read-after-write by routing writes and subsequent reads to the leader, or by including the write’s version/token and retrying until a replica serves that version.- Causal/version tracking: use monotonic counters or vector clocks per session; middleware rejects/redirects reads to up-to-date replicas when the requested version > replica’s current.- Observability/alerts: emit metrics for "session violations" (read returned older version than session last-seen), latency of reaching consistency (time between write and first readable), and replica lag. Trace session tokens through distributed traces to debug where guarantees break.- Automation: alert if session-violation rate exceeds SLO, auto-failover or re-route sessions away from lagging replicas, and add read-repair background jobs to reduce lag.These measures let SREs both enforce guarantees (routing, version checks) and observe when guarantees are violated (metrics, traces) so they can act (alerts, failover, capacity adjustments).
MediumTechnical
72 practiced
You receive reports of stale reads for a small percentage of users. Outline a step-by-step troubleshooting plan to identify root cause: what logs, traces, metrics, and checks would you run; how to reproduce; and how to validate whether the issue stems from replication lag, caching, DNS, or client behavior.
Sample Answer
Situation overview: a small percentage of users report stale reads. My goal: methodically determine where stale data is introduced (replication, cache, DNS/load-balancing, or client).Step-by-step plan:1. Triage & scope- Get timestamps, user IDs, request IDs, affected endpoints, and sample client environments.- Check incident frequency/percentile and any recent deploys or config changes.2. Collect logs & traces- Pull application logs for affected request IDs (app servers, API gateway) and backend DB query logs.- Pull distributed traces (Jaeger/Zipkin/NewRelic): compare client request -> cache -> DB spans.- Check CDN/edge logs if reads go through CDN.3. Metrics to inspect- DB replica lag (seconds) across replicas and regions.- Cache hit/miss rates, eviction rates, TTLs.- LB/DNS logs: request routing, any sudden changes in weighted routing.- Error rates and p50/p95 latencies for reads.4. Reproduce- Use affected user headers/geo/IP to simulate same route. Reproduce with: same user ID, same auth token, and same read path.- Run controlled reads in sequence: bypass cache (cache-control/no-cache) vs normal; force read from primary vs replica.5. Targeted checks & hypotheses- Replication lag: query replica status (e.g., mysql: SHOW SLAVE STATUS; Postgres: pg_stat_replication). Correlate lag timestamps with stale read times.- Caching: inspect cache keys/versions, TTL, stale-while-revalidate behavior, client cache-control. Check if cache key includes user-specific tokens.- DNS/Load-balancer: verify routing to outdated region or old deployment (compare service versions). Check DNS TTL anomalies and cached entries on client/edge.- Client: review client SDK versions, offline mode, local storage, clock skew, or retries returning old cached UI state.6. Validation- If replication: induce a write and verify read consistency across replicas; measure lag fall-back.- If cache: invalidate key and confirm immediate fresh read; turn off cache for test users.- If DNS/LB: force DNS refresh or route one test user to a different backend and observe behavior.- If client: reproduce by simulating client network and SDK; inspect request/response headers (Cache-Control, ETag).7. Remediation & monitoring- Short-term: invalidate caches, promote/repair replication, roll back faulty deploys, or instruct clients to refresh.- Long-term: add observability (per-request read-source tag), alert on replica lag and cache anomaly, add read-after-write guarantees for affected flows, add client telemetry.Key logs/commands examples:- DB: SHOW SLAVE STATUS; SELECT now() - pg_last_xact_replay_timestamp();- Cache: redis MONITOR / INFO stats- Tracing: fetch span by trace-id in Jaeger- LB/DNS: dig +trace, check LB access logsThis approach isolates components, reproduces the issue deterministically, and validates root cause with targeted tests.
Unlock Full Question Bank
Get access to hundreds of Data Consistency and Recovery interview questions and detailed answers.
