DoorDash Site Reliability Engineer (Mid-Level) Interview Preparation Guide

Direct answer

Blue-green gives you a full, already-validated environment and an instant, simple rollback, at the cost of running two complete copies of your infrastructure during the change; canary gives you gradual, lower-risk validation at the cost of more routing and monitoring complexity, and it is harder to reason about while it is in flight. Pick blue-green when a clean, all-or-nothing cutover is affordable and rollback speed matters most; pick canary when the blast radius of being wrong needs to stay small and you can invest in the routing and monitoring to support gradual exposure.

Structured elaboration

Comparison

Other strategies worth naming

Blue-green and canary are not the only tools:

• Progressive expansion: widen the set of inputs a change applies to in stages rather than widening a traffic percentage, for example rolling a new IAM policy or network ACL out to one account, then one region, then everywhere, independent of any single request's traffic path.
• Feature flags for infra: gate exposure to a new code or config path behind a flag rather than, or in addition to, a routing change, so the change can be turned off instantly without touching load-balancer configuration at all. This is especially useful when the change is a behavior toggle inside already-deployed infrastructure rather than a new fleet.

Why this gets harder for networking or storage than for a stateless app release

Blue-green and canary are easy to describe for a stateless app release because the unit being duplicated (a fleet of identical, disposable instances) and the unit being routed (an HTTP request) line up cleanly. That stops being true for infrastructure-level changes:

• Networking: a routing or firewall change often affects an entire connection or an entire flow, not a single request, so traffic cannot always be shifted 5% at a time the way it can with an HTTP load balancer. A stateful TCP connection, a VPN tunnel, or a peering change uses the old path or the new path for its whole lifetime; the workaround is to canary at the level of an entire subnet, account, or long-lived connection cohort rather than per request, and to accept a coarser blast radius.
• Storage: there is no cheap duplicate of a stateful data store the way there is for a stateless instance. Standing up a second, fully synced database is expensive and introduces replication lag as a new failure mode. The common workaround is to validate the new storage layer with shadow traffic or dual writes before anything reads from it in production, and to use expand-then-contract schema migrations so both the old and new consumers can run against the same underlying store during the transition instead of trying to canary the store itself.

Worked example

A schema change on a Postgres-backed service being rolled out with blue-green: adding a NOT NULL column safely requires an expand-then-contract sequence rather than a single migration, because both blue and green must work against the same schema during cutover.

1. Expand: add the column as nullable, deploy it (both blue and green tolerate a nullable column).
2. Backfill: populate the column for existing rows.
3. Cut over blue-green as normal, both versions still tolerate the nullable column.
4. Contract: once green is fully promoted and blue is decommissioned, add the NOT NULL constraint in a separate migration, since only green's code path needs to rely on it.

This is the same three-step compatibility pattern regardless of whether the deployment strategy is blue-green or canary; what changes is only how long the "both versions must tolerate the old and new shape" window needs to hold.

Trade-offs & pitfalls

• Choosing canary by default because it "sounds safer," without the routing and monitoring maturity to support it, is a common mistake; a canary that cannot actually be measured is worse than a blue-green cutover that can be instantly reversed.
• Forgetting that duplication cost for blue-green is temporal, not permanent, leads teams to avoid it for cost reasons when the actual bill is only for the cutover window.
• Assuming the app-release playbook (weighted HTTP routing) transfers directly to networking or storage changes is the single most common failure mode; check whether the thing being changed is even divisible at the granularity being planned for the canary.

Direct answer

Run one agent pod per node via a DaemonSet, with a small in-memory ring buffer for burst absorption and a bounded on-disk queue for surviving a network partition, both sized from the node's actual telemetry volume rather than a guess. Handle dynamic configuration through a CRD that a controller validates and pushes to agents for hot-reload (no pod restart, no dropped telemetry mid-reload). For multi-tenant isolation, tag every event with the owning tenant at collection time, enforce per-tenant rate limits so one noisy tenant can't starve another's telemetry, and never let two tenants' data share an in-memory buffer.

Structured elaboration

Buffering and backpressure: an in-memory ring buffer absorbs short bursts (for example a log storm from a crash-looping pod); once it's full, the agent spills to a bounded on-disk queue rather than blocking the node's workloads or dropping data outright. A token-bucket rate limiter per tenant caps how fast any one tenant's telemetry can consume shared agent resources, with system/infra telemetry prioritized over tenant traffic when the two compete.

Dynamic configuration via CRDs: a cluster-level controller watches TenantConfig and AgentPolicy custom resources, validates them, and pushes signed configuration to each agent (via a local endpoint on the node, not a full pod restart). Agents apply the new config without dropping in-flight telemetry, which is the actual requirement behind "dynamic configuration" here: a config change that requires a restart isn't dynamic, it's just a faster redeploy.

Multi-tenant isolation:

• Data: every collected event is tagged with tenant identity at the point of collection; per-tenant buffers, not a shared one, so one tenant's backpressure can't block another's.
• Compute/resource: standard Kubernetes resource requests/limits on the DaemonSet pod bound how much CPU/memory the agent itself can consume on a shared node.
• Network: network policies restrict the agent's egress to only the tenant-authorized destinations, and short-lived, tenant-scoped credentials (issued by the controller) authenticate the agent's connection to each tenant's ingest endpoint.

Safe upgrades: rolling DaemonSet update with maxUnavailable bounded per zone, and a preStop hook that drains the on-disk queue (flushes buffered telemetry) before the old pod terminates, so an upgrade doesn't silently lose whatever was queued at that moment.

Failure handling: liveness probes restart a crashed agent, which then replays from its on-disk queue on startup; a corrupted on-disk queue file is rotated into a quarantine location and alerted on rather than silently discarded.

flowchart LR
    N[Node: pods + kubelet] --> A[Agent, one per node]
    A --> RB[In-memory ring buffer]
    RB -- full --> DQ[(On-disk bounded queue)]
    DQ -- network restored --> OUT[Tenant ingest endpoint]
    CRD[TenantConfig / AgentPolicy CRD] --> CTRL[Controller: validate + sign]
    CTRL -- push config, hot-reload --> A
    PS[preStop hook] -- drain before upgrade --> DQ

Worked example

Sizing the on-disk buffer for a network partition. Assume $110$ pods/node (a common Kubernetes per-node pod ceiling), each emitting $5$ log lines/sec averaging $250$ bytes/line:

For a target of surviving a 10-minute network partition to the ingest endpoint without dropping data:

With 4x headroom for longer partitions or bursty traffic during the partition itself:

That's a small, concrete disk request per node, roughly a third of a gigabyte, not the multi-gigabyte allocation teams often over-provision by default, because it's derived from the actual per-node telemetry rate rather than a round number.

In-memory burst buffer. If traffic spikes to 3x baseline for short periods (a log storm from a restart loop) and the agent needs 2 seconds of headroom before spilling to disk:

A sub-megabyte in-memory buffer is enough to absorb a realistic burst at this node's telemetry rate before the disk-spill path takes over, which keeps the agent's steady-state memory footprint small and predictable across a large fleet of nodes.

Trade-offs & pitfalls

Common wrong turns: sizing the on-disk buffer as a fixed number (like "10GB, should be plenty") instead of deriving it from the node's actual telemetry rate and a stated outage-tolerance target, which either wastes disk on quiet nodes or under-provisions busy ones; treating a DaemonSet config change as safe to apply via full pod restart because "it's just a redeploy," which momentarily drops whatever was in the in-memory ring buffer at restart time, exactly the data loss the hot-reload CRD path exists to avoid; and sharing one rate-limit bucket across all tenants on a node instead of per-tenant buckets, which means a single noisy tenant can still starve every other tenant's telemetry even though "backpressure" was implemented.

Opening (2 minutes)

• One-line summary: what happened, current status, and ETA for containment.
• Why this matters: impacted service, customer segments, and business/risk posture.

Five-section briefing (25 minutes total)

1. Timeline & Scope (3 min)

• Short timeline from detection → key events (deployments, config changes, traffic spikes).
• Services/regions/components affected and number of customers/transactions impacted.

2. Current Status & Actions Taken (5 min)

• What we've done (mitigations, rollbacks, circuit breakers, rate-limits).
• Who is on the incident war room and ETA for next checkpoint.
• Live status: degraded/partial outage/resolved.

3. Measured Impact (6 min)

• Customer-facing metrics: %requests failed, p95/p99 latency, successful transactions/sec, sessions dropped.
• Business metrics: active customers affected, estimated revenue per hour impacted, SLA/SLO breach status and error budget burn rate.
• Operational metrics: CPU/memory, thread/connection exhaustion, queue depth, DB slow queries, propagation of errors across services.

4. Probable Cause & Confidence (4 min)

• Hypothesis ranked by confidence (e.g., recent deployment + schema mismatch → cascading retries).
• Evidence: error types from logs/traces, correlation in tracing spans, recent code/config changes, timing with autoscale behavior.

5. Near-term Plan & Asks (7 min)

• Immediate next steps (containment actions for next 30/60/120 minutes): rollback or patch, traffic steering, scaling, disabling feature flags.
• Medium-term: root-cause fix timeline, testing/validation plan, customer remediation plan.
• Risks & contingency: what could go wrong and fallback (full rollback, customer throttling).
• Asks from leadership: approve rollback, prioritize on-call engineers, communications go-ahead, legal/PR involvement.

Telemetry & Dashboards to Surface

• High-level: service health dashboard, SLO burn chart, incident timeline.
• Requests/Errors: error rate (5m/1h), error types and top stack traces, p95/p99 latency, throughput by region/tenant.
• Resource/Infra: host/container CPU, memory, thread pool utilization, DB latency and QPS, queue lengths, autoscaler events.
• Tracing: top failing traces, end-to-end latency waterfall.
• Deployment metadata: commit IDs, build artifacts, config changes, feature flags toggled.
• Customer signals: number of open support tickets, NPS/CSAT alerts, partner escalations.

Recommendations (executive focus)

• Short-term: contain (rollback/feature-flag off) to restore availability within next X minutes; extend mitigation window if necessary.
• Communication: coordinated customer communication template (what we know, who’s impacted, ETA, mitigation steps) and cadence for updates.
• Post-incident: expedited RCA with timeline, remediation plan (code fix, automation, tighter pre-deploy checks), and SLO/alert tuning to detect earlier.
• Investment asks: improve observability (distributed tracing coverage, per-tenant metrics), runbook automation, and chaos testing for this class of failure.

Close (1 minute)

• Reiterate current state, next checkpoint time, and one clear decision requested from the board/CTO now (e.g., approve rollback / approve public statement).

Direct answer: Flapping means the underlying issue is real and ongoing, it just isn't crossing the alert condition cleanly. Fix it with hysteresis (a different threshold to clear an alert than to raise it) and grouping (collapse the repeated fire/clear events into one ongoing incident), not by silencing the alert, silencing would hide the exact persistent problem the question is asking you not to hide.

Structured elaboration

Applying these without hiding the real problem

• Hysteresis and debounce reduce the number of individual fire/clear notifications, but the underlying incident record should stay open and visible the entire time the condition keeps recurring, not just during the moments it's actively firing.
• Grouping should merge notifications, not merge away the evidence: the incident should retain a count of how many times it flapped and the full timestamp history, so on-call can see "this has fired 14 times in the last hour" rather than a single anonymous alert with no memory of the pattern.
• Escalate severity based on persistence, not suppress it: a condition that's been flapping for an hour is a stronger signal that something is wrong, not a weaker one, and the escalation should reflect that (raise severity or broaden the audience) rather than quietly muting it because "it keeps clearing on its own."

Worked example

A disk-usage metric oscillates between 78% and 82% because of periodic log rotation, with a single static threshold at 80%. With no hysteresis, this fires and clears repeatedly as the metric crosses 80% each cycle. Applying hysteresis with a fire threshold of 85% and a clear threshold of 75%:

• The metric's actual range (78% to 82%) never touches either the fire line (85%) or, once fired, the clear line (75%) inconsistently, it stays comfortably inside the dead zone between the two thresholds and doesn't flap.
• If disk usage genuinely climbs past 85% (a real leak, not the periodic rotation pattern), it fires once and stays fired until it drops all the way back under 75%, giving one clean incident instead of a stream of fire/clear notifications for the same underlying trend.

This is a case where the "fix" isn't detecting the flapping better, it's recognizing that a single threshold was never the right model for a metric with a normal 78 to 82% oscillation range, and setting the dead zone wide enough to cover that normal range is what actually solves it.

Trade-offs & pitfalls

• Every one of these techniques trades detection speed for noise reduction. A condition that genuinely needs an immediate page (imminent data loss, a security event) should not be debounced or given wide hysteresis, the noise-reduction techniques above are for conditions where a few minutes of delay is an acceptable cost for not paging someone six times an hour.
• Grouping by a fingerprint that's too broad merges genuinely unrelated issues into one incident (masking that there are actually two separate problems); too narrow, and it fails to group the flaps it was meant to collapse. The fingerprint needs to key on the actual root-cause dimension (host, service, error signature), not just the alert name.
• A hysteresis dead zone set too wide delays real detection meaningfully; too narrow, and it doesn't solve the flapping at all. The right width comes from looking at the metric's actual normal oscillation range (as in the worked example), not from a generic default.

Situation: I’m an SRE on-call and dashboards/logs show inconsistent metrics across regions; it's unclear if regional failover should have occurred.

Triage approach (quick, evidence-first):

• Scope & impact: confirm which regions, services, and customers are affected. Check alerts, pagers, and user-facing error reports.
• Cross-check independent telemetry:
  • Compare metrics from multiple sources: application metrics, infra metrics (host CPU/network), load balancer, CDN, DNS, and uptime probes.
  • Pull raw logs (not aggregated) from affected region and a healthy region for the same time window.
  • Run active synthetic checks (curl/health endpoints) from multiple locations to reproduce.
• Validate control plane and config:
  • Check recent deploys, config changes, infra automation (Terraform/Ansible), and rollout history for that region.
  • Inspect DNS records, load-balancer config, and routing (BGP, Anycast) for recent changes.
• Time & consistency checks:
  • Verify clocks/timezones and log timestamps to rule out delayed/duplicated telemetry.
  • Look for replication lag, message-queue backpressure, or monitoring pipeline errors (prometheus remote write failures, metric exporters down).
    Decision heuristics:
• If user-facing synthetic checks fail and infra logs show errors → likely real outage.
• If only aggregated dashboards differ but raw logs and probes are OK or monitoring exporters are failing → likely telemetry pipeline issue.

Immediate risk mitigation (short-term):

• Open incident channel, assign roles (commander, communications, exec updates).
• If telemetry pipeline is suspected: mark dashboards as degraded, silence noisy alerts tied to that pipeline, and enable fallbacks (alternate exporters, pull metrics directly).
• If region appears down: divert new traffic away (LB/DNS failover), scale healthy regions, enable read-only modes or feature flags to reduce load, and roll back recent changes if correlated.
• Preserve forensic data: snapshot logs, export metrics, avoid auto-scaling churn that would erase evidence.

Communicating uncertainty to stakeholders:

• Initial message (within 5–10 min): concise facts: what we see, potential impact, what we’re doing, and what we don’t know. Example: “We see inconsistent metrics across eu-west and us-east. User-facing health checks from eu-west are failing; raw app logs show 5xx errors in eu-west. We’re investigating whether this is a telemetry pipeline or a real regional outage. Actions: running synthetic checks, checking recent deploys, and preparing traffic failover. Next update in 15 minutes.”
• Use structured updates: timeline, verified facts, actions taken, mitigation plan, ETA for next update. Call out uncertainty explicitly (what is unverified and why).
• Escalation: notify product/ops leads and customer-facing teams if customer impact is likely; provide suggested messaging.
• Post-incident: commit to a postmortem with timeline, root cause, and concrete remediation (improve runbooks, add cross-region synthetic probes, harden failover docs, add telemetry health SLOs).

This approach balances quick protective actions with evidence gathering, prevents premature failovers, preserves data for RCA, and keeps stakeholders informed without overclaiming certainty.

Treat credibility like a limited, rechargeable budget. Rank the initiatives competing for the same stakeholders by expected impact and by how big an ask each one actually requires, spend the smallest ask that moves you closer to the goal first, and save the most expensive kind of ask for the initiative where it buys the most lasting change.

The budget

• Inventory. List every initiative that needs support from the same stakeholders, and rate each on impact (what actually breaks or improves if it happens or doesn't) and on cost (a small favor versus a full re-prioritization).
• Cheap versus expensive asks.

• Sequence across initiatives. Where two initiatives compete for the same stakeholder's goodwill, lead with whichever can be proven cheaply first, bank the resulting trust, and use it to justify the costlier ask on the second initiative rather than spending big on both at once.
• Replenish deliberately. Credibility isn't only spent, it's earned back: deliver on what you said you would, credit others publicly, and be visibly right on the smaller asks. Without replenishment, the sequencing above collapses, because there's nothing left to spend when the second initiative needs it.
• Reserve escalation. Save the most expensive form of capital, going over someone's head or forcing a public reversal, for cases where the cost of not acting clearly exceeds the relationship cost, and even then, spend it having already banked smaller wins with that same stakeholder.

Worked example

A team lead has two initiatives in flight that both need buy-in from the same product stakeholder: a smaller data-quality fix that would quietly improve a metric the stakeholder already cares about, and a larger platform refactor that would require the stakeholder to accept a slower roadmap for a quarter.

Asking for both at once risks getting a lukewarm yes on neither, or a reluctant yes on the small ask while the larger one is flatly refused because there's no banked trust to draw on yet.

The lead sequences deliberately: pitches the data-quality fix first, as a narrowly scoped, low-risk pilot, delivers it, and makes sure the stakeholder gets visible credit for the resulting improvement in their own reporting. Only after that win is delivered and credited does the lead bring the larger refactor ask, explicitly framed against the credibility just built: "the same kind of investment made the number you're now presenting to your leadership possible. This is the same category of work, at a larger scale."

The stakeholder grants the larger ask more readily, because the smaller one was delivered and credited correctly first, rather than both asks being treated as independent, equally costly requests made at the same time.

What a senior person does differently here: never spends the expensive ask first, and always makes sure a stakeholder sees an earlier, smaller win pay off for them personally before asking for something bigger.

Trade-offs and pitfalls

• Spending all your capital early to move fast on every initiative at once usually produces a string of shallow "sure, fine" agreements that dissolve the moment the stakeholder has to defend the decision to someone else.
• Treating every ask as if it costs the same leads to either overspending on trivial requests or badly underestimating the cost of a genuinely expensive one.
• Sequencing only works if you're honest about which initiative actually has the most impact; using credibility to win on the initiative you personally prefer, rather than the one that matters most, burns trust once the mismatch becomes visible.

Situation: I inherited hundreds of ad‑hoc automation scripts scattered across repos with no tests or owners, many used in critical ops.

Plan (step‑by‑step):

1. Triage & inventory (2 weeks)

• Run repo scan (language, entry points, cron/jenkins triggers, cloud infra hooks).
• Create a catalog (owner, usage frequency, last run, inputs/outputs, side effects, criticality, runtime).
• Add lightweight telemetry: wrap scripts with a tracer that emits run metadata to a central store.

2. Prioritize (1 week)

• Rank by impact: production‑facing, manual toil reduction, incident frequency, security/exposure.
• Define acceptance SLOs and risk levels per script.

3. Stabilize in place (ongoing)

• For top tier scripts, add input validation, timeouts, and idempotency guards without changing logic.
• Add logging and feature flags to safely toggle behavior.

4. Refactor into maintainable artifacts (4–8 weeks per batch)

• Create templates: CLI/packaged module, container images, or serverless functions depending on latency and environment.
• Implement idempotent design, parameterization, and secrets handling (use vault).
• Add unit tests for business logic and small integration tests for external calls.

5. CI/CD and testing

• Central CI pipeline that runs lint, unit, integration (mocking external services), and end‑to‑end smoke tests in a sandbox.
• Introduce test fixtures and contract tests for external systems.

6. Ownership & docs

• Assign owners or a rotation squad, add runbooks, playbooks, and SLA commitments.
• Publish the catalog and add automated alerts for failures and drift.

7. Migrate traffic safely

• Use canary deploys and run both old and new implementations in parallel (shadow runs) validating outputs.
• Gradually cut over when metrics (success rate, latency, resource use) remain within thresholds.

8. Retire legacy

• After sustained parity and owner sign‑off, decommission old scripts and update monitoring/dashboard.

Risks and mitigations

• Breaking production: mitigate with feature flags, shadowing, canaries, and throttled rollouts.
• Data/state drift: add checksums, dry‑run modes, and reversible operations when possible.
• Secrets leakage: enforce secret management and rotate creds pre‑cutover.
• Knowledge loss: record runbooks and pair original users during handover.

Rollback strategies

• Quick toggle via feature flag or scheduler pause.
• Revert to legacy by switching cron/trigger endpoints to the old implementation.
• Use immutable deployments so rollback is a single deploy of previous image.
• If state mutated incorrectly, have snapshot/backup and compensating scripts ready; run compensations in dry‑run first.

Success metrics

• % automations tested, owner coverage, MTTR for automation failures, reduction in manual incidents, and error budget impact.

This phased, risk‑aware approach preserves service availability while turning brittle scripts into testable, owned automation.

Direct answer

Node labels, taints/tolerations, affinity, and topology spread each answer a different scheduling question, and this scenario needs all of them combined rather than one chosen over the others. Taints on the volatile pool (spot) provide the exclusion guarantee that keeps critical workloads off it by default. Affinity provides the positive pull that sends the right workload to the right pool, and for GPU (graphics processing unit) nodes that pull has to be paired with the device-plugin's extended resource accounting, not just a label. Topology spread (or pod anti-affinity) spreads critical replicas so a single node or zone loss never removes more than one replica.

Structured elaboration

GPU nodes need more than a label

Labeling a node hardware=gpu is necessary but not sufficient. The scheduler only knows a node has usable GPU capacity once the NVIDIA device plugin (or a vendor equivalent) DaemonSet advertises it as an extended resource in the node's allocatable, for example nvidia.com/gpu: 4. A pod requests it like any other resource:

resources:
  limits:
    nvidia.com/gpu: 1

Without the resource request, a pod with only nodeSelector: {hardware: gpu} can land on a GPU node without ever actually reserving a device, or worse, several pods could expect the same already-claimed device. Pair the label or affinity rule (which steers the pod to the right node family) with the extended-resource request (which makes the scheduler actually reserve a device).

Excluding spot from critical workloads

kubectl taint nodes -l node-lifecycle=spot spot=true:NoSchedule

Only pods that explicitly tolerate it can land there:

tolerations:
- key: "spot"
  operator: "Equal"
  value: "true"
  effect: "NoSchedule"

Critical services carry no such toleration, so the scheduler treats spot nodes as invisible to them by default, with no risk of a nodeSelector typo accidentally placing one there.

Spreading critical replicas

topologySpreadConstraints:
- maxSkew: 1
  topologyKey: topology.kubernetes.io/zone
  whenUnsatisfiable: DoNotSchedule
  labelSelector:
    matchLabels:
      app: critical-service

whenUnsatisfiable: DoNotSchedule makes this a hard constraint; ScheduleAnyway makes it best-effort. For a service that must survive a zone loss, DoNotSchedule is the right choice even though it can leave a pod Pending if the under-represented zone runs out of capacity, which is a signal to add capacity, not to loosen the constraint.

Priority for the pool that can vanish without warning

Spot capacity can disappear with only seconds of notice. Give critical workloads a higher priorityClassName so, if critical and batch pods ever land in the same resource pool during a capacity burst, the scheduler preempts lower-priority batch pods rather than the reverse.

Worked example

A cluster has 2 GPU nodes (4 GPUs each, 8 total), 3 on-demand nodes for critical services, and 4 spot nodes for batch. A training job requests 2 GPUs; with the device plugin installed, allocatable nvidia.com/gpu across the 2 GPU nodes totals 8, and the scheduler places the pod on whichever GPU node currently has 2 or more free, refusing to schedule (Pending, correctly) if all 8 are already claimed elsewhere. A critical service asks for 3 replicas with topologySpreadConstraints maxSkew: 1 across the 3 on-demand zones: with exactly 3 zones and 3 replicas the constraint is satisfiable exactly, one per zone, so losing any single zone loses exactly 1 of 3 replicas, never more.

Trade-offs and pitfalls

• A taint on the spot pool only stops pods without the toleration; it does not, by itself, stop a batch pod from landing on an on-demand node. If batch must also be kept off on-demand capacity, that pool needs its own taint plus a matching toleration on batch pods, not just a one-directional rule.
• Required affinity and required topology spread can leave pods unschedulable when capacity is tight. Monitor scheduling events (Pending plus FailedScheduling) rather than reflexively loosening a hard rule to preferred, which quietly reintroduces the correlated-failure risk the rule existed to prevent.
• GPU device counts are not overcommittable the way CPU or memory can be with limits set above requests. Plan capacity on whole-device granularity.

Direct answer

PACELC, short for "if Partition, Availability vs. Consistency; Else, Latency vs. Consistency", says that CAP's dilemma, choose Consistency or Availability when a network Partition is happening, is only half the story. Even when there is no partition at all, a system still has to choose between Latency and Consistency for every write it replicates, because making a write durable on every replica before acknowledging it takes longer than acknowledging it once it's durable on a single node. PACELC packages this as: if Partition occurs, trade off Availability against Consistency (exactly what CAP already says); Else, meaning no partition, trade off Latency against Consistency.

Restating CAP precisely first

CAP says that during an actual network partition, a distributed system can guarantee only one of Consistency (every read sees the latest completed write) or Availability (every request gets a non-error response) for the nodes on either side of the split, not both. A common misreading treats CAP as "pick two of three, always"; it isn't. CAP's teeth are specifically about behavior during a partition. Most systems are both consistent and available almost all of the time, precisely because a true network partition is a rare event relative to total uptime, not something happening continuously.

flowchart TD
    Start[Write occurs] --> P{Partition active?}
    P -->|Yes| AC[Choose Availability or Consistency]
    P -->|No| LC[Choose Latency or Consistency]

What PACELC adds

PACELC names the trade-off CAP is silent about: during normal operation, with no partition, you still choose between Latency (L) and Consistency (C), because synchronous replication that waits for a majority of replicas costs a round trip before it can acknowledge a write, while asynchronous or single-node-acknowledged replication returns faster but risks a reader seeing stale data, or the acknowledged write being lost outright if that one node fails before it propagates. Systems are commonly labeled by both branches together, for example PA/EL (favor Availability under partition, favor Latency otherwise, the Cassandra/Dynamo-style default) or PC/EC (favor Consistency in both cases, the HBase-style default).

Worked example: a decision with no partition occurring

A write to a piece of user data must be replicated to three nodes: R1 in the local region, and R2, R3 in two remote regions. All three are reachable; no partition is happening anywhere in this example.

• Favor consistency (the "C" side of the Else branch): the write path waits for acknowledgment from a majority, at least two of the three replicas, say R1 and R2, before returning success to the caller. Any subsequent read from a majority quorum is now guaranteed to see this write. Cost: the caller's write waits on the round trip to R2, a remote replica, even though R1, the local one, already has it durably.
• Favor latency (the "L" side of the Else branch): the write path acknowledges as soon as R1 has it durably, and replicates to R2 and R3 asynchronously in the background. Cost: the caller gets a fast, local acknowledgment, but a read served from R2 immediately afterward, before the async replication catches up, will not see the write yet. If R1 crashes before that background replication completes, the already-acknowledged write can be lost entirely, with zero partition ever occurring.

This decision, wait for two of three versus acknowledge on one, is made on every single write regardless of whether any partition is happening, which is exactly the trade-off PACELC's Else branch names and CAP alone has nothing to say about, since CAP only speaks to a system that is not fully connected.

Trade-offs & pitfalls

A common misconception is treating a database's PACELC label as a fixed law of the software rather than a description of its typical default: most systems let you tune the replication wait per request (via quorum size), so "Cassandra is PA/EL" describes its usual configuration, not something it's incapable of changing. It's also easy to blur this Else-branch trade-off with an availability discussion; in the worked example above, no node was ever unreachable, so the trade being made is purely about how long the write path waits before acknowledging, not about surviving an outage, which is a separate concern belonging to the partition branch of the theorem.

Direct answer

A health check that returns success just because the process is running, without checking anything it actually depends on, lies by reporting healthy while the service can't do its job, for example because its database connection pool is exhausted. Liveness checks should answer "should this process be restarted," and readiness checks should answer "can this instance serve traffic right now," and conflating the two causes an orchestrator to make the wrong decision.

Structured elaboration

• Liveness check: answers "should this be restarted?" It should catch deadlocks or unrecoverable internal state, but should not check external dependencies, because a database outage isn't fixed by restarting the app, and a liveness check that fails on a database outage causes the orchestrator to restart every instance simultaneously, making the outage worse.
• Readiness check: answers "should traffic go to this instance right now?" This one should check dependencies, database reachable, cache reachable, disk space available, because an instance that can't reach its database shouldn't receive traffic even though it doesn't need to be restarted.
• The lying health check anti-pattern: an endpoint that just returns success with no logic reports healthy even when every downstream dependency is down, the single most common health-check bug in production systems.
• Defensive design: a readiness check needs its own short timeout and shouldn't itself become a source of load, since orchestrators poll it frequently.

Worked example

A service's health endpoint returns success as long as the HTTP server thread is alive, with no check of its database connection. The database goes down; the app is still "alive" and keeps receiving traffic, returning errors to every real request, while the health check keeps reporting green the entire time, hiding the outage from the orchestrator.

Trade-offs and pitfalls

Making a liveness check too strict, checking dependencies, causes cascading restarts during a dependency outage, turning a partial outage into a full one. Making a readiness check too shallow, like the example above, means the orchestrator keeps sending traffic to instances that can't serve it, arguably worse than doing nothing.

What the interviewer probes next

Whether the candidate distinguishes liveness from readiness at all, since conflating them is extremely common and causes real incidents.