Complex System Troubleshooting and Incident Diagnosis Questions

Tests systems thinking and approaches for diagnosing problems that span multiple components services layers or domains and present multiple related symptoms. Candidates should show how they map interdependencies prioritize which symptoms to address first generate and test hypotheses correlate telemetry across logs metrics and traces and distinguish root causes from secondary effects. The topic includes using instrumentation and monitoring to isolate failures reproducing issues in controlled environments understanding cascading failures and failure modes across networking storage database and application layers and applying mitigations rollbacks or fixes while minimizing user impact. Candidates should also describe incident communication documentation and post incident analysis to prevent recurrence.

HardSystem Design

17 practiced

An emergency schema change is required to remediate a critical production bug, but the target table is huge and heavily used. Describe a safe, low-downtime plan to perform the schema change (online schema change, shadow writes, backfill, dual-write gateways), how to detect migration regressions in real-time, and rollback strategies if the change must be aborted mid-migration.

Sample Answer

Requirements & constraints:- Zero/low downtime, no data loss, ability to abort safely mid-migration, minimal performance impact on a large, hot table. Support strongly consistent reads/writes where possible.

High-level plan (safe, low-downtime):1. Clarify change: adding column type/nullable, index, changing type, constraint. Choose tool: pt-online-schema-change/gh-ost for MySQL; online ALTER variants for Postgres (CONCURRENTLY for indexes) where supported.2. Shadow schema & dual-write gateway: - Add shadow column/table (e.g., col_new) or shadow table (table_new) to receive new-format data. - Deploy a lightweight dual-write gateway in application service or sidecar that writes both old and new representations for every write. Use feature flag to enable dual-write gradually.3. Backfill: - Launch controlled, rate-limited backfill workers that read existing rows and populate shadow column/table. Use partitioned scan, token/range-based workers, and respect system load (CPU, I/O, tail latency). - Ensure idempotency and tombstone handling; track progress checkpoints.4. Validation & canary: - Start with a small canary (single shard or <1% traffic). Compare reads from old vs new paths. - Implement read-compare endpoints or shadow reads for a sample percentage: read both representations and surface diffs. - Use checksums (row-level hash), row-count, cardinality, and histogram comparisons.5. Cutover: - When validation passes for sufficient time, flip reads to use new column/table via config/feature flag. Keep dual-write for a settling period. - Remove dual-write and legacy column after safe retention.

Detect migration regressions in real-time:- Metrics & alerts: - Application errors (5xx), latency P50/P95/P99 for reads/writes, queue/backpressure lengths, DB CPU, IOPS, replication lag. - Migration-specific metrics: backfill throughput, per-worker error rate, rows processed, diff count from read-compare, checksum mismatches.- Real-time validation: - Streaming diffs: emit any row mismatches to a monitoring stream (e.g., Kafka) and dashboard with alerts on >X mismatches per minute or any mismatches for critical keys. - Canary health checks and synthetic transactions that exercise hot paths. - Monitor replication lag (for replicas) and fail if it exceeds thresholds.- Automated gating: - If error rate, latency, replication lag, or diff rate exceeds threshold, auto-pause backfill and switch dual-write to stop writing new column (or revert to single-write) and notify SRE/engineers.

Rollback strategies if aborted mid-migration:1. Pause and assess: - Immediately pause backfill, stop cutover, and keep dual-write. This preserves consistency and allows safe inspection.2. Rollback without data loss (preferred): - Turn off the new-read path (feature flag) so all reads go to old column/table. - Keep writing only to the original schema (disable dual-write) so new writes land on the old format. - Drain and discard or quarantine partially backfilled data (mark shadow rows as stale) for offline investigation.3. Full rollback (if schema applied in-place): - If using online-schema-change tools, use their built-in abort/rollback (gh-ost/pt-osc provide safe abort patterns: stop migration process, ensure triggers are removed, and leave table unchanged). - If an ALTER completed but must be undone (e.g., new NOT NULL/constraint causing failures), perform reverse migration: add reverse shadow, dual-write back to old, backfill missing values, then drop offending constraint in controlled way.4. Emergency fallback: - If application-level incompatibility exists, use request routing to send traffic to previous stable service version that expects old schema, or temporarily disable the feature using a global feature flag.5. Post-rollback cleanup: - Validate integrity of old data via checksums, reconcile gaps, and run postmortem.

Trade-offs and considerations:- Dual-writes add write latency and increased IO — measure and rate-limit.- Long dual-write windows increase complexity but minimize risk.- Strong consistency vs availability: if changing types that break reads, use translation layer in gateway.- Test thoroughly in staging with realistic dataset size and workload, rehearse abort and rollback procedures.

Summary checklist for execution:- Pre-migration: run dry-run, backups, runbook, communication, monitoring hooks, kill-switch feature flags.- Migration: canary → rate-limited backfill → continuous validation → cutover.- Abort: pause → revert feature flags/gateway → stop writes to shadow → ensure no partial reads served from new schema → cleanup/forensic.This approach balances safety and low downtime while providing clear observability and reversible steps.

EasyTechnical

21 practiced

A single Kubernetes pod shows sustained CPU saturation and the service responds slowly. List the exact kubectl and system commands you would run in the first 10 minutes to inspect the pod, its node, and the cluster (including pod logs, events, resource usage, and describe outputs). For each command, explain what it reveals and how you would interpret typical results.

Sample Answer

1) Inspect pod status and location

bash

kubectl get pod my-pod -n svc-namespace -o wide

Reveals pod phase, IP, node. If STATUS!=Running or RestartCount>0, suspect crashes or OOMs; node shows where to continue.

2) Detailed pod info + events

bash

kubectl describe pod my-pod -n svc-namespace
kubectl get events -n svc-namespace --sort-by='.lastTimestamp' | tail -n 20

Describe shows conditions, container state, restart reasons, resource requests/limits, mounting issues. Events show recent scheduling/eviction/throttle messages.

3) Pod logs (container and previous)

bash

kubectl logs my-pod -n svc-namespace -c app
kubectl logs my-pod -n svc-namespace -c app --previous

Shows application errors, GC pauses, long requests. Previous helps if container restarted.

4) Live check of pod CPU/memory

bash

kubectl top pod my-pod -n svc-namespace

If CPU equals or is near limit -> CPU throttling likely; if low but service slow, check latency inside app.

5) Check controller & HPA

bash

kubectl get deployment,statefulset -l app=svc -n svc-namespace -o yaml
kubectl get hpa -n svc-namespace

Ensures replicas and autoscaling config; missing HPA or insufficient replicas explains saturation.

6) Exec into container for process-level view

bash

kubectl exec -it my-pod -n svc-namespace -c app -- /bin/bash
top -b -n1
ps aux --sort=-%cpu | head

Shows which process threads consume CPU, spin loops, or blocked syscalls.

7) Inspect node health (from control plane)

bash

kubectl describe node node-1
kubectl top node node-1

Shows node capacity, allocatable, kubelet conditions, and if other pods are contending for CPU.

8) SSH to node for system-level diagnostics (if allowed)

bash

ssh core@node-1
sudo journalctl -u kubelet -n 200
sudo dmesg | tail -n 50
sudo top -H -p $(pgrep -d, -f kubelet|true)   # threads if investigating kubelet
sudo sar -u 1 5   # or vmstat 1 5

Reveals host CPU saturation, kernel issues, throttling, OOM killer, or disk/IO waits causing apparent CPU issues.

9) Container runtime / cgroup metrics

bash

sudo crictl stats $(kubectl get pod my-pod -n svc-namespace -o jsonpath='{.status.containerStatuses[0].containerID}' | cut -d'/' -f3)

Confirms container-level CPU/cgroup usage and throttling stats.

Interpretation guidance:- Pod CPU ~= limit and throttling: raise limit/requests or scale horizontally; check HPA.- High node CPU with multiple pods high: node capacity issue — drain or scale nodes.- Single process hot loop: fix app code or add concurrency controls.- Low CPU but high latency: investigate blocking IO, GC, thread pool starvation, or networking.- Frequent restarts/OOM: increase memory or fix leak.

Prioritize: pod describe/logs/top -> exec/ps -> node-level top/journalctl -> container runtime stats; take snapshot (metrics/logs) for postmortem.

HardSystem Design

21 practiced

Your service operates in two active regions behind a global load balancer. One region begins degrading (high latency and 50% errors) due to networking issues within that region. Explain, step-by-step, how you'd perform a safe failover to the healthy region while minimizing user impact and data loss. Cover DNS or anycast strategies, failover criteria, session affinity, database replication consistency, and rollback plans.

Sample Answer

Requirements & criteria:- RTO < X minutes, RPO < Y seconds (set beforehand). Failover triggers: sustained error rate > 5xx threshold (e.g., >20% for 2m), p95 latency above SLO for 3 consecutive checks, and region-level network health alerts (BGP/peering errors).- Confirm: control plane/API access to both regions, healthy capacity in target region ≥ projected load.

Step-by-step failover plan:1. Triage & gating- Validate metrics (alerts, traces), check region health dashboards, confirm degradation isn’t transient (retry window). Notify stakeholders and open incident channel.

2. Traffic-shift strategy- If using Anycast/Global IPs: withdraw route advertisement from degraded POPs or adjust BGP prepends to prefer healthy region; this gives near-immediate shift without DNS TTL waits.- If using DNS: reduce TTL proactively (pre-failure) to <60s where possible. On failover update DNS to point to healthy region IPs or endpoints and monitor DNS propagation. Use health-check-aware global DNS (Route53 latency/health-based) to accelerate.

3. Session affinity & in-flight requests- Drain connections at edge in degraded region: mark instances unhealthy in LB, set connection drain timeout to allow in-flight completion. For long-lived sessions, use sticky cookies with session tokens that are portable or centralize session state (Redis, JWT).- If session state is local and cannot be migrated, inform users of potential re-auth and degrade gracefully.

4. Data consistency & DB replication- Ensure target region is a read/write-capable leader or promoted replica. If active-active with conflict resolution, continue. If active-passive: - Verify replication lag (seconds). If lag acceptable for RPO, promote replica using automated, tested playbook (disable writers in degraded region, perform leader election, promote). - If lag is high, consider controlled throttling of traffic or accepting higher RPO with clear stakeholder signoff.- Use synchronous replication for critical metadata where feasible; for high-volume data rely on async with safe cutover rules.

5. Validation- Run smoke tests: healthchecks, end-to-end flows, DB read/writes, consistency checks. Monitor error rates, latency, and business metrics (orders/sec).

6. Rollback plan- Keep rollback steps ready: revert DNS/BGP changes, re-enable region when health restored. If partial data divergence occurred, run reconciliation jobs (idempotent merges, forward-only log application) and pause cross-region writes until consistency guaranteed.- Maintain immutable audit logs to reconcile any user-facing anomalies.

7. Post-incident- Postmortem, update playbooks, automate runbook steps (BGP/DNS/API), add capacity or replication tuning, and lower DNS TTLs proactively for future windows.

Key trade-offs:- Anycast/BGP = fastest but requires network control and careful routing testing. DNS = simpler but slower.- Synchronous replication reduces RPO but adds latency; choose by data criticality.- Prefer stateless services and central session stores to minimize user impact.

MediumTechnical

32 practiced

A configuration change rolled via feature flag caused errors across multiple services. You have a global feature-flag platform. Walk through how you'd detect which flag change caused the problem, how to rollback or patch the flag safely, how to validate the rollback succeeded, and how to prevent the same flag from being re-enabled during the incident.

Sample Answer

Situation: During peak traffic, alerts started firing across multiple services — increased 5xx rate, high error budget burn, and downstream queue backpressure. We knew a global feature-flag platform had a recent rollout window.

Task: Rapidly identify which flag change caused the issue, stop blast radius (rollback/patch), validate recovery, and prevent re-enablement until root cause is fixed.

Action:1. Rapid detection (triage)- Query feature-flag audit logs (who/when/flag/key/value) and filter changes in the incident window.- Correlate with service error timelines: use distributed traces and logs (trace IDs, span start times) to match requests that hit a flag-evaluated path.- Run a binary search across candidate flags: for each flag change timestamp, compare traffic/error cohorts before/after using monitoring (Prometheus, Datadog). A sharp delta after a specific flag change narrows the culprit.- If flags are targeted per region/percent: compare affected vs unaffected cohorts.

2. Rollback / patch safely- If flag platform supports server-side toggle: immediately set the suspicious flag to its previous safe state, via API with an automated script that includes a safety check (confirm current value equals expected bad value to avoid race).- If SDK caching or client-side rollout exists, issue a kill-switch: push a global override that disables evaluation or returns default safe value.- If rollout scope is percentage-based, reduce to 0% using the platform API rather than deleting the flag.- Use automation (CI/CD playbook) to perform the change and create an auditable operation (who/why/timestamp).

3. Validate rollback- Watch key SLO metrics (5xx rate, latency, error rate), traces, and health checks — expect reversion to baseline within a rollback window (usually < N minutes depending on SDK refresh).- Check cohorts that previously errored (region/percentile) for recovery.- Confirm the flag state across all SDKs/nodes: sample nodes, or call the platform API to confirm effective config; for client-side SDKs verify configuration push and any refresh TTL.

4. Prevent re-enablement during incident- Lock the flag: change targeters to a read-only or "frozen" state if supported, or add a conditional guard (e.g., require a PR + code owner approval).- Add a temporary policy in the feature-flag platform or org-wide: all flag changes require two-step approval during the incident window.- Annotate the flag with incident ticket and set an automated block rule in the platform (deny re-rollouts).- Communicate: post in incident channel, tag engineering leads and product owners; update runbook to prevent accidental reinstatement.

Result / learnings:- After reverting the flag and enforcing the lock, 5xx rates returned to baseline within the SDK propagation time. Postmortem added: improve flag auditability, add canary cohorts, enforce progressive rollout guards (circuit-breaker thresholds), and require automated pre-deployment smoke tests for flag logic.

HardTechnical

19 practiced

Your team wants to implement automated remediation for common incidents (restart pod, scale replica, clear cache). Describe the safeguards you would implement to ensure auto-remediation does not worsen incidents (for example, restart loops), how to test and stage auto-remediation safely, and which metrics to monitor to detect when an automated action has unintended side effects.

Sample Answer

Situation: We want safe automated remediation for common incidents (restart pod, scale replica, clear cache) without making things worse (restart loops, cascading failures).

Safeguards (what I’d implement)- Idempotency & safety checks: remediation actions must be safe to run repeatedly and verify pre/post conditions (e.g., only restart if liveness probe failing for N consecutive checks).- Rate-limiting & backoff: enforce per-resource throttles (max 1 restart per pod per X minutes) and exponential backoff after retries.- Circuit breaker / fail-open policy: if a remediation action fails M times in T minutes across the service, disable automatic remediation and escalate.- Leader-election & mutexing: only one controller instance acts on a given resource to avoid concurrent conflicting actions.- Scope & RBAC: restrict what runbooks can do, require higher privileges for destructive actions.- Dry-run & safe-parameter defaults: e.g., scale-up increments small (±1), limit max replicas.- Audit, logging & tagging: every automated action logged with trace-id and correlation to alert/incidents.- Pre-checks: validate cluster capacity, downstream dependencies, and resource quotas before scaling.

Testing & staging- Unit tests for runbook logic; integration tests in CI that simulate probe failures.- Staging/canary rollout: enable auto-remediation in a staging cluster and then a canary subset of services (5–10%) using feature flags.- Chaos & fault-injection: inject failures (kubelet kills, pod OOMs, network partition) to validate behavior and detect loops.- Replay historical incidents in a sandbox to confirm remediation would have improved vs worsened.- Gradual rollout with measurement gates: promote only if key metrics improve or remain neutral.- Runbooks + runbook drills: have human-review mode (auto-suggest, manual confirm) for first N occurrences.

Metrics and signals to monitor for unintended side effects- Remediation action rate: number of automated restarts/scales per minute per service.- CrashLoopBackOff / restart count trend: rising restarts after remediation indicates loop.- Error rate / 5xxs and SLOs: increased application errors after action.- Latency and tail-latency (p95/p99): spikes after scaling or cache clears.- Resource utilization: CPU/memory/IO before and after scale to detect overload.- Upstream/downstream error propagation: errors in dependent services.- Incident recurrence rate and mean time to recovery: if MTTx worsens, disable automation.- Alert noise and on-call escalations: increased pages indicates harm.- Change-rate correlation: correlate remediation timestamps with metric deviations.

Example policy (conceptual)- Only restart if liveness probe failed 3x in 5m AND pod restarts < 3 in last 30m; max 1 restart/pod per 30m; if 5 pods in same deployment were auto-restarted in 10m → open circuit and notify on-call.

This combination of conservative, auditable safeguards + staged testing and focused monitoring minimizes risk while enabling safe self-healing.

Unlock Full Question Bank

Get access to hundreds of Complex System Troubleshooting and Incident Diagnosis interview questions and detailed answers.

Join thousands of developers preparing for their dream job.