Complex System Troubleshooting and Incident Diagnosis Questions

Tests systems thinking and approaches for diagnosing problems that span multiple components services layers or domains and present multiple related symptoms. Candidates should show how they map interdependencies prioritize which symptoms to address first generate and test hypotheses correlate telemetry across logs metrics and traces and distinguish root causes from secondary effects. The topic includes using instrumentation and monitoring to isolate failures reproducing issues in controlled environments understanding cascading failures and failure modes across networking storage database and application layers and applying mitigations rollbacks or fixes while minimizing user impact. Candidates should also describe incident communication documentation and post incident analysis to prevent recurrence.

EasyTechnical

20 practiced

A production service is restarting repeatedly (crash loop) and the monitoring shows continuous restarts. Describe how you would stabilize the environment quickly to reduce user impact, capture crash evidence safely, inspect logs and core dumps, determine whether a configuration or code change is responsible, and plan the safe re-introduction of the service.

Sample Answer

**Stabilize quickly (reduce user impact)**

- Immediately prevent further restarts: stop the service or scale replicas to 0 / set restart policy to "no".- If behind a load balancer, remove instance from rotation or enable maintenance mode to avoid user-facing errors.- If clustered, shift traffic to healthy nodes.

**Capture crash evidence safely**

- Preserve logs and metadata before restarting: copy /var/log, systemd journal, container logs.- Disable auto-cleanup of core files and set core pattern if needed.- Example commands:

bash

sudo systemctl stop myservice
sudo journalctl -u myservice --no-pager > /tmp/myservice.journal.log
sudo cp /var/log/myservice/* /tmp/
sudo sysctl -w kernel.core_pattern=/tmp/core-%e-%p-%t

**Inspect logs and core dumps**

- Search logs for stack traces, OOM, segfaults, or config errors (grep, awk).- Use gdb to get backtrace from core:

bash

gdb /usr/bin/myservice /tmp/core-myservice-1234 -batch -ex "bt full" > /tmp/core.bt.txt

- For containers, use docker/kubectl logs and describe pod to see events.

**Determine config vs code**

- Compare recent config changes (git, config management, timestamps) and deployment history.- Reproduce crash with previous known-good config or binary in staging.- If stack trace points to code paths or symbol names, lean code; if errors are missing keys, permissions, or env vars, lean config.

**Plan safe re-introduction**

- Roll back to last known-good version or config.- Reintroduce via canary: 1 instance, health probes, tight alerts.- Monitor metrics/logs and ramp gradually.- Post-mortem: root cause, corrective action (fix code/config), add health checks, alerting and automated rollback.

EasyTechnical

25 practiced

A critical production server reports 100% disk usage on the root partition and multiple services are failing. Walk through the prioritized steps you would take in the next 10 minutes to reduce user impact without risking data loss: how to find and safely clear space, how to determine what can be removed (logs, caches, orphaned files), commands you would run, and how you verify services will restart and remain stable.

Sample Answer

**Immediate priorities (first 1–2 minutes)**- Stop further user impact: avoid reboot unless unavoidable.- Identify full mount and large consumers.

Commands:

bash

df -h /
du -xh / | sort -hr | head -n 20
lsof +L1        # open deleted files still holding space

**Safe quick frees (2–6 minutes)**- If processes hold deleted files, restart the process or truncate the file:

bash

# If safe to restart service
systemctl restart <service>
# Or truncate large log without deleting
: > /var/log/large-log.log
# For deleted-but-open files, find PID and restart
lsof +L1 | awk '{print $2}' | tail -n +2 | xargs -r -n1 -I{} kill -HUP {}

- Remove safe caches:

bash

apt-get clean || yum clean all
docker system prune -af   # only if Docker used and safe
rm -f /var/cache/*/*     # inspect before rm

- Move noncritical large files to another volume if available:

bash

mv /path/to/largefile /mnt/otherdisk/

**Determine removable items (6–8 minutes)**- Inspect logs, old kernels, core dumps, tmp, package caches:

bash

ls -lh /var/log
rpm -qa kernel\* | sort
find /var/tmp -type f -mtime +7 -print

- Never delete unknown files; prefer archiving/moving.

**Verify services & stability (8–10 minutes)**- Restart critical services and check status:

bash

systemctl restart <service>
systemctl is-active --quiet <service> && echo ok || journalctl -u <service> -n 50

- Monitor disk and service health:

bash

df -h /
watch -n 5 'df -h /; systemctl --type=service --state=failed'

Notes/forensics: record commands run, preserve removed files list, escalate if space not recoverable (attach new disk or restore from backups).

EasyTechnical

24 practiced

Explain what a correlation ID is, where to inject it in an HTTP request path and asynchronous job boundaries, how it should propagate through logs, metrics, and traces across multiple services, and common pitfalls (e.g., lost context, third-party services not forwarding IDs). Provide best practices for header names, length, and privacy considerations.

HardSystem Design

19 practiced

Design an automated rollback plan for a complex release that includes application code, database schema migrations, and feature flags. Explain how you would instrument and validate the rollback in production to ensure minimal user impact: include canary rollbacks, database migration reversibility or compensating migrations, safety checks, runbook steps, and how to test the rollback plan in staging and canary environments before using it in production.

Sample Answer

**Overview**I would build an automated, safe rollback pipeline that treats app code, DB schema, and feature flags as coordinated artifacts, with canary-first rollback, automated safety checks, and clear runbook automation.

**Components**- Release metadata store: records deploy ID, schema migration ID, FF state, canary targets.- Orchestrator (CI/CD): can trigger forward or rollback steps; idempotent.- Migration library: supports reversible SQL + explicit compensating migrations.- Feature-flag service with API to atomically toggle flags by release ID.- Observability: dashboards, SLO/alerts, request tracing, per-canary metrics.

**Rollback strategy**1. Canary rollback first: revert app pods in canary pool to previous image; toggle FFs off for canary.2. Validate stability (error rate, latency, business metrics) for X minutes.3. If green, progressively roll back remaining pools (staged percentages).4. DB: if migration is reversible, run rollback migration against canary DB replica; validate data integrity. If not reversible, run compensating migration that preserves data and restores behavior (e.g., backfill columns, map values).5. Finalize by restoring prior schema and re-enabling FFs globally.

**Safety checks & gating**- Pre-rollback checks: backup verification, replication lag < threshold, schema compatibility tests.- Kill-switch: immediate stop of deployment traffic via LB or FF if thresholds breached.- Dry-run mode that simulates migrations and flag changes.

**Runbook (automated + manual steps)**- Automated: trigger canary rollback, run DB compensating scripts against read-replica, run smoke tests, promote if OK.- Manual escalation: if data loss risk, pause and require engineering approval; checklist: snapshot DB, export failing datasets, contact stakeholders.

**Testing**- Staging: full replay of production data subset; run forward + rollback including migrations and FF toggles.- Canary environment: periodically run rollback drills against a small pool with production-like load and chaos injection.- Automated tests: migration forward/reverse unit tests, schema diff checks, and end-to-end smoke tests in CI.

**Validation in production**- Instrument rollbacks with tracing and per-release tags; compare canary vs baseline metrics.- Run automated integrity checks (row counts, checksum queries) post-DB rollback.- Postmortem metrics: time-to-detect, time-to-rollback, user-impact delta.

This approach minimizes user impact by failing fast at the canary stage, ensuring DB reversibility or safe compensations, automating repetitive steps, and enforcing human checkpoints when irreversible data changes are involved.

MediumTechnical

23 practiced

When an incident shows many simultaneous symptoms, how do you methodically distinguish a root cause from secondary effects? Provide a concrete example incident, the hypotheses you would generate, the telemetry you would correlate, and the validation steps you would run to confirm the root cause versus a symptom.

Sample Answer

**Approach (methodical steps)**

1. Establish scope and timeline (what failed, when, blast radius).2. Generate prioritized hypotheses (root-cause candidates vs. downstream effects).3. Correlate telemetry across layers (infra, network, app).4. Validate by isolation, reproduction, and controlled rollback.

**Concrete example incident**Situation: Multiple web servers return 503, CPU spikes on DB replicas, increased error rate in app logs, and recent config push 10 minutes before alerts.

**Hypotheses (ordered)**- H1: Recent config change broke load-balancer routing → downstream 503s (primary).- H2: DB replica overload causes app timeouts → 503s (secondary).- H3: Network partition between LB and app instances → simultaneous symptoms.

**Telemetry to correlate**- Deployment/change audit (who/what/when)- Load balancer metrics and config (routing tables, health checks)- Server sysmetrics (CPU, memory, disk I/O) around timestamps- Application logs (timeouts, stack traces) with request IDs- DB metrics (connections, queries/sec, replication lag)- Network flows (tcpdump, netflow, traceroutes)

**Validation steps**- Roll back the config push in a staging-safe way or revert on one LB node (if possible) to see if errors stop.- Pinpoint time-aligned events: if errors start exactly at config push, H1 gains weight.- Isolate DB: temporarily direct a failing app instance to a healthy DB replica; if 503s persist, DB is less likely root.- Reproduce under controlled load with original config vs. reverted config.- Enable increased logging and health-check verbose mode to capture request path.

**Decision criteria**- If reverting the change restores normal behavior immediately → change is root cause.- If DB metrics show saturation before config change and reverting does not fix → DB is root cause.- If network traces show packet loss between LB and app, and symptoms follow that, treat network as root cause.

I would document findings, corrective action, and a prevention plan (review CI/CD guardrails, improve health checks, add rollback playbook).

Unlock Full Question Bank

Get access to hundreds of Complex System Troubleshooting and Incident Diagnosis interview questions and detailed answers.

Join thousands of developers preparing for their dream job.