Debugging and Recovery Under Pressure Questions

Covers systematic approaches to finding and fixing bugs during time pressured situations such as interviews, plus techniques for verifying correctness and recovering gracefully when an initial approach fails. Topics include reproducing the failure, isolating the minimal failing case, stepping through logic mentally or with print statements, and using binary search or divide and conquer to narrow the fault. Emphasize careful assumption checking, invariant validation, and common error classes such as off by one, null or boundary conditions, integer overflow, and index errors. Verification practices include creating and running representative test cases: normal inputs, edge cases, empty and single element inputs, duplicates, boundary values, large inputs, and randomized or stress tests when feasible. Time management and recovery strategies are covered: prioritize the smallest fix that restores correctness, preserve working state, revert to a simpler correct solution if necessary, communicate reasoning aloud, avoid blind or random edits, and demonstrate calm, structured troubleshooting rather than panic. The goal is to show rigorous debugging methodology, build trust in the final solution through targeted verification, and display resilience and recovery strategy under interview pressure.

EasyTechnical

63 practiced

You are on-call and receive an alert that a nightly ETL job failed 5 minutes ago. You have 30 minutes to restore data availability. Describe your step-by-step debugging approach to reproduce the failure, isolate the root cause, and apply the smallest safe fix. Explain how you preserve working state, validate the fix, and communicate progress to stakeholders while working under pressure.

Sample Answer

Situation: I get an alert that last night’s ETL failed 5 minutes ago and I have 30 minutes to restore data availability for downstream consumers.

Step-by-step approach:1. Triage & communicate (0–3 min)- Acknowledge the alert in the on-call channel and set expectation: “Investigating; will update every 10 minutes.”- Check severity: which table/dataset and SLAs affected; identify immediate consumers.

2. Reproduce quickly (3–10 min)- Inspect job run logs (Airflow/GCP Dataflow/Spark driver logs) and timestamped stderr/stdout.- Re-run the failing task locally or in a dev sandbox with the same input batch ID (use same config/env) to reproduce the error deterministically.- If logs are large, grep for ERROR/Exception and the batch ID.

3. Isolate root cause (10–18 min)- Determine failure class: upstream data change, schema drift, resource (OOM), network/storage permissions, or code bug.- Validate hypotheses: check source file counts and schema in S3/GCS, DB connectivity, recent deploys, and cluster metrics (CPU/memory).- Example: if Spark executor OOM in transform_step -> likely data skew or memory config changed.

4. Smallest safe fix (18–24 min)- Prefer mitigations over risky code changes: backfill missing data with previous successful output if available; restart job with increased resources, smaller parallelism, or skip a noncritical step.- If schema drift, add tolerant parsing or fallback path for the problematic field and re-run single failed batch.- Always run the fix on a single batch or in a canary before broader rollout.

5. Preserve working state- Snapshot logs, note failed batch IDs, and tag current cluster state. If modifying configs, do so in ephemeral override (not commit) and keep originals to revert.- Create checkpoints or mark processed offsets to avoid double-processing.

6. Validate fix (24–28 min)- Re-run the failed task/batch and verify job succeeds end-to-end.- Run quick data quality checks: row counts, key distributions, schema conformance, and sample records into downstream table.- Notify downstream consumer that data looks good and provide TTL for final verification.

7. Communicate closure & post-incident (28–30+)- Update channel with root cause, fix applied, verification results, and next steps (hotfix commit, postmortem).- If needed, open a follow-up ticket for permanent fix, monitoring rule improvements, or regression tests.- After incident, produce a short postmortem with timelines, impact, and preventive actions.

Why this works:- Prioritizes safety and minimal blast radius, preserves ability to rollback, validates with data checks, and keeps stakeholders informed under SLA pressure.

HardSystem Design

93 practiced

A distributed job has severe data skew: a small subset of keys slows whole job and causes executor OOM. Describe advanced debugging and mitigation steps: how to detect skew from metrics, split heavy keys, use map-side pre-aggregation, salting strategies, and how to verify the fix with a minimal repro.

Sample Answer

Requirements:- Detect and fix severe data skew that causes long tasks and executor OOM in a Spark job while preserving correctness and performance.- Provide debugging, mitigation (split heavy keys, map-side pre-aggregation, salting), and a minimal repro/verification approach.

High-level approach:1) Detect skew from metrics- Inspect Spark UI: task time/GC/Shuffle read/write per task, executor logs. Skew shows small number of tasks with very high duration/shuffle/read bytes or repeated OOMs.- Use job metrics: per-partition sizes from mapOutputMetrics, stage/task metrics, and application logs. Export to monitoring (Prometheus/Grafana) and alert on 95/99th percentile task time, max shuffle bytes per task.- Run a quick job to compute partition size histogram:

python

# pyspark: size per key partition
rdd = sc.textFile(...)
pairs = rdd.map(lambda x: (key_fn(x), 1))
partition_sizes = pairs.partitionBy(100).mapPartitions(lambda it: [sum(1 for _ in it)]).collect()

2) Root-cause analysis- Find heavy keys by counting keys and top-N by size (sample if data huge) using approximate algorithms (HyperLogLog, Count-Min Sketch) or a reduceByKey(count). Identify whether skew arises from logical key hotspot or bad joins.

3) Mitigations- Map-side pre-aggregation / combiner - aggregateByKey / map-side combine to reduce shuffle volume before partitioning. - Example: use combineByKey or reduceByKey with numPartitions tuned.- Split heavy keys (key splitting) - For a small set of heavy keys, expand them into sub-keys before shuffle: (k -> (k, salt)) where salt in 0..N-1 chosen by hashing or round-robin, aggregate, then merge subkeys back. - Use two-stage approach: 1. map: (k -> (k, salt)) and aggregate 2. reduce: remove salt and final aggregate- Salting strategies - Deterministic salting for joins: add salt to both sides where heavy-key counts known; otherwise asymmetric salting (salt only larger side) then replicate small side appropriately. - Choose salt factor based on heavy-key size and target partition size: salt = ceil(size_of_key / target_partition_size)- Adaptive partitioning - Use custom Partitioner that routes heavy keys across many partitions; or use RangePartitioner after sampling.- Memory tuning and spill controls - Increase shuffle.memoryFraction (Spark config), enable external shuffle service, tune spark.memory.offHeap if offheap used, increase spark.executor.memory temporarily. - Prefer algorithmic fixes over memory increases.

4) Verification with minimal repro- Create a small dataset that reproduces skew: include heavy key with scaled-up count and run the pipeline locally or on a small cluster with same partitioning logic.- Example repro:

python

# create RDD with one heavy key
heavy = sc.parallelize([("hot", i) for i in range(1000000)])
others = sc.parallelize([(f"k{i}", i) for i in range(1000)])
rdd = heavy.union(others)
# run join/aggregation steps to trigger skew

- Apply fix (e.g., salting) and compare: - Task duration distribution (max vs median) - Shuffle bytes per task - Peak executor memory / GC pauses - Correctness of aggregate results- Automate tests in CI with scaled-down but proportionally skewed data.

Trade-offs:- Salting increases shuffle/metadata and complicates logic — must remove salt after aggregation.- Map-side aggregation is best when aggregation function is associative/commutative.- Memory increases are short-term relief; algorithmic fixes scale better.

Summary checklist to run in production:- Alert on skew metrics (max shuffle bytes, task time)- Identify top keys via sampling/aggregates- Implement map-side pre-aggregation + selective salting for hot keys- Validate on minimal repro and measure task distributions and memory- Deploy with feature flag, monitor, and roll back if regressions appear.

MediumSystem Design

66 practiced

Design a concise observability plan (logs, metrics, traces, alerts) that would let you detect the earliest signs of a failing ETL job and help you debug rapidly under time pressure. Specify 6-8 concrete metrics/logs and the alert thresholds you'd set to balance noise vs sensitivity.

Sample Answer

Requirements & goals:- Detect earliest signs of an ETL failure (ingestion lag, data loss, schema drift, performance regressions).- Rapid root-cause with minimal noise.- Actionable alerts with runbook links.

High-level plan:- Centralized logs (structured JSON) with pipeline/job-id, task, stage, dataset, record counts, durations, errors, input offsets.- Metrics exported to a time-series DB (Prometheus/Cloud Monitoring).- Tracing for critical steps (ingest → transform → write) with trace IDs correlated to logs.- Dashboards: job health, throughput, latency, error breakdown, recent failures.

Concrete metrics/logs & alert thresholds:1) Job run success rate: percentage of successful runs over last 1h < 99% → Alert (pager). (sensitive)2) End-to-end latency: job completion time > baseline_mean + 3*std OR > SLA (e.g., 2× normal) for 2 consecutive runs → Alert (pager).3) Input lag / watermark delay: ingestion lag > 5 minutes (stream) or > scheduled window + 10% (batch) → Alert (page if sustained 2 windows).4) Processed record count deviation: current run count < 90% of median of past 7 runs → Alert (email first, pager if persists 2 runs).5) Error rate (per-stage): errors per minute > 5 OR error% > 0.5% of processed records → Alert (pager if in write/ingest stage; email otherwise).6) Schema-change detected: unexpected schema fields/types mismatch → Immediate Alert (pager).7) Downstream write retries / DLQ size: retries > 50 in 10m or DLQ growth rate > 1000/min → Alert (pager).8) Trace-sampled spans showing >50% slow operations in transform stage (p95 duration > threshold) → Alert (email + link to trace).

Alerting strategy:- Tiered: immediate paging for failures causing data loss or missed SLAs; email/Slack for anomalies needing investigation.- Dedup/cooldown: group by job-id and error type, suppress repeated alerts for 15 min.- Escalation: auto-escalate if unresolved 30 min.

Runbook (linked in alert):- Quick checks: recent logs (job-id), trace link, last successful run, input offsets, DLQ samples.- Typical fixes: restart job, backfill window, rollback schema change, increase resources.- Postmortem checklist and owner assignment.

Why this balances noise vs sensitivity:- Combines absolute thresholds (latency/SLA) with relative baselines (record deviations) to catch regressions but avoid transient spikes.- Correlated logs + traces enable one-click jump from alert to root cause.

EasyTechnical

63 practiced

Explain how you would use git bisect (binary search across commits) to find a commit that introduced a regression in a data pipeline test suite which fails intermittently in CI. Describe how to make the bisection reliable when tests are flaky and how to minimize developer time during the bisect.

Sample Answer

Approach (overview):- Make the test outcome deterministic enough for bisect, automate the binary search, and minimize manual runs by scripting retries and using CI to run bisection steps.

Steps:1. Reproduce and narrow range- Confirm a commit range where "good" (old) and "bad" (failing) are known.- Re-run the flaky test locally/CI to understand failure rate and logs.

2. Make verdicts robust (handle flakiness)- Decide on a statistical rule: run the test N times per commit and treat commit as bad iff failures >= T (e.g., N=5, T=2) or use majority.- Capture deterministic environment: freeze external dependencies, use recorded datasets or mocked inputs, fix timestamps, set seeds.

3. Automate git bisect with a runner- Use git bisect start; git bisect good <good-commit>; git bisect bad <bad-commit>.- Use git bisect run <script> where script builds and runs the pipeline test N times and exits 0 for good, 1 for bad, 125 to skip ambiguous commits.

Example runner (bash):

bash

#!/usr/bin/env bash
# bisect-run.sh: exit 0 if commit is good, 1 if bad, 125 to skip
set -e
N=5
T=2
failures=0

# build the project (adjust for your build system)
./build.sh >/tmp/build.log 2>&1 || { echo "build failed"; exit 125; }

for i in $(seq 1 $N); do
  # run test in isolated env; ensure deterministic inputs
  if ! ./run_pipeline_test.sh >/tmp/test_$i.log 2>&1; then
    failures=$((failures+1))
  fi
  # optional: short sleep to avoid transient infra limits
  sleep 1
done

if [ $failures -ge $T ]; then
  echo "BAD: $failures/$N failures"
  exit 1
else
  echo "GOOD: $failures/$N failures"
  exit 0
fi

Run:- git bisect start- git bisect good <good-commit>- git bisect bad <bad-commit>- git bisect run ./bisect-run.sh

Minimize developer time- Run the bisect inside CI runner (use a dedicated job that checks out commits and runs bisect). This frees the developer machine and provides consistent environment.- Cache heavy artifacts (Docker layers, built jars) or use incremental builds to speed each bisect step.- Use skip (exit 125) for commits that fail to build or have unrelated CI flakiness.- Parallel strategy: if range is large and you have CI capacity, split the range manually into halves and run two concurrent bisects to find candidates faster, then narrow.

Notes and trade-offs- Larger N reduces false positives but increases time per step; pick N based on observed flakiness.- Ensure external services are mocked or stable; otherwise bisect may blame infrastructure changes instead of code.- Keep logs for each tested commit to ease root-cause analysis once the offending commit is found.

HardTechnical

88 practiced

An intermittent bug affects ~1% of records and only shows up in production. Lay out a systematic hunt plan: sampling strategies, binary search across pipeline transforms, invariant checks to add, and approaches to capture payloads for offline root-cause analysis without leaking sensitive data.

Sample Answer

Start by framing scope and goals: reproduce root cause offline for affected ~1% while minimizing production impact and avoiding PII leaks. Plan has four phases: sample & isolate, binary-search through pipeline, add invariants/telemetry, capture safe payloads.

1) Sampling strategies- Targeted sampling: instrument a lightweight filter that tags records meeting production-failure signature (error logs, downstream nulls, schema violations). Sample 100% of tagged records; sample ~0.1–1% of all others as baseline.- Time-window sampling: increase sampling during suspected high-failure windows (rolling 5–15m).- Reservoir sampling at ingress to get unbiased sample of raw traffic across time.- Stratified sampling: sample across keys (user_id hash buckets, partition ranges) to detect hot spots.

2) Binary search across transforms- Treat pipeline as ordered transforms T0 (ingest) → T1 → ... → Tn (sink). Insert lightweight "canary" assertions or diff snapshots between Ti and Ti+1.- Do staged replay of sampled raw records through prefix of pipeline until divergence appears; this localizes offending transform.- Use delta checks: compare pre/post record counts, schema, and key distributions. If divergence occurs between Ti and Ti+1, instrument Ti+1 more deeply.

3) Invariant checks to add- Schema invariants: required fields non-null, types, ranges.- Referential invariants: foreign key existence, deduplication idempotency.- Statistical invariants: value distributions, cardinality, percent nulls, entropy on categorical fields.- Business invariants: totals sum consistency, monotonic sequence constraints, timestamp ordering.- Add fail-fast counters and histogram metrics per transform and per data shard.

4) Capturing payloads safely- Minimal capture: capture only fields necessary to debug; avoid PII.- Pseudonymize: replace raw identifiers with stable salted hashes (HMAC-SHA256 with key rotated/stored in KMS) so you can correlate records without exposing PII.- Tokenization & redaction: mask emails, phone numbers; redact free-text but keep structure length info.- Differential capture modes: high-fidelity for sampled tagged records (salted hashes + truncated text), low-fidelity for baseline.- Encryption-at-rest and access controls: store captured payloads in a secured bucket with IAM, audit logs, short TTL.- Use deterministic hashing of keys so you can aggregate and trace without raw values.

Operational items- Implement feature flags to enable/disable extra instrumentation with zero-downtime.- Backfill replays in sandbox with same code/version; use synthetic amplification of failing cases.- Automate alerting on invariant breaches with links to captured payload IDs and hashes.- Post-mortem: once fixed, add unit/integration tests covering the failing pattern and remove heavy instrumentation.

Example tools & implementations- Kafka: add a side-topic for sampled records with headers marking sample reason.- Spark: add mapPartitions checks that emit Metrics to Prometheus and write sampled rows to secure S3 with KMS encryption.- Use OpenTelemetry/Logs: attach record-level trace_id (salted) to correlate logs across services.

This approach localizes the bug efficiently, preserves privacy via hashing/tokenization, and provides reproducible payloads for offline root-cause analysis while keeping production impact minimal.

Unlock Full Question Bank

Get access to hundreds of Debugging and Recovery Under Pressure interview questions and detailed answers.

Join thousands of developers preparing for their dream job.