Debugging and Systematic Troubleshooting Questions
Diagnosing defects methodically: reproducing failures, forming and testing hypotheses, reading stack traces and logs, bisecting changes, and reasoning about error handling and edge cases. Covers a disciplined root-cause approach that applies from local bugs to production issues, distinct from embedded hardware-level debugging. A universally probed engineering-craft skill.
Common Spark errors include 'Task not serializable' and 'Executor lost.' For each error, describe the likely root causes, the log sources and Spark UI pages you would inspect, quick reproductions you would run locally or on a small cluster, and both immediate mitigations and long-term fixes a data engineer should apply to prevent recurrence.
Sample Answer
Direct answer
"Task not serializable" means Spark tried to ship a closure to executors that references something non-serializable, almost always an outer class instance or a connection/resource object captured accidentally by the closure; "Executor lost" means an executor process died or became unreachable, most commonly from memory exhaustion, but also from a node failure or a network partition, and the fix in each case is specific to that mechanism, not a generic "add more resources" response.
Structured elaboration
"Task not serializable":
- Root cause: Spark closures that run on executors must serialize everything they reference; a lambda that accidentally captures
this(an outer class instance, common in Scala/Java when a method reference implicitly captures its enclosing object) or a non-serializable object (a database connection, a file handle, a non-serializable third-party client) directly in scope causes this at job-submission or task-scheduling time. - Where to look: the Spark UI's failed-stage detail and the driver log show the exact exception with a stack trace naming the non-serializable class; that class name is usually the direct answer to "what got captured."
- Quick reproduction: the error surfaces immediately at job submission in most cases (Spark validates serializability of the task closure before dispatching), so it's typically reproducible locally in a small test job referencing the same closure, without needing cluster-scale data.
- Immediate mitigation and long-term fix: immediate: rewrite the closure to avoid capturing the problematic object (extract only the specific primitive values needed, rather than the whole enclosing object or connection). Long-term: establish a pattern for resources that genuinely can't be serialized (like a database connection), such as creating them fresh WITHIN each executor task (e.g., inside a
mapPartitionscall, one connection per partition) rather than trying to serialize and ship a connection created on the driver.
"Executor lost":
- Root cause (most common): memory exhaustion, either from data skew (one partition far larger than others, causing that executor to OOM while others are fine) or from an operation that materializes more data in memory than the executor's allocation allows (a large broadcast join, a
collect()on a large dataset, excessive caching). - Where to look: the Spark UI's executor tab shows which specific executor was lost and its resource usage leading up to the loss; cluster-manager logs (YARN, Kubernetes, standalone) show whether it was an explicit OOM kill, a health-check failure, or a genuine node/hardware failure.
- Quick reproduction: reproduce on a smaller cluster or with a data sample sized to trigger the same skew pattern, checking per-partition size distribution specifically (via
.glom().map(len), a Spark call that groups each partition's rows into one list so you can measure its size directly, or the Spark UI's stage-level task-duration/shuffle-size skew view) to confirm skew as the mechanism before assuming it. - Immediate mitigation and long-term fix: immediate: increase executor memory as a stopgap, or reduce the offending stage's parallelism/partition size to lower per-task memory pressure. Long-term: address data skew directly (salting a skewed join key, or using an adaptive query execution feature if the engine supports it (AQE: the engine watches actual partition sizes while the job runs and automatically re-splits an oversized one instead of you having to predict and fix it up front)) rather than just over-provisioning memory to paper over an ever-growing skew problem, which will resurface as data volume grows.
Worked example
A job fails with "Task not serializable" naming a Kafka producer client held as a field on the driver-side class whose method is being called inside a map closure. The Spark UI confirms the exact class in the stack trace. Fix: move producer creation inside a mapPartitions closure so each executor creates its own producer instance locally, never serializing the producer itself across the wire.
Separately, "Executor lost" recurring on the same stage each run: the executor tab shows the SAME executor index consistently hitting its memory ceiling, and the stage's task-duration view shows one task taking 20x longer than its peers, both pointing at skew on that specific partition rather than a general under-provisioning issue. Salting the skewed join key to spread that key's records across multiple partitions resolves it, while simply raising memory limits (tried first, as the quick mitigation) had only delayed the failure to a larger data volume rather than fixing it.
Trade-offs and pitfalls
Treating "Executor lost" as always a generic capacity problem (just add more memory) is the most common mistake, since it works as an immediate mitigation for genuinely under-provisioned jobs but does nothing for skew-driven cases, where the SAME task will keep needing more memory than its peers no matter how much is provisioned, until the underlying data distribution problem is actually fixed.
Explain, with examples, how cognitive biases such as confirmation bias, anchoring, and sunk-cost fallacy can hinder a debugging investigation. Describe concrete practices, such as pair debugging, rotating investigators, hypothesis logs, and clear acceptance criteria, that you have introduced on a team to mitigate these biases.
Sample Answer
Direct answer
Confirmation bias, anchoring, and sunk-cost fallacy each distort a root-cause investigation in a specific, predictable way: confirmation bias makes you notice evidence supporting your first guess and discount evidence against it; anchoring makes an early, possibly wrong hypothesis dominate the rest of the investigation even after better evidence emerges; and sunk-cost fallacy keeps you investigating a disproven lead because of time already invested in it, rather than switching based on current evidence. Concrete team practices (pair debugging, rotating investigators, hypothesis logs, explicit acceptance criteria) counter each of these by introducing structure that doesn't rely on individual willpower to overcome the bias.
Structured elaboration
Confirmation bias: once you suspect a cause, you unconsciously interpret ambiguous evidence as supporting it and are quicker to dismiss evidence that doesn't fit. In debugging, this looks like reading a log line as "consistent with my theory" when a more neutral read would call it inconclusive, or stopping the investigation the moment ANY supporting evidence appears rather than continuing to look for disconfirming evidence too.
- Mitigation: hypothesis logs. Writing down each hypothesis and what specific evidence would DISCONFIRM it, before looking for evidence, forces a falsifiable framing up front; when you later find evidence, you check it against the pre-written disconfirmation criteria rather than retroactively deciding it "counts" as support.
Anchoring: the first plausible explanation offered (by you, or by someone else on the call) tends to dominate the rest of the investigation's framing, even as better evidence appears, because everyone's mental model has already organized around it.
- Mitigation: rotating investigators / a fresh pair of eyes. Someone joining the investigation LATE, without the accumulated anchor, will naturally form hypotheses from the current evidence rather than the initial framing, and is often the person who notices the anchor was wrong; deliberately bringing in a fresh perspective partway through a long investigation is a structural way to interrupt this.
Sunk-cost fallacy: having spent two hours pursuing one lead makes it psychologically harder to abandon, even once evidence stops supporting it, because abandoning it "wastes" the time already spent (which is, of course, already spent either way, and not actually recoverable by continuing).
- Mitigation: explicit acceptance criteria and timeboxing set in advance. Deciding, BEFORE starting to investigate a specific hypothesis, what evidence would confirm it and how much time is reasonable to spend testing it, makes the "abandon or continue" decision a pre-committed rule rather than an in-the-moment judgment call that sunk cost can distort.
Pair debugging as a mitigation for all three simultaneously: a second person, thinking independently, is less likely to share the exact same anchor or the exact same sunk-cost attachment to a specific lead, and naturally provides a real-time check on confirmation bias by asking "does that evidence actually support that, or are we reading it generously?"
Worked example
A team investigating an intermittent failure anchors early on "it's probably the recent deploy" (a reasonable first guess, given the timing). Two hours in, with the deploy's code reviewed thoroughly and nothing found, sunk cost starts to argue for continuing to scrutinize that same deploy rather than considering it possibly unrelated. A hypothesis log, written at the start, had specified "if the failure recurs on a service instance that predates this deploy, that disconfirms the deploy hypothesis"; checking that specific, pre-committed criterion shows the failure DID recur on an older instance, cleanly disconfirming the deploy theory despite two hours of sunk investigation into it. A rotating fresh investigator, brought in specifically because the original two were stuck, asks a question neither anchored investigator had considered ("what else changed around that time besides the deploy?") and identifies an unrelated infrastructure change that actually explains the failure.
Trade-offs and pitfalls
These practices have a real cost (pairing takes two people's time instead of one; hypothesis logs take a few minutes to write that could otherwise go straight into investigating), and the trade-off is worth it specifically for investigations that are ALREADY taking a long time or where the cost of a wrong conclusion is high; for a five-minute, low-stakes bug, the overhead of formal hypothesis logging isn't proportionate to the risk these biases actually pose in that context.
An upstream service emits events with out-of-order timestamps, causing incorrect downstream aggregations, and the same class of issue has broken a streaming ingestion pipeline's aggregations before. Describe methods to detect out-of-order events, the windowing strategies you would use to handle event-time processing including watermarking and allowed lateness, and how you would design transformations to be robust to reordering and late arrivals. Separately, describe the immediate mitigations you would apply while a fix is in flight, and the diagnostic steps and long-term design changes, such as buffering or idempotency, you would implement afterward.
Sample Answer
Direct answer
Detecting out-of-order events means comparing each event's OWN timestamp (event time) against the order it actually arrived in (processing time), and handling them robustly means designing the aggregation logic to tolerate a bounded amount of lateness via watermarking rather than assuming arrival order matches event order. Separately, when out-of-order arrival has already broken downstream aggregations in production, the response splits into an immediate mitigation and a long-term design fix.
Structured elaboration
Detecting out-of-order events: track, per event source or partition, the maximum event-time timestamp seen SO FAR; any newly arriving event whose timestamp is earlier than that running maximum (minus some small, expected jitter tolerance) is out of order. This is cheap to compute as a running comparison and doesn't require buffering to detect, only to CORRECT for.
Windowing strategies for event-time processing:
- Watermarking: the processing engine tracks an estimate of "how far behind can we assume events have arrived by now," advancing as time passes; a window is finalized once the watermark passes its end boundary, giving late-but-within-tolerance events a chance to be included before the window closes.
- Allowed lateness: an explicit, bounded grace period after a window's watermark-triggered close during which further late events are still accepted and merged into an already-emitted (but revisable) result, rather than silently dropped; the width of this tolerance is a direct trade-off between correctness (fewer legitimately-late events dropped) and result latency (how long you wait before considering a window "final").
- Choosing tolerances: based on the OBSERVED lateness distribution of the actual upstream source (measure it, don't guess), setting the allowed-lateness window wide enough to capture the large majority of legitimately late events without indefinitely delaying every window waiting for a rare, extreme outlier.
Designing transformations to be robust to reordering: use event-time (not processing-time) as the basis for all windowing and aggregation logic; make aggregation operations idempotent/commutative where possible (a sum or count naturally tolerates late arrival if the window is still open or within its lateness tolerance, simply incrementing the existing partial result); and explicitly define and monitor a dropped-events metric for anything arriving PAST the allowed-lateness window, so silently-dropped late data is at least visible and quantified rather than invisible.
Immediate mitigations while investigating an active incident: widen the allowed-lateness window temporarily if the current lateness distribution has shifted wider than the configured tolerance (a quick, reversible change); if a specific upstream source is confirmed to be the origin of unusually late events, consider a temporary buffering/reordering stage specifically for that source before it enters the main aggregation pipeline, isolating the fix to where it's needed rather than loosening tolerance globally.
Diagnostic steps and long-term design changes: confirm WHY the upstream source is emitting out-of-order events in the first place (a multi-producer system with no synchronized clock or sequencing, a retry that re-emits an event with its original timestamp after a delay, a partition rebalance causing reordering) since the long-term fix differs depending on the mechanism: a synchronized/monotonic sequence number from the source (if achievable) can eliminate ambiguity entirely; a redesigned watermarking strategy (per-partition rather than global, if the lateness pattern differs meaningfully by partition) can better match the actual data's behavior; and buffering with an explicit, monitored reordering stage is the right fix when the source's out-of-order behavior is a known, bounded, and unavoidable characteristic rather than a bug to fix at the source.
Worked example
An upstream service occasionally retries a failed publish, re-emitting an event with its ORIGINAL event-time timestamp well after the window it belongs to has already closed and been finalized downstream, because the retry logic (correctly) preserves the original timestamp rather than using retry-time. Immediate mitigation: widen the allowed-lateness window to comfortably cover the observed retry-delay distribution, measured from the retry mechanism's own logs. Long-term: add explicit monitoring for events arriving past the lateness window (quantifying how much is actually being dropped, which turns out to be a small, bounded, and now-understood amount from this specific retry pattern) and document the trade-off explicitly rather than continuing to silently drop them with the tolerance too tight.
Trade-offs and pitfalls
Widening the allowed-lateness window indefinitely to catch every possible late event trades away result LATENCY (windows take longer to finalize) for completeness, which isn't free; the right tolerance is set from the actual, measured lateness distribution of the specific source, not a defensively-large guess, and should be revisited if the upstream source's behavior changes.
Give a concise strategy for debugging an intermittent data-corruption issue in a distributed system. Include how you would trace where the corruption occurs, whether in the producer, transport, or storage layer, what checksums or verification techniques you would add, and how you would prove the fix actually worked.
Sample Answer
Direct answer
Trace intermittent data corruption by checking each stage of the pipeline in order (producer, transport, storage) with a technique that can PROVE where the value diverges from what was intended, most effectively a checksum or content hash computed at the producer and re-verified at each downstream stage, since that turns "the data looks wrong somewhere" into "the data was correct through stage N and wrong starting at stage N+1," which directly localizes the bug.
Structured elaboration
- Add a verifiable checksum at the origin. Compute a hash (even something simple like a CRC32 or SHA-256 of the payload) at the producer, at the moment the data is known-correct, and propagate that checksum alongside the data through every subsequent stage.
- Re-verify the checksum at each stage boundary, not just at the final destination. Checking only at the end tells you corruption happened SOMEWHERE in the whole pipeline; checking at each intermediate stage (after transport, after any transformation, before and after storage) tells you exactly which stage introduced it, turning a system-wide investigation into a single-stage one.
- Distinguish the three candidate layers by what kind of corruption each typically produces. Producer-side corruption usually means a logic bug (an encoding mismatch, a race condition writing the record, a serialization bug) and tends to be deterministic given the same input. Transport-layer corruption is more often non-deterministic and load-dependent (a buggy network device, a race in a message broker's replication, a partial write under backpressure). Storage-layer corruption can look like either, depending on whether it's a write-path bug (deterministic) or a hardware/replication issue (often not).
- Correlate corruption events with load and timing, not just presence/absence. Corruption that only appears under high throughput points toward a race condition or a buffer-reuse bug somewhere in the pipeline; corruption on a fixed, small subset of records regardless of load points toward a specific data shape or encoding edge case.
- Reproduce in isolation once localized. Once the checksum trail narrows the bug to one stage, build a minimal, isolated reproduction of just that stage (feeding it the confirmed-good input from the prior stage) to nail down the exact mechanism, rather than continuing to debug against the full, harder-to-control production pipeline.
Worked example
Adding a SHA-256 checksum at the producer and re-verifying after transport (a message queue) and after storage (a database write) on a system experiencing occasional corrupted records: checksums confirm the data is correct through transport but starts failing verification specifically after storage, and specifically only under high write concurrency. That localizes the bug to the storage write path under concurrent access, not the producer or the queue, and points toward a race condition in the write logic (perhaps a non-atomic read-modify-write on a shared buffer) rather than a data-encoding issue, which is exactly the kind of localization a checksum trail provides that "the data looks wrong sometimes" alone cannot.
Proving the fix worked: after applying a fix (in this example, making the storage write path atomic), re-run the same checksum verification at every stage under the same high-concurrency load that originally triggered the corruption, ideally for a duration and volume comparable to what exposed the bug the first time, and confirm zero checksum mismatches, not just "it looks fine on a quick spot check."
Trade-offs and pitfalls
Adding checksums at every stage has real overhead (compute cost, and the plumbing to propagate and verify them), so the pragmatic approach for a purely diagnostic investigation is to add them TEMPORARILY at stage boundaries during the investigation, then decide whether any subset is worth keeping permanently as an ongoing data-integrity guard once the root cause and fix are understood. The trap is declaring victory once corruption stops being VISIBLE without ever confirming the checksum trail is clean end to end under representative load, since intermittent bugs can easily appear "fixed" simply because you got lucky on the next few observations.
You are on-call and discover that a schema change caused duplicate records in a data warehouse after an incremental load. Outline the immediate mitigation steps to stop further duplication, the debugging steps to find the root cause, and the long-term fixes to prevent similar regressions, considering change-data-capture, dedup keys, and backfills.
Sample Answer
Direct answer
Stop further duplication immediately (pause or disable the incremental load, since every additional run compounds the problem), then find and fix the root cause of the schema change, and only then remediate the already-duplicated data, in that order, because remediating data while the load is still actively creating more duplicates is chasing a moving target.
Structured elaboration
- Immediate mitigation: stop the bleeding first. Pause the incremental load job (or, if it can't be paused cleanly, block it from running its next scheduled execution) the moment duplication is confirmed, so the scope of the problem is bounded and known rather than continuing to grow while you investigate.
- Debugging steps to find the root cause. Check what specifically changed in the schema, and how the incremental load's own logic determines "is this a new row or an update to an existing one": a very common mechanism is that incremental loads rely on a specific key or a change-detection column (a primary key, a last-modified timestamp) to decide whether to insert or upsert, and a schema change that altered, renamed, or changed the semantics of that specific column can silently break the load's ability to recognize a row it already has, causing it to insert a fresh duplicate instead of updating the existing record.
- Confirm the mechanism concretely by comparing the schema before and after the change specifically for any column the load logic depends on for deduplication or matching, not just a general schema diff; the bug is almost always localized to exactly the column(s) the incremental logic keys on.
- Long-term fixes to prevent recurrence, considering CDC, dedup keys, and backfills:
- If using change-data-capture (CDC): confirm CDC's own key/identity tracking survived the schema change correctly; a CDC setup that identifies rows by a column that got renamed or restructured can silently start treating every row as new.
- Dedup keys: ensure the incremental load's matching key is a stable, intentionally-designed business key (not an incidental column that happens to work today), and add an explicit uniqueness constraint or a deduplication step immediately after load as a structural safety net, so even a future matching-logic bug produces a constraint violation (loud, caught immediately) rather than silent duplicate rows.
- Backfills: any future schema migration affecting a matching/dedup key should have an explicit backfill or migration step for the load logic itself (not just the table schema), tested against a realistic incremental-load scenario before the schema change ships, not discovered afterward via a production incident.
Worked example
An incremental load matches rows on a customer_id column to decide insert-vs-update; a recent schema change split what was previously a single customer_id column into a normalized customer_id plus a separate region_code, and the incremental load's matching logic (still comparing on the old combined format) now fails to recognize existing rows, inserting a duplicate for every incrementally-loaded row post-migration. Immediate mitigation: pause the load, preventing further duplication. Root-cause confirmation: comparing pre/post-migration schema for the specific matching column confirms the mechanism directly. Long-term fix: update the incremental load's matching logic to use the new key structure, and add a uniqueness constraint on the correct post-migration key as a structural safety net against a similar future mismatch. Data remediation: identify duplicate rows via the corrected key logic (grouping by the TRUE business key, not the broken matching key that created the duplicates), keep the earliest or most-complete record per group per an agreed business rule, and delete the rest in a single, carefully-validated batch operation, run only after confirming the load's matching logic is fixed and re-tested, so the cleanup isn't immediately undone by the next incremental run.
Trade-offs and pitfalls
Remediating duplicate data BEFORE fixing the root cause (a tempting instinct, since duplicates are the visible, urgent symptom) risks having the very next incremental run recreate the exact same duplicates, wasting the remediation effort; the ordering (stop, diagnose, fix, then remediate) exists specifically to avoid that. Adding a hard uniqueness constraint as a structural safety net is valuable precisely because it converts a FUTURE instance of this same bug class from a silent data-quality issue into a loud, immediately-visible load failure.
Unlock Full Question Bank
Get access to all 47 Debugging and Systematic Troubleshooting interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.