Data Pipeline Monitoring and Observability Questions

Focuses on designing monitoring and observability specifically for data pipelines and streaming workflows. Key areas include instrumenting pipeline stages, tracking health and business level metrics such as latency throughput volume and error rates, detecting anomalies and backpressure, ensuring data quality and completeness, implementing lineage and impact analysis for upstream failures, setting service level objectives and alerts for pipeline health, and enabling rapid debugging and recovery using logs metrics traces and lineage data. Also covers tooling choices for pipeline telemetry, alert routing and escalation, and runbooks for operational playbooks.

HardTechnical

28 practiced

Create a runbook template and an automated playbook for a common incident: 'nightly batch delayed > 2 hours'. Include triage steps, key metrics/queries to run, common root causes to check, immediate mitigations (automated where possible), notification wording for stakeholders, and escalation criteria.

Sample Answer

Runbook: Nightly batch delayed > 2 hours

Summary: nightly_batch job(s) expected to complete by 03:00. Trigger: observed delay > 2h past scheduled end.

Triage (first 10–15 min)1. Confirm alert validity: check scheduler state (Airflow/GCP Composer/AWS MWAA/K8s cron). - Query: SELECT state, start_date, end_date FROM dag_run WHERE dag_id='nightly_batch' ORDER BY start_date DESC LIMIT 5;2. Check job progress and last successful step logs (last 2 hours). - tail logs via log service or: gsutil/az storage list / aws s3 ls3. Verify cluster/worker health: check Yarn/Spark UI, executor counts, queue backlog. - Spark REST: /applications → stages/running4. Check upstream data availability (source landing timestamps). - Query: SELECT max(arrival_ts) FROM source_table WHERE partition=date_sub(current_date(),1);

Key metrics/queries- Run duration, start lag, task failure count, executor memory usage, shuffle read/write sizes.- Example: Prometheus: rate(task_failures[15m]), spark_executor_cores_used

Common root causes- Resource starvation (too few executors, autoscaling failed)- Downstream/upstream data missing or delayed- Schema/data quality errors causing job crash or retries- External service outage (DB, S3)- Backfill or unexpected job concurrency

Immediate mitigations (automated preferred)1. Auto-check & restart: - If scheduler shows running but no progress >30m on same stage → restart task pod/job automatically with backoff.2. Scale-up: - If CPU/memory >85% and cluster autoscaler disabled → trigger autoscaler API to add nodes.3. Bypass bad data: - If data quality rule fails for small percentage (<1%), auto-apply quarantine and continue (flag partitions).4. Fallback to last-good snapshot: - If upstream missing, run consumer against previous day's partition and mark data as late.5. Circuit-breaker & suppress alerts during automated remediation steps (30–60m).

Automated playbook (pseudoflow)- Alert -> Run diagnostics script: - gather: scheduler state, last logs, spark metrics, source arrival timestamps - if transient (retrying tasks <3 and errors are transient) → wait 30m and recheck - if resource saturation → call autoscale API, notify oncall - if task failed due to data schema → quarantine partition, run schema-compat check, notify data producer - if external dependency down → open incident with vendor, trigger fallback snapshot- After remediation, run verification job; if success -> close alert with runbook notes; else escalate.

Notification wording (initial)Subject: INCIDENT: Nightly batch delayed >2h — action startedBody:- Summary: nightly_batch for [date] delayed by X hours.- Impact: downstream reports/jobs affected: [list]- Actions taken: diagnostics started; autoscale triggered / job restart attempted / data quarantined.- ETA: initial remediation in progress — expect update in 30 minutes.- Oncall: [name] contact: [phone/Slack]

Escalation criteria- After 30 min remediation no progress OR error repeats after automated restart → pager escalation to senior data engineer.- After 60 min unresolved with critical business impact (reports unavailable for stakeholders) → escalate to engineering manager and SRE on-call.- If external vendor outage with SLA breach or >4 hours → notify product and business ops and initiate postmortem.

Post-incident- Record timeline, root cause, actions, and RCA owner.- Implement preventive action: e.g., alert tuning, autoscaling tests, schema-contract checks, SLAs with upstream.- Add automated tests to CI that simulate common failure modes.

This template balances fast automation with clear human escalation and stakeholder communication; adapt thresholds and APIs to your stack.

HardSystem Design

26 practiced

Design SLOs, error budget allocation, and an automated mitigation strategy for a mission-critical dataset that powers both ML models and business dashboards. Include per-consumer prioritization, how to charge error budget for partial degradations, and possible automated mitigations such as partial-replay, prioritized backfills, or graceful degradation.

Sample Answer

Requirements & constraints:- Mission-critical dataset serves ML models (low-latency, deterministic freshness) and business dashboards (aggregate freshness, eventual consistency).- Scale: high ingest, multiple consumers with different SLAs.- Non-functional: detect, quantify partial degradations, automated mitigations with minimal manual ops.

SLO design (examples):- Freshness SLO (per consumer class): time since last successful partition update. - ML-online: 99.9% of minutes with <30s lag. - ML-batch: 99% of hourly windows <5m lag. - Dashboards: 99% of hourly snapshots <15m lag.- Accuracy SLO: schema-valid rows and QC checks pass on 99.9% of records for ML; dashboards tolerate 99.0%.- Availability SLO: dataset read success rate: ML-online 99.95%, others 99.5.Associate each SLO with an Error Budget = 1 - SLO.

Per-consumer prioritization & error budget allocation:- Classify consumers: P0 (ML-online inference), P1 (ML-training), P2 (dashboards/analytics).- Split global error budget proportionally (e.g., 50% to P0, 30% to P1, 20% to P2) or by business impact.- Track budgets per consumer and per failure mode (freshness, accuracy, availability).- Maintain a central ledger: each incident consumes budget for affected consumers.

Charging model for partial degradations:- Define degradation severity functions: - Freshness: proportional to lag over threshold. Example: if ML-online threshold 30s, a 90s lag for 10 minutes consumes (90-30)/30 * (10/total window) of budget. - Accuracy: percent of failed QC * weight. - Availability: percent of failed requests.- For multi-tenant consumption, charge each consumer only for the portion of pipeline affecting them (e.g., a partition-level failure affects only consumers reading that partition).- Cap single incident charge to avoid blowing budget on noise; accumulate rolling-window.

Automated mitigation strategy:1. Detection & classification - Streaming monitors (Prometheus/Cloud Monitoring) + data-quality checks (Deequ, Great Expectations) feed an alerting engine. - Auto-classify incident: ingestion delay, transform failure, sink/backpressure, partial data loss.

2. Priority-driven mitigation flow - If incident affects P0 consumers and error budget remains: - Graceful degradation: switch ML-online to cached feature snapshots or scoreboard fallback model (predictor less reliant on freshest features). - Partial-replay: replay recent raw messages for missing partitions to a hot path (stream-first replay) prioritizing partitions used by P0. - Prioritized backfill: launch targeted backfill for keys/partitions with highest traffic/importance. - If P0 budget exhausted but P1/P2 budgets available: - Reserve recovery capacity to P0 until budget replenishes; throttle non-critical rebuilds. - For P2-only impact: - Defer expensive reprocessing, serve approximate aggregates (approx quantiles, pre-aggregated rollups), annotate dashboards with data freshness warnings.

3. Orchestration - Use an automation engine (Airflow/Kubernetes operator) implementing runbooks: - Detect -> evaluate budget -> select mitigation policy -> execute (partial replay, backfill, fallback) -> verify -> close incident. - Implement safety checks: ensure replays are idempotent / deduplicated; use watermarking to avoid double-counting.

Implementation details & trade-offs:- Store per-partition metadata (last-good-offset, schema-version, QC-timestamp) in a metadata store (DynamoDB/Postgres).- Maintain consumer manifests mapping features/endpoints -> partitions; enables targeted replays/backfills.- Replay mechanism: keep raw event log (Kafka) for N days; fast-path replay uses Spark Structured Streaming or Beam to reconstruct hot partitions.- Prioritized backfill engine supports incremental windowed re-computation and resource-aware scheduling.- Trade-offs: complexity vs coverage. Favor targeted mitigations to minimize compute cost and time-to-recovery. Graceful degradation reduces user impact but requires building fallback logic in consumers.

Monitoring & governance:- Dashboards: error budget burn rate, per-consumer SLOs, incident taxonomy.- Automated alerts on burn-rate thresholds (e.g., 25%, 50%, 90%).- Post-incident: auto-create postmortem with consumed budget, root cause, manual actions required; adjust SLOs or capacity as needed.

This design ensures per-consumer prioritized SLAs, rational budget charging for partial degradations, and automated, low-latency mitigations (partial-replay, prioritized backfills, graceful degradation) to reduce business impact while controlling cost and operational risk.

HardTechnical

21 practiced

Given time series for consumer lag, producer throughput, and broker CPU/disk metrics, design an algorithm to detect correlated anomalies and produce a ranked list of likely root causes (e.g., producer slowdown, broker disk pressure, consumer saturation). Describe feature extraction, correlation windows, scoring function, and how to present results to an operator.

Sample Answer

Clarify goal: detect temporally correlated anomalies across consumer lag, producer throughput, and broker CPU/disk and rank likely root causes (producer, broker disk, consumer).

Algorithm overview:1. Feature extraction- Normalization: per-metric robust scaling (median, MAD) to handle skew.- Short/long windows: compute rolling stats at two scales (fast 1–5m, slow 30–60m): mean, std, z-score, rate-of-change, percent drop/rise, backlog deltas (lag derivative), throughput delta.- Binary anomaly flags: z-score > 3 or adaptive percentile threshold; produce anomaly magnitude = |z-score|.

2. Correlation & causal signal- For each metric pair, compute sliding-window cross-correlation and find max correlation and argmax lag (positive lag means metric A leads B).- Complement with Granger-causality (on aggregated windows) or transfer-entropy for non-linear signal when data supports it.- Build directed anomaly propagation graph where edges weight = correlation_strength * decay(lag) * co-occurrence_score (fraction of windows both anomalous).

3. Scoring / ranking root causes- Candidate causes = producers, brokers (CPU/disk), consumers.- For each candidate, score = sum_over_outgoing_edges(weighted_edge) * avg_anomaly_magnitude(candidate) * lead_bonus. - lead_bonus = exp(-alpha * positive_lag) (leaders get boost). - Penalize if candidate anomaly is transient while downstream sustained (reduce score).- Normalize to [0,1] and produce top-K ranked list.

4. Presentation to operator- Time-series swimlane with aligned anomalies and annotated lead-lag arrows.- Ranked causes with score, top contributing metrics, evidence: correlation, lag, windows where co-occurred, example timestamps.- Suggested actions: e.g., "Investigate broker disk I/O: sustained high disk utilization + producer throughput drop follows within 2m."- Confidence band & provenance: show which tests produced the score (cross-corr, Granger).

Implementation notes:- Real-time pipeline: use Spark Streaming/Beam for rolling stats, persist features in TSDB (Prometheus/Influx), compute correlations in windowed jobs.- Tune alpha, window sizes via historical labeled incidents.Edge cases: missing data, periodic maintenance windows (suppress via calendar), multi-causal events (show composite causes).

MediumTechnical

41 practiced

Design an alerting policy that decides when to page on-call versus create a ticket. Include severity levels, required conditions to trigger paging (duration, evidence like SLO breach), deduplication rules, suppression during maintenance windows, and escalation paths when no acknowledgement occurs.

Sample Answer

Requirements:- Minimize pages to on-call for issues that require immediate human action (data loss, pipeline stall, SLO breach causing customer impact). Create tickets for non-urgent work (failures auto-retry, transient errors, low-severity data quality).- Support multi-tier severity, dedupe noisy signals, suppress during planned maintenance, and escalate when no ack.

Severity levels & routing:- Sev P0 (Page, immediate): Complete pipeline failure (no progress for >10 min), production data loss/corruption confirmed, SLO breach >5% users for 5m. Page primary on-call, ring phone+SMS.- Sev P1 (Page, urgent): Partial pipeline stall affecting critical tables or downstream jobs failing repeatedly (>=3 failures in 15m) or SLO degradation 2–5% for 10m. Page primary (SMS), notify Slack channel.- Sev P2 (Ticket, monitor): Single job failure with automatic retry not exhausted, or non-critical data quality alerts. Create ticket assigned to owning team with 24–72h SLA.- Sev P3 (Ticket, backlog): Low-impact anomalies, suggestions, non-urgent improvements.

Trigger conditions (examples):- Duration windows: only page if condition persists beyond threshold (P0: 10m steady-state; P1: 15–30m or repeated failures). This avoids paging on transient flakiness.- Evidence: require two signals for paging where applicable — e.g., SLO breach plus pipeline no-progress OR datastore write errors + consumer lag > X. For cardinal metrics, use rolling 5m and 1h windows to confirm trend.- Auto-suppression: if automated remediation (retry or restart) succeeds, auto-resolve and do not page.

Deduplication & grouping:- Group alerts by pipeline/job ID and root cause fingerprint (error signature + task). Suppress duplicates for a sliding window (default 30m). If same fingerprint reappears after remediation, re-evaluate severity.

Maintenance windows:- Planned maintenance must be declared; during window, suppress paging for expected failures and instead create tickets/notifications. If unexpected failure occurs causing customer-impacting SLO breach during maintenance, ignore suppression and page (safety override).

Escalation path:- Primary on-call receives page. If no ack in 5 min for P0 / 15 min for P1, escalate to secondary; after another 10 min escalate to team lead and incident commander and open a paging loop to SRE. All paging events create an incident record with context and runbook link.

Acknowledgement & runbooks:- Each alert links to owner, runbook, rollback steps, and mitigation checklist. On ack, on-call must update incident status and ETA.

Post-incident:- Require blameless postmortem for all P0/P1 within 72 hours, actions tracked as tickets.

Metrics to track policy effectiveness:- Mean time to acknowledge (MTTA), mean time to resolve (MTTR), number of false pages, pages suppressed by maintenance, and postmortem count.

Rationale:- Thresholded durations + multi-signal evidence reduce noisy paging; clear severity routing balances urgency vs engineer fatigue; dedupe and maintenance suppression reduce distractions; escalation ensures reliability when no ack.

HardSystem Design

29 practiced

Design alert routing logic that maps alerts to pipeline owners using ownership metadata, prioritizes routing by severity, deduplicates alerts originating from the same root cause, supports escalation policies, and handles failure modes when the alerting system or notification channel is down. Include an outline of the metadata model and routing rules.

Sample Answer

Requirements & constraints:- Map alerts to pipeline owners using ownership metadata (team, primary/secondary, on-call schedule)- Prioritize routing by severity (P0/P1/P2...), support escalation timelines- Deduplicate alerts from same root cause (correlation by fingerprint)- Support retries/fallbacks when alerting system or channels fail- Low latency for critical alerts; auditability and metrics

High-level architecture:- Alert Ingest (Kafka/Cloud PubSub) → Normalizer & Enricher → Correlator/Deduper → Router → Notifier(s) + Escalation Service + Audit Store

Metadata model (example fields):- pipeline_id, pipeline_name- owner_team, owner_user_id, owner_role (primary/secondary)- oncall_schedule_id, contact_methods: [{type: pager, priority}, {type: email, priority}]- severity_mapping_rules: mapping of alert_type → default_severity- dedupe_key_template: e.g., "{pipeline_id}:{error_signature}"- escalation_policy_id

Routing rules:1. Enrich incoming alert with pipeline_id via asset catalog lookup (match by namespace, job name, cluster tags).2. Determine severity: explicit alert.severity OR severity_mapping_rules.3. Compute dedupe_key using dedupe_key_template; Correlator groups alerts sharing key within time-window T (configurable per severity).4. For each deduped group, pick highest severity and generate single incident with aggregated context.5. Lookup owner(s) by pipeline metadata; if primary on-call unavailable (heartbeat/no-ack), route to secondary.6. Apply escalation_policy: e.g., P0: notify primary immediately via pager; if no ack in 5m escalate to secondary and on-call manager; after 15m create incident ticket.7. Rate-limit/aggregation: collapse repeated identical alerts into count + last_seen.

Deduplication & root-cause correlation:- Use deterministic fingerprint: hash(pipeline_id, error_signature, stack_trace_hash, root_cause_tag)- Use streaming windowed state (e.g., Flink/Spark Structured Streaming) to maintain active incidents and attach new alerts to existing incident if within TTL and matching fingerprint or correlated via causal graph (job dependency edges).

Failure modes & mitigations:- Notifier down: Router writes to durable queue + fallback channels (SMS/email) and marks incident as degraded; alerting system retries with exponential backoff and switches channel priority.- Alerting system outage: enqueue to persistent storage (S3/KV) and run background replayer; send health alert to SRE channel via alternative provider.- Owner metadata lookup failure: fall back to cached metadata snapshot; if missing, route to oncall-ops alias.- Correlator state lost: recompute fingerprints and rehydrate from recent alerts in storage; mark reconciled incidents and suppress duplicates.

Observability & Ops:- Metrics: routing latency, ack times, escalation counts, dedupe rate, delivery failures.- Audit logs for each routing decision; ability to override routing rules via admin UI.- Test harness for simulated alerts and chaos-testing notifier failures.

Trade-offs:- Strong dedupe reduces noise but risks hiding related regressions — keep aggregation visible and provide granular drill-down.- Aggressive escalation reduces MTTR but increases pager fatigue — tune per severity and team.

This design balances low-latency routing for critical data pipelines with robustness (dedupe, retries, fallbacks) and operational visibility.

Unlock Full Question Bank

Get access to hundreds of Data Pipeline Monitoring and Observability interview questions and detailed answers.

Join thousands of developers preparing for their dream job.