Designing, operating, and optimizing reliable data pipelines and ensuring data quality across ingestion, transformation, and consumption. Covers extract transform load and extract load transform patterns, efficient incremental and batch loading, idempotent processing, change data capture, orchestration and scheduling, and performance tuning to meet service level objectives. Includes data validation strategies such as schema enforcement, null and type checks, range and referential integrity checks, deduplication, handling late arriving and out of order data, reconciliation processes, and data profiling and remediation. Emphasizes observability, monitoring, alerting, and root cause analysis for data quality incidents, as well as data lineage tracking, metadata management, clear ownership and process discipline, testing and deployment practices, and governance to maintain data integrity for analytics and business operations. Also covers data integration concerns across customer relationship management systems, marketing automation systems, reporting systems, and other operational systems, including pipeline error handling, data contracts, and how test and validation checks can be integrated into pipelines to prevent regressions.
EasyTechnical
33 practiced
You are responsible for a training-data freshness SLO that requires data to be available within 6 hours of source event arrival. List and explain the factors that influence meeting that SLO (ingest latency, transformation time, job queueing, downstream bottlenecks), describe checks and monitoring to enforce the SLO, and outline an operational playbook to follow if the SLO is violated.
Sample Answer
Factors that influence meeting a 6‑hour data‑freshness SLO- Ingest latency: time from source event to landing (producer throughput, network, connectors like Kafka/CDC, batching). High batch intervals or backpressure increase delay.- Transformation time: ETL/ELT compute time (Spark/Beam/DBT), complexity of joins, resource sizing, and data skew.- Job queueing & scheduling: scheduler (Airflow/Kubernetes) delays, concurrency limits, backlogged DAG runs, retries, and task dependencies.- Downstream bottlenecks: storage write throughput (S3/GCS/warehouse), schema migrations, model training job startup, or feature-store ingestion rate.- Upstream variability & errors: late-arriving or malformed events causing retries and backfills.Checks and monitoring to enforce SLO- End-to-end latency metric: event_timestamp → availability_timestamp per partition, tracked with alert thresholds (e.g., >4h warn, >6h critical).- Component-level metrics: ingest lag (Kafka consumer lag), connector errors, task durations, queue length, executor utilization, storage write latency.- Synthetic tests: periodic test events injected and traced through the pipeline.- SLIs + dashboards: Grafana dashboards, anomaly detection, and on-call alerts integrated to pager/Slack.- Health checks: per-task success/failure rates, schema validation, and drift detection.Operational playbook when SLO violated1. Triage: check end‑to‑end metric, identify affected time window and partitions.2. Component check: inspect ingest (consumer lag), scheduler (backlog), and compute (failed/slow tasks) logs.3. Immediate mitigations: - Increase parallelism/resources (scale Spark executors/K8s pods). - Temporarily increase concurrency in Airflow or run urgent ad‑hoc job for missing window. - Disable noncritical jobs to free resources.4. Recovery: run targeted backfill to bring training data up to date; validate with synthetic checks.5. Root cause & postmortem: record timeline, RCA, and permanent fixes (tune batch interval, autoscaling, retry policies, alert thresholds).6. Preventive actions: add more guardrails (circuit breakers, feature‑store buffering), improve observability, and run regular load tests.
MediumTechnical
24 practiced
Explain delivery semantics in streaming systems: at-most-once, at-least-once, and exactly-once. For each, describe how failures can surface in ML feature pipelines and outline concrete implementation techniques to achieve exactly-once semantics when reading from Kafka and writing to a stateful sink (for example, Spark Structured Streaming transactional sinks or Kafka transactions).
Sample Answer
At-most-once, at-least-once, and exactly-once define how many times events are delivered to consumers in the presence of failures.Definitions & failure symptoms in ML feature pipelines- At-most-once: Events are never retried; some messages may be lost. In an ML pipeline this can mean missing feature updates—models see stale or incomplete feature values, causing degraded accuracy or bias without obvious errors.- At-least-once: Messages are retried until acknowledged; duplicates are possible. For features this can produce duplicate updates (double-counting aggregated features, inflated counters, or training data leakage), biasing features and downstream model metrics.- Exactly-once: Each event affects state exactly once despite retries/failures. This is ideal for correct aggregations, incremental feature computation, and training/serving parity.How failures surface- Producer crash after send but before commit → duplicates or loss depending on semantics.- Consumer crash after processing but before checkpoint/commit → at-least-once causes reprocessing (duplicates), at-most-once can lose processing.- Network partitions and out-of-order retries can cause inconsistent feature state or partially applied updates.Techniques to achieve exactly-once with Kafka + stateful sink- Idempotent writes + transactional boundaries: - Use Kafka producer idempotence and transactions: produce and commit offsets in the same Kafka transaction (consume-transform-produce pattern). This ensures either the processing and offset commit both succeed or neither do. - In Spark Structured Streaming, use the built-in Kafka integration with "readKafka" + "writeStream" and the transactional sink (e.g., Kafka sink with txnIdPrefix) or checkpointing. Spark can atomically write output and commit progress via its checkpoint/commit protocol when the sink supports atomic commits.- Atomic sink semantics: - Use sinks that support atomic/transactional commits (e.g., transactional Kafka sink, databases with upsert + transactions, or storage with atomic renames). Spark’s transactional sinks implement a two-phase commit: write staging files, then atomically commit metadata so partial failures don’t expose partial results.- Exactly-once with external state: - Maintain state inside the streaming engine (e.g., Spark state store) that is checkpointed; combine with idempotent or transactional writes externally. For external stores that lack transactions, use deduplication with unique event IDs (watermark + dedupe state) so replays are ignored.- Practical recipe: - Enable Kafka producer idempotence and transactions. - In consumer (Spark), process records, write outputs to transactional sink, and use the engine’s ability to commit Kafka offsets only after the sink commit (or use Kafka transactions to include offsets). Alternatively, use Spark Structured Streaming’s “exactly-once” guarantees by configuring checkpointing and using supported sinks (Kafka transactional sink, Delta Lake/ACID stores) and ensure deterministic processing and stable event IDs for dedupe.- Operational considerations: - Monitor end-to-end latency and transactional throughput. - Handle long-running transactions carefully (keep them short). - Test failure scenarios (consumer/driver/executor crashes, network partitions) and validate no duplicates or losses with synthetic workloads.Takeaway: exactly-once requires coordination between source offset commit and sink commit—use Kafka transactions or engine-native two-phase commit sinks, idempotent operations, and deterministic processing to ensure ML features remain correct and reproducible.
EasyBehavioral
35 practiced
Tell me about a time you discovered a data quality issue that would have impacted model predictions. Use the STAR framework (Situation, Task, Action, Result). Explain how you detected the issue (metrics, tests, spot checks), how you communicated to stakeholders, how you remediated the root cause, and what long-term guardrails you implemented to avoid recurrence.
Sample Answer
Situation: At my previous company I owned a fraud-risk model that scored transactions in real time. Shortly after a model refresh, we saw a sudden 12% drop in true positive rate (TPR) and a rise in false positives compared to the prior week.Task: I needed to quickly determine whether this was model drift or a data quality issue, mitigate impact on production scoring, and put measures in place to prevent recurrence.Action:- Detection: I compared feature distributions (KS test and population stability index) and per-feature missingness between offline training and live inference logs. A specific categorical feature ("merchant_category") showed a jump in nulls from 0.5% to 18% and an unexpected new token prefix. I also ran spot checks on raw event payloads and histogrammed downstream embeddings — discrepancies matched the feature change.- Root cause analysis: Traced ETL logs and found a recent upstream change in the event schema that renamed a nested field; our parser silently filled missing values instead of throwing errors.- Remediation: Rolled back the parser change in a canary environment, patched the ETL to handle both schemas, and reprocessed the affected day's data to retrain and re-score where necessary.- Communication: Notified stakeholders (product, data engineering, ops) within an hour with a short incident summary, impact assessment, and remediation plan; provided daily status updates until resolved.Result: TPR returned to baseline within 48 hours and false positives normalized. The incident avoided customer-impacting declines and we regained stakeholder confidence.Long-term guardrails implemented:- Schema validation with strict Avro/JSON schema enforcement and CI checks for backward-incompatible schema changes- Automated data-quality monitors (missingness, cardinality drift, PSI thresholds) that trigger alerts and pause model refresh pipelines- End-to-end integration tests in CI that include sample synthetic payloads for known schema variants- Runbook and on-call rotation for rapid investigation of production score degradationThis experience reinforced validating feature pipelines as rigorously as model code and the importance of clear, timely stakeholder communication.
EasyTechnical
34 practiced
What is data profiling for ML pipelines, and which metrics would you collect per column to evaluate dataset readiness? List at least six metrics (for example: null rate, cardinality, distinct count, min/max, mean/stddev, percentiles, sample values), explain how to compute them efficiently at scale, and describe how profiling results inform feature selection and preprocessing.
Sample Answer
Data profiling for ML pipelines is the systematic measurement of column-level statistics and data quality signals to judge dataset readiness, detect issues (missingness, skew, outliers, leakage) and guide feature engineering and preprocessing.Key per-column metrics (recommended >=6):- Null / missing rate — fraction of rows with NULL/NaN.- Distinct count & cardinality — number of unique values and cardinality bucket (low/medium/high).- Mode / top-k sample values — most frequent values and their frequencies.- Mean, stddev — central tendency and dispersion for numeric columns.- Min, max, percentiles (e.g., 1st, 25th, 50th, 75th, 99th) — range and tail behavior.- Skewness & kurtosis — asymmetry and tail-heaviness to detect non-normality.- Sparse density / zeros proportion — for sparse features.- Type/format consistency and regex error rate — for categorical/text/date columns.- Cardinality temporality (new uniques over time) — for drift/leakage detection.Efficient computation at scale:- Single-pass aggregations: compute counts, sums, min/max, nulls in one scan (map-reduce).- Approximate algorithms: HyperLogLog for distinct counts, t-digest or Q-digest for quantiles, Sketches (Count-Min) for heavy hitters — trade small error for O(1)/O(log n) memory.- Streaming statistics: Welford’s online algorithm for mean/stddev, online skew/kurtosis.- Partition-wise profiling: compute per-shard aggregates and merge (associative reducers).- Sampling + stratified sampling for expensive operations (e.g., top-k samples or regex checks) with confidence bounds.- Use distributed frameworks (Spark, Flink) and columnar formats (Parquet) to minimize IO.How profiling informs feature selection and preprocessing:- Drop or impute: high null rate suggests drop or imputation strategy; choose mean/median/mode or model-based imputation.- Encode strategy: low cardinality → one-hot; high cardinality → target encoding, hashing, or embedding.- Scaling/transform: high skew/kurtosis → log/box-cox or quantile transform; normalize based on mean/stddev.- Outlier handling: extreme min/max/percentiles → winsorize or robust scaling.- Reduce leakage: detect columns with near-perfect correlation to label or temporally new uniques — exclude or restructure.- Feature interactions: correlated columns (from profiling covariance) suggest dimensionality reduction (PCA) or drop redundancy.- Monitoring & drift: track profiling metrics over time (cardinality growth, mean shifts) to trigger retraining or alerts.Profiling should be automated in CI/CD for datasets, stored as artifacts, and used to drive deterministic preprocessing pipelines and feature tests.
HardTechnical
28 practiced
You maintain a Delta Lake used by both analytics and ML training where producers may add or drop fields. Propose a strategy combining schema registry, backward/forward compatibility policies, contract testing, migration steps, and operational procedures that enable safe schema evolution with minimal downtime and clear rollback paths.
Sample Answer
Situation: We run Delta Lake tables consumed by analytics and ML pipelines. Producers sometimes add/drop fields; we need safe evolution with minimal downtime and clear rollback.Strategy (high-level):- Use a centralized schema registry (Avro/Protobuf/JSON Schema) as the canonical contract; store schema versions and compatibility rules. Link each Delta table to a schema id + consumer feature spec.- Enforce compatibility policy: for ML, prefer backward-compatible adds (adding optional fields) and deprecate-by-version for drops. Define compatibility matrix: backward (consumers read newer producer data) required for online training; forward allowed only with controlled migration windows.Concrete components:1. Schema registry & policies- Tooling: Confluent Schema Registry or in-house service exposing REST + TTL.- Rules: "Add optional field" = allowed; "Change type" = forbidden without explicit migration; "Drop field" = only after deprecation period and consumer signoff.2. Contract & contract-testing pipeline- CI contract tests that validate: - Producer schema evolves according to rules relative to registry. - Consumers’ read schemas (expected features) are compatible with new producer schema.- Implement consumer-driven contract tests: run a lightweight Spark job that reads new schema via registry and asserts all required features exist (names/types), default behaviors for missing optional features, and type-castability.- Run property-based tests that serialize/deserialize example records.3. Migration plan (zero-downtime pattern)- Phase 0: Announce change + create migration ticket + owners.- Phase 1 (Dual-write / shadow writes): Producers write both old and new schema (or include new fields nullable) for N days. Consumers continue reading old schema.- Phase 2 (Canary readers & training): Deploy a canary training job that reads new schema and verifies model inputs, feature distributions, and training metrics vs baseline.- Phase 3 (Gradual roll-forward): Switch low-risk consumers to new schema; run contract tests and cached-model A/B evaluations.- Phase 4 (Deprecation & drop): After all consumers have migrated and no regressions for the deprecation window, perform an explicit Delta compaction/migration job to remove dropped columns (e.g., spark SELECT with explicit schema, write to a new table with new schema), then atomic swap (Delta table replace/rename or transactional MERGE), and update registry.Delta-specific ops- Use Delta time travel for rollback: keep delta log retention long enough during migration (set spark.databricks.delta.retentionDuration to cover migration window). Before destructive migration, create a snapshot/export of current table (copy to backup path).- To remove columns safely, write a new Delta table with desired schema using overwrite with "replaceWhere" false and then use atomic rename to swap. Alternatively use OPTIMIZE/REWRITE MANIFEST after schema change.- Use table properties to lock schema evolution (delta.enableChangeDataFeed = true, delta.appendOnly = false as required).Monitoring & validation- Schema alarms: registry webhook triggers when schema changes.- Runtime validation: Spark job validates each batch against expected schema; log and metric counts of missing/extra fields.- Model-check metrics: feature distribution checks, gradients of loss, validation metrics; baseline thresholds trigger rollback.- End-to-end smoke tests: from data landing → feature store → training → model evaluation.Rollback plan- If regression: revert producers to previous schema or switch consumers to read previous snapshot via Delta time-travel (VERSION AS OF or TIMESTAMP AS OF) or point-in-time read. Restore backup table if swapped.- Revert registry version to previous schema and redeploy consumer parsing logic.Operational practices- Maintain runbook with owners, migration windows, test checklist.- Require schema-change RFC with impact analysis (which models use fields, feature importance).- Automate as much as possible: pre-merge CI gating schema changes, automated canary training, and dashboards for schema & model health.- Limit scope: prefer additive changes and explicit deprecation processes to minimize churn.This combination provides enforceable contracts, automated verification, low-risk phased rollout, Delta-native rollback using time travel and backups, and operational guardrails so ML pipelines remain stable while allowing safe evolution.
Unlock Full Question Bank
Get access to hundreds of Data Pipeline and Data Quality interview questions and detailed answers.
