Design end to end data pipeline solutions from problem statement through implementation and operations, integrating ingestion transformation storage serving and consumption layers. Topics include source selection and connectors, ingestion patterns including batch streaming and micro batch, transformation steps such as cleaning enrichment aggregation and filtering, and loading targets such as analytic databases data warehouses data lakes or operational stores. Cover architecture patterns and trade offs including lambda kappa and micro batch, delivery semantics and fault tolerance, partitioning and scaling strategies, schema evolution and data modeling for analytic and operational consumers, and choices driven by freshness latency throughput cost and operational complexity. Operational concerns include orchestration and scheduling, reliability considerations such as error handling retries idempotence and backpressure, monitoring and alerting, deployment and runbook planning, and how components work together as a coherent maintainable system. Interview focus is on turning requirements into concrete architectures, technology selection, and trade off reasoning.
MediumSystem Design
52 practiced
Design a CI/CD pipeline for ML data pipelines and feature engineering code. Include stages for data validation/unit tests, integration tests with sample datasets, model training triggers, reproducible artifacts (container images, dataset manifests), deployment to feature store and model registry, canarying, and automated rollback. What tools and checkpoints do you recommend?
Sample Answer
Requirements & constraints:- Verify data and feature-engineering code correctness before training- Reproducible artifacts (containers, dataset manifests, hashes)- Automate model training when data/code change- Deploy features to feature store and models to registry- Canary and automated rollback based on metricsHigh-level pipeline stages:1. Commit & PR - Tools: Git + GitHub/GitLab - Checkpoints: Linting, unit tests for featurizers (pytest), type checks (mypy), pre-commit hooks2. Data validation & small-scale integration tests (CI) - Tools: GitHub Actions / GitLab CI / Jenkins - Data checks: great_expectations or pandera run against sampled datasets checked into test-data or generated fixtures - Tests: unit tests for transforms, schema contract tests, statistical/regression checks (schema drift, nulls, ranges) - Fail fast on schema breaks or test regressions3. Build reproducible artifacts - Containerize code with Docker (BuildKit), pin base images & dependency hashes, build image → push to registry (ECR/GCR/ACR) - Produce dataset manifests: content-addressed manifests (JSON+SHA256) pointing to versioned data in object store (S3/GCS) or DVC/Pachyderm commits - Record artifact metadata (image digest, data manifest, code commit SHA, random seeds, env) in ML metadata store (MLflow/Conda-lock + provenance) - Tools: DVC / LakeFS / Pachyderm for data versioning; MLflow Tracking or Metaflow for runs metadata4. Integration tests & training trigger - Run integration pipeline on representative sample dataset in an isolated environment (Kubernetes namespace or ephemeral VM) via Argo Workflows / Kubeflow Pipelines / Airflow - Validate end-to-end feature pipeline and a short train run (smoke training) producing checkpoints/artifacts - If passes, trigger full training job (batch on GPU cluster or cloud training service) using same image + dataset manifest (ensures reproducibility)5. Register artifacts & lineage - Register trained model in MLflow Model Registry (or Sagemaker Model Registry) including model signature, metrics, artifact links, data manifest, and feature store version - Publish features/feature definitions to Feast (or internal feature store) with version tags and lineage info - Store provenance in metadata store (MLMD, MLflow), and persist artifacts to artifact store (S3, Artifactory)6. Deployment & canarying - Deploy model to serving (Seldon Core / KFServing / BentoML) using container image + model artifact - Use service mesh (Istio) + progressive delivery tool (Flagger) to perform canary rollouts: route small % traffic to new model, compare key metrics (latency, error rate, business metric) - Metrics/observability: Prometheus + Grafana + application logs + structured request/response tracing (OpenTelemetry)7. Automated evaluation & rollback - Define SLOs and automated checks (e.g., model AUC, calibration, request error rate, downstream business KPIs) - Flagger/Argo Rollouts monitors metrics and automatically promote or rollback based on thresholds and statistical tests (e.g., sequential hypothesis testing) - If rollback triggered, Flagger re-routes traffic back to stable model and pipeline creates alert & incident ticket8. Post-deploy monitoring & drift detection - Continuous data drift detection (Evidently, WhyLabs) and model performance monitoring (Prometheus exporters + MLflow metrics) - Retrain triggering rules: data drift beyond threshold or performance degradation → create retrain job or open PR for human reviewRecommended checkpoints & governance:- Mandatory data validation (great_expectations) and schema gating in CI- Signed artifact manifests (image digest + data manifest + commit SHA) stored in registry and logged into ML metadata- Reproducible training: use the same container image, data manifest, random seeds, and config stored as immutable run input- Access control and approval gates for production promotion (manual approval optional for high-risk models)- Automated canary with statistical significance checks and automatic rollback via Flagger/ArgoWhy these tools & choices:- Reproducibility: containers + data versioning (DVC/LakeFS) + MLflow metadata ensure exact re-runs- Orchestration: Argo/Kubeflow provide Kubernetes-native pipelines for isolation and scale- Feature serving: Feast separates feature engineering lifecycle from model serving, enabling consistent feature values between training and inference- Canary & rollback: Istio + Flagger enable automated, metric-driven progressive delivery with safe rollbackExample minimal stack:- GitHub + Actions → Docker image (GCR) + DVC data manifests → Kubeflow/Argo pipeline run (integration) → Full training on Kubernetes (TF/PyTorch) → MLflow Model Registry → Feast feature store → Seldon + Istio + Flagger for canary → Prometheus/Grafana monitoring.Edge considerations:- Secure credentials for data/artifact stores in CI via vaults/secret manager- Costs for full training runs — use sampled smoke runs in CI- Compliance: data lineage and audit logs required for regulated modelsThis design ensures automated, reproducible ML CI/CD with safety checkpoints, observable canary rollouts, and reliable rollback mechanisms.
MediumTechnical
65 practiced
How do you design monitoring and alerting for data pipelines that serve production ML models? List key metrics (pipeline lag, processing errors, feature distribution changes, cardinality, schema violations), sensible alert thresholds, escalation policies, and strategies to avoid alert fatigue while ensuring timely detection of impactful issues.
Sample Answer
Approach: Treat ML data pipelines as part of the product SLOs — define what “healthy” means for latency, correctness, and model inputs; instrument observability; alert on actionable breaches with clear escalation.Key metrics and suggested thresholds- Ingestion / pipeline lag: time since event ingestion or end-to-end feature freshness. Alert: > SLA (e.g., 5 min) for high‑throughput online; > 1 hour for batch. Severity tiers: warn at 50% of SLA, critical at 100%+.- Processing errors / failure rate: percent failed jobs or records. Alert: warn at 0.5% failure rate, critical at 2% or sustained failures (>3 consecutive runs).- Feature distribution drift: alert on statistically significant drift (KS test or PSI). Alert: PSI > 0.1 (investigate), PSI > 0.25 (critical).- Cardinality changes: sudden spike/drop in unique keys (users, IDs). Alert: ±50% change vs rolling 7‑day median or absolute threshold if known.- Schema violations: any new/missing fields or type changes. Alert: immediate critical for required features.- NULL / missing rate per feature: warn at >1–5% depending on feature importance; critical at >10%.- Downstream model performance (if available): prediction rate drop, latency increase, or labeled metric degradation. Alert thresholds aligned to SLOs (e.g., >5% drop in AUC or >10% latency increase).Escalation policy- Triage channel: PagerDuty critical, Slack/Teams for warnings. Critical → on-call ML engineer + data infra owner + product owner within 15 minutes. Warning → ticket to on-call with 2-hour SLA.- Runbooks: link alerts to playbooks with quick checks and rollback/mitigation steps (e.g., pause feature source, switch to fallback features, retrain flag).Avoiding alert fatigue- Multi-condition alerts: require combined signals (e.g., lag + error spikes) before paging.- Rate-limit and deduplicate alerts per incident key.- Use severity tiers and non-paging dashboards for low-priority anomalies.- Suppression windows for planned maintenance and known data-source holidays.- Auto-silence transient blips: only escalate on sustained breaches (e.g., 3 consecutive runs or 10 minutes continuous).- Regular review: monthly alert audit to tune thresholds and remove noisy alerts.Additional strategies- Canary pipelines and shadowing: validate changes on a small subset before full rollout.- Synthetic test events: ensure end-to-end pipeline health continuously.- Data contracts and schema registry with automated gating on incompatible changes.This design balances timely detection (immediate for schema/critical failures) with noise reduction via tiering, multi-signal detection, and operational runbooks.
HardSystem Design
63 practiced
Architect a globally-distributed data pipeline to support real-time ML inference across three regions with active-active traffic. Requirements: <100ms read latency for feature queries, eventual consistency across regions for feature stores, tolerance for a single region failure, and GDPR constraints for regional data residency. Describe components, replication strategy, conflict resolution, data routing, and deployment approach.
Sample Answer
Requirements clarified:- 3 regions active-active, <100ms read latency for feature queries, eventual cross-region consistency, tolerate one-region failure, GDPR: regional data residency (some features can't leave region).High-level architecture:- Per-region stack: Feature Store (online and offline), Model Serving, Inference Gateway, Redis/ROCKSDB online store, CDC pipeline, Control plane for metadata.- Global components: Global Router (edge-aware), Replication broker (async), Monitoring & config registry.Components & responsibilities:- Online Feature Store (per-region): low-latency KV store (Redis Cluster + local SSD-backed RocksDB for cold hot tier). Exposes gRPC/HTTP API for feature lookups.- Offline Store: region-local data lake (S3/GCS) for batch training.- CDC/Stream layer: Kafka (regional clusters) + MirrorMaker2 or Confluent Replicator for async cross-region replication of non-resident feature updates and metadata.- Replication broker: applies policies to prevent GDPR-bound data from leaving origin region.- Model Serving: per-region replicas behind local LB; models pulled from global model registry but weights stored per-region.- Global Router: DNS + geo-routing (EDNS + Anycast) routes inference traffic to nearest healthy region; client SDK supports fallback and multi-get for features.Replication strategy & consistency:- Primary-write-per-region for resident data; async multi-master for non-resident or derived features.- Use per-feature metadata marking: residency=regional/global and conflict-policy=last-write-wins (LWW)/CRDT/merge function.- For eventual consistency: use Kafka with per-key log ordering and vector clocks for causal ordering where needed. For counters/cumulative features use CRDTs (PN-Counters or LWW-Registers).- Conflict resolution: deterministic merge functions: - numeric aggregates: CRDT (G-Counter, PN-Counter) - latest behavioral features: LWW with synchronized clocks via NTP+hybrid logical clocks (HLC) - complex objects: application-level merge hooks in replication broker.Data routing & GDPR:- Feature metadata determines routing: if residency=region A only, reads for user scoped to A must go to A. Global Router and client SDK ensure requests for EU users stay within EU regions.- For cross-region inference that needs resident features: either ship anonymized derived features (stateless transformations) or perform remote feature lookup with consent and audit logs. Prefer model sharding per-region to avoid cross-border calls.- Encryption at rest and in transit; region-restricted keys via KMS with regional key policies.Failure tolerance & HA:- Active-active: all regions serve traffic; state replicated async. If a region fails, Global Router fails over to remaining regions. Client SDK supports degraded mode (serve with best-effort features, mark missing features).- Stateful stores backed by cross-region backups and automated restore playbooks. Kafka clusters use replication.factor >=3 per region; MirrorMaker handles re-sync on recovery.Deployment & operations:- Kubernetes per-region clusters for services (feature store API, model servers, Kafka Connect, replication broker). Use Helm and GitOps (ArgoCD) for consistent manifests.- CI/CD: model build -> container image -> canary deploy across regions; traffic split by weights for A/B.- Observability: global tracing (sampled), Prometheus federated metrics, per-region SLOs for <100ms tail latency; synthetic tests simulate GDPR routing and failover.- Security & compliance: automated policy checks in pipeline to enforce residency; audit logs forwarded to region-local compliance store.Trade-offs:- Async replication reduces write latency but allows short-term divergence — acceptable per eventual consistency requirement.- CRDTs add complexity but give conflict-free convergence for aggregates.- Keeping resident data per-region increases complexity for global models; mitigated by per-region model training/transfer learning.This architecture meets <100ms read by local online stores, tolerates single-region failure via active-active routing and async replication, preserves GDPR residency via metadata and KMS policies, and ensures deterministic conflict resolution via CRDTs/HLC/LWW as appropriate.
EasyTechnical
55 practiced
You're designing a pipeline to ingest logs from 1,000 web servers into a central store for model training. What connector and ingestion options would you evaluate (e.g., Kafka, Kinesis, Filebeat, Fluentd, S3 collectors), and what factors (throughput, ordering, ease of operations, vendor lock-in, cloud integration) should drive the choice?
Sample Answer
Situation: We need a reliable, scalable pipeline to ingest logs from 1,000 web servers into a central store for ML training.Connector/ingestion options to evaluate (with short pros/cons):- Kafka (self-managed or MSK): high throughput, low latency, strong ordering per partition, durable, good ecosystem (Connectors). Ops overhead if self-hosted; minor vendor lock-in.- Kinesis (AWS): managed, similar streaming semantics, easier ops on AWS, limits on shard throughput and stricter ordering per shard.- Filebeat/Fluentd (agents on hosts): lightweight, flexible routing, can ship directly to Kafka/Kinesis/S3/Elasticsearch; easy to deploy with config management; less durable if network drops unless buffered.- S3 collectors (batch): push logs to S3 (or GCS) then run batch ingestion (Glue, Dataflow). Cheap, durable, easy for large-volume ML training, but higher latency and weaker ordering.- Managed streaming services (Confluent Cloud, Pub/Sub): reduce ops, good integrations, may cost more.Factors driving the choice:- Throughput: Kafka/Kinesis for high continuous throughput; S3 for bursty/batch.- Ordering: Kafka/Kinesis guarantee ordering per partition/shard; S3 loses strict ordering.- Durability & exactly-once: Kafka (with transactions) and Kinesis + careful consumer design; S3 is durable but needs de-duplication.- Ease of operations: Managed services or S3 win; Filebeat/Fluentd require host management.- Cloud integration & cost: Prefer native services (Kinesis, S3) if on AWS to simplify auth, IAM, and cost predictability.- Latency needs: Real-time features/models → streaming (Kafka/Kinesis); offline training → S3 batch is fine.- Vendor lock-in & portability: Kafka + open-source connectors offer portability; managed services increase lock-in.Recommendation (ML-focused):If you need near‑real-time feature updates or streaming training pipelines: use Kafka (or Confluent Cloud/MSK) + Filebeat/Fluentd agents on servers to buffer and forward. For primarily offline model training with large historical data, write logs to S3 via agents and run batch ETL. Combine: stream critical events to Kafka and archive all logs to S3 for reproducibility.
EasyTechnical
64 practiced
Define 'exactly-once', 'at-least-once', and 'at-most-once' delivery semantics in distributed data pipelines. Give a concrete example of why exactly-once is difficult to achieve end-to-end and describe one practical mitigation an ML feature pipeline can use to avoid duplicate training examples.
Sample Answer
Exactly-once, at-least-once, and at-most-once describe how many times a message/event is delivered to the consumer in the face of failures:- At-most-once: Each event is delivered zero or one time. The system may drop messages to avoid duplicates (no retries). Good for non-critical telemetry where loss is acceptable.- At-least-once: Each event is delivered one or more times. Retries ensure no data loss, but consumers may see duplicates and must be idempotent.- Exactly-once: Each event is processed one and only one time from producer to final sink — no loss, no duplicates. This is the ideal but hardest to guarantee across distributed components.Why exactly-once is difficult end-to-end:Different systems (producers, message brokers, processing engines, storage) have different failure semantics and isolation levels. Achieving true exactly-once requires atomic, coordinated commit across these systems (distributed transactions), which is expensive, complex, and often unsupported. Network partitions, retries, and non-idempotent sinks (e.g., appending rows without keys) make it hard to know whether an operation already succeeded.Practical mitigation for an ML feature pipeline:Use idempotent writes keyed by a deterministic unique event ID (e.g., user_id + event_timestamp + hash). On ingestion, deduplicate by upserting into a feature store or storing events in a dedupe table with the event ID as primary key. For training, materialize feature snapshots using watermarking and windowing so each raw event contributes once; combine this with consumer-side deduplication logic to avoid duplicate training examples without requiring full distributed transactions.
Unlock Full Question Bank
Get access to hundreds of Data Pipeline Architecture interview questions and detailed answers.
