Designing and operating data pipelines and feature platforms involves engineering reliable, scalable systems that convert raw data into production ready features and deliver those features to both training and inference environments. Candidates should be able to discuss batch and streaming ingestion architectures, distributed processing approaches using systems such as Apache Spark and streaming engines, and orchestration patterns using workflow engines. Core topics include schema management and evolution, data validation and data quality monitoring, handling event time semantics and operational challenges such as late arriving data and data skew, stateful stream processing, windowing and watermarking, and strategies for idempotent and fault tolerant processing. The role of feature stores and feature platforms includes feature definition management, feature versioning, point in time correctness, consistency between training and serving, online low latency feature retrieval, offline materialization and backfilling, and trade offs between real time and offline computation. Feature engineering strategies, detection and mitigation of distribution shift, dataset versioning, metadata and discoverability, governance and compliance, and lineage and reproducibility are important areas. For senior and staff level candidates, design considerations expand to multi tenant platform architecture, platform application programming interfaces and onboarding, access control, resource management and cost optimization, scaling and partitioning strategies, caching and hot key mitigation, monitoring and observability including service level objectives, testing and continuous integration and continuous delivery for data pipelines, and operational practices for supporting hundreds of models across teams.
EasyTechnical
26 practiced
Explain watermarking and windowing in stream processing. Define types of windows (tumbling, sliding, session) and show a short example use-case for each (for example: tumbling for hourly metrics, sliding for rolling aggregates, session for user activity).
Sample Answer
Watermarking and windowing are core concepts in stream processing to turn unbounded event streams into bounded computations.Watermarks:- A watermark is a timestamp marker the system uses to indicate “I have seen all events up to time T (likely).” It lets the engine trigger window computations while tolerating late data.- Types: event-time watermarks (driven by event timestamps), processing-time (system clock), and periodic vs. punctuated watermarks.- Trade-off: earlier watermark → lower latency but higher chance of dropping late events; later watermark → higher correctness for late arrivals but more latency/state.Windowing:- Windows group events into finite buckets based on time (usually event time).1) Tumbling windows- Definition: Fixed, non-overlapping intervals (e.g., [0:00–1:00), [1:00–2:00)).- Use-case: Hourly metrics (e.g., hourly pageviews per site). Simple aggregation, easy state management.2) Sliding windows- Definition: Fixed-size windows that advance by a smaller slide, causing overlap (e.g., size=10m, slide=1m → 10-minute windows every minute).- Use-case: Rolling aggregates like a 5-minute moving average of CPU usage to detect trends with fine granularity.3) Session windows- Definition: Dynamic windows that group events separated by less than a configurable gap duration; boundaries determined by user inactivity.- Use-case: User activity sessions on a website (e.g., group clicks into sessions with 30-minute inactivity gap) to compute session length, conversion rates.Practical notes:- Always prefer event-time windows with watermarks for correctness across out-of-order events.- Configure allowed lateness and side outputs for late events to avoid data loss.- Choose window type by business need: tumbling for periodic reports, sliding for continuous trend detection, session for user-behavior aggregation.
HardTechnical
24 practiced
Describe strategies for dataset versioning and lineage to enable reproducible experiments and deterministic backfills. Include concepts like immutable dataset snapshots, content-addressable identifiers (hashing), metadata stores, and how to link dataset versions to model artifacts in a model registry.
Sample Answer
Approach summary: treat datasets as immutable, content-addressed artifacts and record rich provenance in a metadata store so experiments and backfills are deterministic and traceable. Combine immutable storage (data lake tables or object store snapshots), cryptographic/content hashes, and a metadata/lineage service linked to the model registry.Concrete strategies:- Immutable snapshots: store processed datasets as append-only versions (e.g., Delta Lake/Apache Iceberg table versions, or snapshot folders in S3/GCS). Never overwrite—create versioned path or snapshot ID for every run.- Content-addressable IDs: compute stable identifiers (SHA-256) over canonicalized data (e.g., sorted row-level fingerprints or partition-level Parquet file checksums + schema hash). Use these hashes as dataset IDs so identical content yields identical ID.- Metadata & lineage store: centralize metadata (dataset ID, schema, partitions, record-count, min/max, provenance: upstream source IDs, transformation DAG, code commit, container image, parameters, random seeds, runtime) in a scalable metadata service (e.g., Data Catalog + custom lineage DB, OpenLineage/Marquez, or graph DB). Expose APIs to query versions and dependency graphs.- Link to model artifacts: when logging a model to a model registry (MLflow, Sagemaker, Vertex AI), persist dataset IDs and metadata as immutable model metadata fields and artifact links. Store: training_dataset_id, validation_dataset_id, preprocessing_code_commit, container_image, hyperparams. Also store snapshot URIs (Delta version or Iceberg snapshot).- Deterministic backfills & reproducibility: ensure pipelines are idempotent and parameterized by dataset ID and code commit. Capture runtime env (library versions, OS, container digest), random seeds, and fixed partitioning strategies. For backfills, fetch exact snapshot by dataset ID and re-run the same code artifact; verify checksums match.- Lineage graph & audit: maintain directed graph linking raw sources → transformations → dataset versions → model versions. Support queries like “which models used records from source S between dates X–Y?” and enable impact analysis.- Operational practices: automated checksum validation on ingest, immutability enforcement through policies (IAM, lifecycle), periodic tombstone retention policy, retention of snapshots for SLAs, and integration tests that rehydrate snapshot → reproduce metrics.- Tools & patterns: Delta/Iceberg for ACID snapshotting; object store + content hashes for small/partitioned datasets; OpenLineage/Marquez for lineage; MLflow/Model Registry for model linking; Terraform/Git for infra/code versioning; container image digests for runtime reproducibility.- Edge cases: schema evolution (store schema diffs and canonical schema mapping), very large datasets (use partition-level hashing and manifest files), incremental updates (use change-version IDs and deterministic merge logic), and privacy (redact PII and record data retention).Why this works: content-addressing ensures exact reproducibility, immutable snapshots prevent accidental drift, and a centralized lineage+metadata layer ties datasets to code and models so deterministic backfills and impact analysis are straightforward and auditable.
MediumSystem Design
27 practiced
Design metadata and discoverability for a multi-team feature catalog: support search, schema, lineage, ownership, feature tags, feature-quality scores, versioning, and access controls. Sketch data model for the catalog and explain how teams would onboard, discover, and reuse features safely.
Sample Answer
Requirements:- Functional: search by name, tag, schema, owner; view lineage, versions, quality scores; register/onboard features; enforce access controls.- Non-functional: low-latency search, auditability, scalable metadata store, RBAC, immutability for versions.High-level architecture:- UI + API layer → Metadata Service (CRUD, search) + Lineage Service → Metadata store (graph DB for lineage + relations) + Search index (Elasticsearch) + Object store for artifacts (Parquet, proto) + Authz service (IAM/ABAC) + Audit log.Core data model (simplified):Feature { id, name, description, owner_team, owners[], tags[], domain, schema: [{name,type,nullable}], primary_key[], lineage_inputs: [feature_id or dataset_id], versions: [{version_id, commit_hash, created_by, timestamp, change_log}], quality_scores: [{metric, value, calc_timestamp, window}], access_policies: {read: [teams/roles], write: [teams/roles], discoverable: bool}, status: enum{draft, certified, deprecated}, sample_query, doc_link, last_updated}Store relationships in a graph DB (Neo4j/DGraph) for efficient lineage traversal; index searchable fields in Elasticsearch.Onboarding flow:1. Feature dev fills a template or CLI (name, schema, owner, tags, sample SQL, inputs).2. CI pipeline validates schema and test coverage, computes baseline quality metrics, writes artifact (parquet) to object store and registers metadata + initial version.3. Automated lineage extractor parses DAGs (Airflow/Spark) to link inputs → feature nodes.4. Review step: owners certify; status set to “certified”.Discovery & reuse:- Search UI supports keyword, tag, schema, and similarity (schema matching). Result shows owner, quality scores, lineage graph, sample query, access policy, and versions.- Reuse template/SDK: when a consumer selects a feature, CLI can import a specific version; policy checked via Authz service.- Safety: only certified or permitted status can be used in production; CI blocks using deprecated features. Consumers required to pin to a version or use “latest-certified”.Feature-quality & governance:- Quality pipeline runs periodically to compute metrics (completeness, drift, freshness) stored as time-series; dashboard shows trends and raises alerts.- Trust score computed using weighted metrics + certification; exposed in search results.Versioning & immutability:- Each change creates new version object; artifacts immutable in object store with content-hash; metadata references artifact URI and schema diff stored.Access control & auditing:- Integrate with org IAM (SAML/OKTA) for identities; policies support team/role-based rules and attribute-based conditions (env=prod).- All access and metadata changes logged to audit store (immutable) for compliance.Trade-offs:- Graph DB for lineage is stronger than RDB for traversals; adds operational complexity.- ES for search provides low-latency UX but requires sync jobs.This design balances discoverability, safe reuse, and governance for multi-team feature collaboration.
HardSystem Design
26 practiced
Design SLOs and an observability plan for a feature platform that supports both offline materialization of feature tables and online feature serving. Include SLIs, dashboards, alerting thresholds, synthetic checks, and runbook examples for common failure scenarios (materialization fails, online store lag, schema mismatch).
Sample Answer
Requirements & constraints:- Functional: support offline materialization (batch jobs producing feature tables) and online feature serving (low-latency reads from feature store/online store).- Non-functional: availability 99.9% for online reads, freshness guarantees (e.g., 15 min SLA) for critical features, correctness (schema+values).High-level SLOs (examples):- Online read availability: 99.9% monthly (SLO) measured as successful RPCs / total RPCs.- Online freshness: 99% of reads return features updated within T_fresh (e.g., 15 minutes) for streaming/nearline sources.- Materialization success: 98% of scheduled materialization jobs complete successfully within expected window.- Schema compatibility: 99.99% of feature read/write operations conform to approved schema.SLIs (measurements):- RPC success rate (4xx/5xx errors excluded).- P95/P99 read latency.- Freshness distribution: percent of feature keys with last-update delta <= T_fresh.- Job success rate and job duration percentiles.- Schema validation failures per 10k events.Dashboards (panels to include):- Overview: SLO burn rate, current month SLI values, alerts.- Online serving: QPS, error rate, P50/P95/P99 latency, request size, per-feature latency heatmap.- Freshness: histogram of last-update deltas, % within T_fresh per feature group.- Materialization pipelines: job success rate, runtimes, records processed, upstream source lag, per-partition failure counts.- Schema: recent schema changes, validation failure timeseries, per-feature mismatch counts.- Resource health: CPU/memory/io for online store nodes, DB connections, Kafka consumer lag.Alerting thresholds (examples):- Pager (P1): Online error rate >1% for >5m OR P99 latency >300ms for >5m OR availability <99.5% => page on-call.- High burn: SLO burn rate >4x for 1h => page.- Materialization: Scheduled job failure rate >5% for last 30m OR average job runtime >2x expected => page.- Freshness: >5% of keys exceed T_fresh for >10m => page.- Schema: Any schema validation failures >0.01% of operations in 5m => page; single critical schema change that breaks reads/writes => page.Synthetic checks:- End-to-end user-flow every 5 minutes: write test event upstream, wait for materialization and read from online store; verify values & schema.- Health probes: simple read of representative keys per feature group every 1 minute (check latency and correctness).- Materialization smoke: after each scheduled run, validate row counts and sample values for a few keys.Runbook examples1) Materialization fails (scheduled job errors)- Symptoms: job status FAILED, downstream row counts missing, alert triggered.- Immediate actions: 1. Check job logs (airflow/Dataproc/Spark UI). Identify exception (e.g., upstream schema change, resource OOM, permission error). 2. If transient (network, transient HDFS/GCS error), retry job with backoff; monitor synthetic checks. 3. If resource OOM: increase executor resources or partitioning; re-run on staging. 4. If schema upstream change: roll back upstream deploy or patch transformation to accept new schema; coordinate with data producer.- Mitigation: mark affected features as stale in feature registry; notify consumers and throttle dependent pipelines.- Postmortem: capture root cause, fix test coverage, add alerting or schema guard.2) Online store lag / high read latency- Symptoms: elevated P95/P99 latency, increased RPC errors, high CPU or DB connections, cache miss rate up.- Immediate actions: 1. Check node metrics, DB connection pool, network saturation. 2. Check cache hit ratio (Redis/Memcached). If cache miss spike, scale cache or investigate TTLs/evictions. 3. If DB throughput saturated, scale read replicas, increase provisioned IOPS, or rate-limit non-critical queries. 4. If sudden traffic spike: enable autoscaling or apply traffic shaping; fail open/closed per business rules.- Mitigation: temporarily divert low-priority requests, enable degraded mode returning cached or default values; notify downstream teams.- Postmortem: tune indexes, query plans, and caching; add synthetic read keys to detect earlier.3) Schema mismatch (writes or reads fail)- Symptoms: schema validation failures, write rejections, consumer errors.- Immediate actions: 1. Identify the change: check schema registry/log for recent schema diffs. 2. If producer changed: coordinate rollback or implement backward-compatible transform (nullable fields, default values). 3. If consumer expects old schema: add compatibility shim in serving layer or use versioned feature views. 4. Run synthetic end-to-end test for affected feature(s).- Mitigation: quarantine bad deploy, enable schema migration mode, alert producers/consumers.- Postmortem: enforce schema evolution rules, add pre-deploy contract tests, add gating via CI to block incompatible changes.Best practices & trade-offs:- Prioritize SLIs for features driving online models; non-critical features can have laxer freshness SLOs.- Use sampling for high-cardinality freshness checks to reduce cost.- Combine alert thresholds with burn-rate detection to avoid noisy pages.- Automate synthetic checks and surface failures in dashboards; ensure runbooks are searchable and kept up to date.This plan balances reliability, detectability, and operational cost while giving clear on-call guidance for the common failure modes specified.
EasyTechnical
23 practiced
In Spark, explain the difference between map and flatMap, and explain what causes a shuffle. In the context of DataFrame/RDD operations, describe when repartitioning occurs implicitly and how you would control partitioning to optimize performance for joins and aggregations.
Sample Answer
map vs flatMap:- map applies a function to each element and returns exactly one output element per input (1:1). Example: rdd.map(x => x * 2).- flatMap applies a function that returns a collection for each input and flattens results (1:0..N). Example: rdd.flatMap(line => line.split(" ")) splits lines into words and emits each word as a separate record.What causes a shuffle:- A shuffle is a network/between-executor re-partitioning of data that requires disk/serialize/sort. It happens on “wide” transformations that need data from multiple partitions to be co-located: groupByKey, reduceByKey (the reduce side), aggregateByKey, join, cogroup, distinct, repartition, sortBy, and when you explicitly call repartition or when Spark needs to exchange data to satisfy partitioning requirements (e.g., different partitioners).When repartitioning occurs implicitly (DataFrame/RDD):- Joins: if two datasets have different partitioners or hash partitioning keys, Spark will shuffle one or both sides to align partitions.- groupBy/agg: grouping by a key triggers shuffle to aggregate by key.- distinct/orderBy: require global coordination and cause shuffle.- Wide operations returning shuffled partitioner (e.g., repartition(n), join) will trigger exchange.How to control partitioning to optimize joins and aggregations:- Reduce shuffles: ensure datasets are partitioned by the join key with the same partitioner (RDD.partitionBy or DataFrame.repartition(col)). Example: df1.repartition(200, $"userId") and df2.repartition(200, $"userId") before joining.- Choose appropriate number of partitions: use spark.default.parallelism or tune to cluster cores; too many small partitions add overhead, too few cause stragglers.- Use map-side combiners: prefer reduceByKey/aggregateByKey over groupByKey for RDDs to combine locally before shuffle.- Broadcast small table: use broadcast join (broadcast(smallDf)) or df.hint("broadcast") to avoid shuffling the large side.- Use coalesce(n, shuffle=false) to reduce partitions without shuffle when safe; use repartition(n) to increase or evenly reshuffle.- Persist partitioned intermediate datasets (cache with appropriate storage level) when reused.- Consider bucketing/sorting (in Spark SQL) to avoid shuffles across repeated joins.Example patterns:- LargeTable.join(SmallTable) -> broadcast(SmallTable) if small enough.- For repeated joins on key: write dataset bucketed by key or repartition both inputs by key with same numPartitions then join.Key trade-offs: repartitioning reduces downstream shuffle cost but has an upfront cost; broadcasting avoids shuffle but requires driver/executor memory. Choose based on data sizes and cluster resources.
Unlock Full Question Bank
Get access to hundreds of Data Pipelines and Feature Platforms interview questions and detailed answers.
