Designing and operating data pipelines and feature platforms involves engineering reliable, scalable systems that convert raw data into production ready features and deliver those features to both training and inference environments. Candidates should be able to discuss batch and streaming ingestion architectures, distributed processing approaches using systems such as Apache Spark and streaming engines, and orchestration patterns using workflow engines. Core topics include schema management and evolution, data validation and data quality monitoring, handling event time semantics and operational challenges such as late arriving data and data skew, stateful stream processing, windowing and watermarking, and strategies for idempotent and fault tolerant processing. The role of feature stores and feature platforms includes feature definition management, feature versioning, point in time correctness, consistency between training and serving, online low latency feature retrieval, offline materialization and backfilling, and trade offs between real time and offline computation. Feature engineering strategies, detection and mitigation of distribution shift, dataset versioning, metadata and discoverability, governance and compliance, and lineage and reproducibility are important areas. For senior and staff level candidates, design considerations expand to multi tenant platform architecture, platform application programming interfaces and onboarding, access control, resource management and cost optimization, scaling and partitioning strategies, caching and hot key mitigation, monitoring and observability including service level objectives, testing and continuous integration and continuous delivery for data pipelines, and operational practices for supporting hundreds of models across teams.
EasyTechnical
30 practiced
Explain tumbling, sliding, and session windows and give a concrete example of when to use each in user behavior feature calculations. Include considerations about state retention and how session window gaps are chosen.
Sample Answer
Tumbling, sliding, and session windows are common streaming windowing strategies for aggregating events over time — useful when computing user-behavior features for ML.Tumbling windows: fixed, non-overlapping intervals (e.g., 1-hour buckets). Use when you need disjoint summaries like “number of purchases per user per hour” for hourly features. Simple state: only keep current bucket(s); retention equals window length plus allowed lateness.Sliding windows: fixed-size windows that advance by a smaller hop (e.g., size=1 hour, hop=15 minutes) producing overlapping aggregates (rolling views). Use for smoother time-series features like a rolling 1-hour click-rate updated every 15 minutes. State: must maintain partial aggregates for multiple overlapping windows; retention ≈ window size + lateness.Session windows: dynamically sized windows that group events separated by gaps of inactivity (e.g., gap = 30 minutes). Use for session-level features such as “avg actions per user session” or “time-to-next-purchase within a session.” State: keep open sessions keyed by user until gap + allowed lateness expires; this can grow if many long sessions exist.Choosing session gap: base it on domain behavior and trade-offs — short gaps split natural sessions, long gaps merge distinct sessions. Inspect histograms of inter-event times, A/B test gap values, and consider downstream model sensitivity. Also account for late/out-of-order events by adding allowed lateness and retention TTLs to avoid unbounded state.
HardSystem Design
32 practiced
Design a multi-tenant resource management model for a shared feature platform: propose namespace quotas, autoscaling boundaries, priority scheduling, cost attribution and fair-share policies to prevent noisy neighbors while enabling self-service for teams.
Sample Answer
Requirements & constraints:- Support many ML teams running training (batch, distributed), dev/experiment workloads, and real-time inference.- Prevent noisy neighbors (GPU/CPU/memory, storage IOPS, network).- Allow self-service within quotas, enable autoscaling, and provide accurate cost attribution for chargeback/showback.- Low-latency priority for production inference; best-effort for experiments.High-level architecture:- Kubernetes-based shared platform with multi-tenant namespaces per team/project.- Components: Namespace quota controller (hierarchical), Autoscaler (HPA/VPA + cluster autoscaler/GPU-aware Karpenter), Scheduler extension (Volcano + custom scheduler-extender), Admission webhook for policy checks, Usage collector & billing pipeline, Monitoring/alerting.Core components & responsibilities:1. Namespace quotas & hierarchy- Implement Hierarchical Resource Quotas (HRQ) per org → team → project namespaces.- Quotas include CPU, memory, GPUs, ephemeral storage, IOPS, and concurrency slots (max concurrent GPU jobs).- LimitRanges set per-namespace defaults (request/limit floor/ceil) to prevent tiny/huge requests.- Borrowing policy: allow short-term burst borrowing from org pool if spare capacity; tracked with debt and rate-limited.2. Autoscaling boundaries- Per-namespace autoscale policy: min/max pods, min/max GPUs, and cost cap (daily/weekly).- Cluster autoscaler integrates with GPU-aware provisioner (Karpenter) to add spot/on-demand pools with labels (spot=cheap, preemptible=true).- Workloads annotated with acceptable node types (spot vs on-demand). Admission webhook blocks autoscale if cost cap hit.3. Priority scheduling & preemption- Define PriorityClasses: prod-inference > prod-training > interactive/experiment > batch-low.- Use Volcano/Batch scheduler for gang-scheduled distributed training (MPI/TFJob) to ensure all-or-nothing allocation.- Scheduler-extender enforces multi-resource fairness and IOPS/network awareness; preemption allowed only across non-prod tiers and guarded by SLA windows.4. Fair-share policies (noisy neighbor mitigation)- Implement Dominant Resource Fairness (DRF) or Token-based fair-share at scheduler level per-team weights.- Per-namespace concurrency slots for GPU usage and I/O tokens to throttle heavy disk/network users.- Backpressure + autoscale throttle: if one namespace hits I/O limit, reduce its background jobs priority, notify owner.5. Cost attribution & chargeback- Enforce mandatory labels/annotations on jobs (team, project, cost-center, experiment-id).- Usage collector (Prometheus + custom exporter) aggregates CPU/GPU-hours, memory-hours, network/IOps, storage GB-months, plus node-type pricing.- Pipeline writes to data warehouse (BigQuery) and produces daily/weekly chargeback and cost-per-experiment dashboards.- For spot preemptions, attribute cost and preemption events and optionally reimburse/credit teams.Data flow (example training job):- User submits TFJob to namespace with labels. Admission webhook checks quotas/cost cap and annotates node preferences.- Scheduler (Volcano + extender) evaluates DRF fair-share + priority, reserves gang slots.- If insufficient nodes, autoscaler provisions nodes (respecting node pools and cost policy).- Usage collector streams metrics; job finishes; billing pipeline consumes aggregated usage and tags cost to team.Monitoring, observability & enforcement:- Prometheus + Grafana: per-namespace resource usage, quota debt, cost burn rate, preemption events.- Alerts: approaching quota, approaching cost cap, noisy neighbor throttle actions.- Enforcement: admission webhook can reject or convert to low-priority queue when caps exceeded.Policies & guardrails:- Self-service: teams can request quota increases via automated workflow; temporary burst approvals logged for audit.- Safety: production inference requires mandatory SLA and higher priority; experimental jobs default to spot nodes and preemptible pool.- Governance: periodic quota review, automated reclamation of idle GPU pods, sandbox periods for new teams.Trade-offs:- Complexity vs flexibility: HRQ + scheduler-extender adds operational overhead but gives precise multi-resource fairness necessary for ML workloads.- Spot nodes reduce cost but increase preemption; mitigate by letting critical jobs require on-demand nodes.- Strict quotas prevent noisy neighbors but can slow innovation; mitigate with borrowing, short-term burst windows, and easy quota request workflows.Why this fits ML teams:- Supports distributed gang-scheduled training with GPU-awareness.- Enables fast iteration (self-service) while protecting production inference SLAs.- Accurate cost attribution supports responsible usage and optimization of expensive GPU resources.
HardTechnical
27 practiced
Describe how to implement stateful stream processing for event-time windowed feature computation that tolerates out-of-order and late events, using Flink or Beam. Include how you would manage keyed state, event-time timers, checkpointing, state backend sizing, and how to handle very large state per key.
Sample Answer
Requirements & approach:- Use event-time semantics with watermarks, allow configurable allowed-lateness, and use keyed state + event-time timers to emit windowed feature aggregates when the window is complete.- Prefer Flink (similar ideas in Beam): use keyed streams by entity ID, maintain compact aggregations per window, and use timers to close windows and clean state.Flink implementation sketch:- Key by entity ID.- For each event, map to its event-time window bucket (e.g., tumbling/ sliding).- Maintain keyed state (MapState<windowId, Aggregate>) or use WindowOperator, and register an event-time timer at windowEnd + allowedLateness to emit result and another timer at windowEnd + allowedLateness + cleanupDelay to delete state.- Use side outputs for very-late events (beyond allowed lateness) or re-ingest to a different pipeline for backfills.Example (pseudo-Flink Java/Python):
java
// inside KeyedProcessFunction
MapState<Long, Aggregate> windowState;
public void processElement(Event e, Context ctx, Collector<Feature> out) {
long windowId = computeWindow(e.eventTime);
Aggregate agg = windowState.get(windowId);
agg.add(e);
windowState.put(windowId, agg);
long emitTs = windowEnd(windowId) + allowedLateness;
ctx.timerService().registerEventTimeTimer(emitTs);
long cleanupTs = emitTs + cleanupDelay;
ctx.timerService().registerEventTimeTimer(cleanupTs);
}
public void onTimer(long ts, OnTimerContext ctx, Collector<Feature> out) {
if (ts == emitTs) { emit feature, maybe write to feature store; }
if (ts == cleanupTs) { windowState.remove(windowId); }
}
State management:- Use RocksDBStateBackend for large state (local, spill-to-disk), enable incremental checkpoints (Flink: setIncrementalCheckpointing(true)).- Configure heap and RocksDB options: increase write buffers, set bloom filters, tune compaction and memory. Monitor checkpoint duration and throughput.- Enable state TTL to automatically expire unused keys.Checkpointing & correctness:- Use event-time processing + watermarks from source (Kafka/CloudPubSub) with watermark heuristics; ensure source supports offsets for exactly-once (e.g., Kafka + FlinkKafkaConsumer with checkpointing).- Enable periodic checkpoints and set checkpoint timeout > expected checkpoint duration. Use externalized checkpoints and savepoints for deployments.- Use two-phase commit sinks for at-least-once vs exactly-once durability (Flink’s TwoPhaseCommitSink for feature store writes).Sizing & operational concerns:- Measure average state per key and total cardinality. Estimate checkpoint size = sum of state; target per-task state such that checkpointing fits memory and network constraints.- Use vertical partitioning (shard by composite key), increase parallelism (rescale using savepoints) to reduce per-task state.Handling very large state per key:- Avoid storing full raw history per key; keep compact aggregations (counters, reservoirs, quantile sketches, exponential decay summaries).- For use-cases that need full history, store history externally (S3/HBase/Cassandra) and keep index pointers in keyed state. Fetch on demand (async I/O) to avoid blowing up keyed state.- Use incremental or layered storage: keep recent N windows in state, older materialized to external DB; compaction jobs to downsample old data.Late and out-of-order events:- Use watermark strategy tuned to expected lateness; allow-lateness to accept late events for retraction/updates.- If updates to emitted features are allowed, emit retraction/updated records (use changelog semantics) and write idempotent updates to feature store.- For extremely late events beyond allowed lateness, route to a dead-letter/backfill pipeline and optionally recompute affected windows via batch reprocessing.Monitoring & testing:- Track watermark lag, checkpoint duration, state size per task, RocksDB metrics, and late-event counts.- Load-test with synthetic out-of-order patterns and test recovery via savepoints.Trade-offs:- Larger allowed lateness increases correctness for late arrivals but increases state retention and checkpoint size.- RocksDB + incremental checkpoints handles large state but adds IO overhead and tuning complexity.- Externalizing history reduces keyed-state pressure at the cost of increased read latency and complexity.This design balances correctness (event-time semantics, timers, checkpointing) with scalability (RocksDB, incremental checkpoints, sharding, compact aggregates) and offers patterns (side outputs, external storage, sketches) to handle very large per-key state.
EasyTechnical
24 practiced
What does point-in-time correctness mean for feature joins in training data preparation? Given a label occurring at time T, describe how you would join historical features such that no future information leaks into the training sample.
Sample Answer
Point-in-time correctness means that when constructing a training example for a label at time T, every feature value must be what would have been known at T — no values computed from events after T may be used. This prevents data leakage and ensures training-serving consistency.Practical approach:- Define a cutoff_time = T for each label.- Store event timestamps for raw features and feature computation timestamps for derived features.- Join by entity_id and require feature_timestamp <= cutoff_time (or <= feature_computation_time <= cutoff_time for derived features).- For time-series or incremental features, select the most recent record before or at cutoff_time (e.g., using ROW_NUMBER() OVER (PARTITION BY entity_id ORDER BY feature_timestamp DESC) and pick row_number = 1).- Use a feature store or time-travel queries/snapshots so you can reproducibly read historical feature state as of cutoff_time.Example SQL pattern:SELECT L.*, F.*FROM labels LLEFT JOIN ( SELECT *, ROW_NUMBER() OVER (PARTITION BY entity_id ORDER BY event_ts DESC) rn FROM features WHERE event_ts <= L.label_time -- implement via a correlated subquery or snapshot mechanism) F ON F.entity_id = L.entity_id AND F.rn = 1Key considerations:- Use an immutable snapshot or feature-store with materialized views to avoid subtle joins that accidentally read future data.- Handle late-arriving data explicitly (backfill or tag as unavailable).- Ensure timestamps use consistent timezones and align on the same clock.- Validate with unit tests that no feature has event_ts > label_time.This enforces no-future-information leaks and mirrors how the model will see data in production.
MediumTechnical
29 practiced
For a Kafka + Spark feature pipeline, design a CI/CD and testing strategy covering unit tests for transforms, schema checks, integration tests for streaming jobs, and automated validation for backfills. Explain how to run fast checks locally and longer end-to-end tests in CI before production deployment.
Sample Answer
Approach summary: separate fast local checks (unit + schema) from heavier integration/end-to-end (E2E) streaming and backfill validation. Automate both in CI with staged pipelines (fast → integration → canary/production).Unit tests for transforms- Use pytest + pyspark testing utilities (pytest-spark or a local SparkSession fixture) to run DataFrame-level tests in-memory.- Test small deterministic inputs, assert schema, column values, null handling, and idempotence. Example assertions: schema equality, row-count, column-wise approximate equality for floats.- Mock external dependencies (Kafka, schema-registry, storage) with fixtures.Schema checks- Keep schemas in a canonical format (Avro/JSON Schema) in repo. Add a pre-commit/CI step to validate code-generated schemas against canonical files.- Use schema-registry client or fastavro to validate sample records and fail fast on incompatible changes.- Integrate Great Expectations or custom validators to assert column types, ranges, and nullability during unit tests.Integration tests for streaming jobs- Use Testcontainers (Kafka, Schema Registry, MinIO) or docker-compose to run ephemeral infra in CI.- Start Spark in local-cluster mode or use a lightweight Spark image; submit job with test config pointing to test Kafka topics and sink (filesystem/MinIO).- Produce synthetic events to Kafka, wait for processing, then validate outputs in sinks (counts, feature distributions, schema).- Use deterministic event time and short windows to finish quickly. Include checkpoint isolation per test.Automated backfill validation- Backfill runs produce snapshots (per-partition) and metrics: input count, output count, row-level hashes (e.g., murmur3 over key+features), and sample histograms.- Validation job compares new backfill snapshot against golden snapshot or against incremental streaming output: assert counts within tolerance, checksum match, and distribution drift (KS test) below threshold.- If discrepancy, fail pipeline and open PR with diff artifacts.Fast local checks vs longer CI- Local fast checks: run pytest unit suite, schema lint, and small Spark unit tests (pure transformations) targeting single-node Spark. Use tox/poetry to run quickly; keep test data tiny.- CI longer tests: staged runners that spin up Testcontainers and run integration streaming tests and full backfill validations. Use parallelization and caching (cached docker images, prebuilt Spark jars) to reduce wall time.- CI pipeline example stages: 1. lint, unit tests, schema validation (fast) 2. build artifact (container/jar) 3. integration tests with Testcontainers (kafka + schema-registry + minio) 4. backfill validation against a small synthetic dataset 5. canary deployment to a limited topic/namespace and smoke tests 6. promote to production on successOperational + safety- Require schema compatibility checks in CI before merge.- Produce reproducible test data (seeded RNG).- Expose test artifacts (logs, sample outputs, diffs) for debugging.- Use feature flags/canarying, monitoring (metrics, alerting), and rollback playbooks for production deploys.This strategy balances quick feedback for developers with rigorous integration and backfill validation in CI to minimize data/regression risk when deploying Kafka+Spark feature pipelines.
Unlock Full Question Bank
Get access to hundreds of Data Pipelines and Feature Platforms interview questions and detailed answers.
