Designing and operating data pipelines and feature platforms involves engineering reliable, scalable systems that convert raw data into production ready features and deliver those features to both training and inference environments. Candidates should be able to discuss batch and streaming ingestion architectures, distributed processing approaches using systems such as Apache Spark and streaming engines, and orchestration patterns using workflow engines. Core topics include schema management and evolution, data validation and data quality monitoring, handling event time semantics and operational challenges such as late arriving data and data skew, stateful stream processing, windowing and watermarking, and strategies for idempotent and fault tolerant processing. The role of feature stores and feature platforms includes feature definition management, feature versioning, point in time correctness, consistency between training and serving, online low latency feature retrieval, offline materialization and backfilling, and trade offs between real time and offline computation. Feature engineering strategies, detection and mitigation of distribution shift, dataset versioning, metadata and discoverability, governance and compliance, and lineage and reproducibility are important areas. For senior and staff level candidates, design considerations expand to multi tenant platform architecture, platform application programming interfaces and onboarding, access control, resource management and cost optimization, scaling and partitioning strategies, caching and hot key mitigation, monitoring and observability including service level objectives, testing and continuous integration and continuous delivery for data pipelines, and operational practices for supporting hundreds of models across teams.
MediumSystem Design
47 practiced
Design a small feature pipeline in prose that does the following: consumes clickstream events from Kafka, enriches events with user profile data from a key-value store, computes per-user hourly click-through rate (CTR) feature, writes offline feature materialization to a data lake for training and writes per-user CTR to an online store for serving. Outline components, data contracts, fault-tolerance mechanisms, and how you would validate that offline and online feature values are consistent.
Sample Answer
Requirements (clarified): consume clickstream from Kafka, enrich with user profile from KV store, compute per-user hourly CTR, write offline materialization to data lake for training, and write per-user CTR to an online store for serving. Ensure fault-tolerance and validate offline vs online consistency.High-level architecture:- Kafka cluster (clickstream topic, partitioned by user_id)- Stream processor (Apache Flink / Beam) for enrichment and hourly CTR aggregation- KV store for profile enrichment (Redis/Cassandra) with a caching layer- Data lake (S3/ADLS) for offline materialization (Parquet + partitioned by date/hour)- Online feature store (Redis/Bigtable/DynamoDB) for per-user CTR- Monitoring, reconciliation and QA jobs (batch Spark/Flink) and dead-letter topicComponents & responsibilities:1. Kafka consumer (stream app): reads Avro/Protobuf-encoded click events (schema: user_id, event_time, item_id, action_type, request_id).2. Enrichment: stream job performs async/cached lookup to KV store for user profile (schema: user_id, cohort, signup_date, active_flag). Cache with TTL; on miss, mark enrichment null and optionally emit enrichment request to backfill.3. Aggregation: use keyed hourly tumbling windows by user_id to compute CTR = clicks / impressions (or clicks / exposures depending on events). Keep intermediate state in RocksDB (checkpointed).4. Materialization: - Offline: at end of each hour write feature records in Parquet partitioned by date/hour and user_id to data lake. Feature schema includes window_start, window_end, user_id, ctr, profile_snapshot, source_watermark, compute_version. - Online: write per-user CTR to online store with key = user_id:feature:ctr and value = {ctr, window_ts, compute_version, ttl}. Use idempotent upserts and a compare-and-set or version tag to avoid regressions.Data contracts:- Event schema (Avro): required fields and types; include request_id, event_time (ISO/epoch), event_type.- Feature record schema (Parquet/Avro): window_start, window_end, user_id, metric_name, value, compute_version, source_watermark.- API for online reads: GET /feature/{user_id}?version=latest returns value + window_ts.Fault-tolerance mechanisms:- Kafka consumer uses committed offsets, processing with exactly-once semantics (Flink two-phase commit / transactional writes) for writing to the data lake and ensuring at-least-once to online store but guarded by idempotency/versioning.- Stream state is checkpointed to durable storage; RocksDB local state + periodic snapshots.- Enrichment lookup retries with exponential backoff; cache fallbacks and a dead-letter queue for enrichment failures.- DLQ Kafka topic for malformed events or permanent enrichment failure.- Partitioning by user_id ensures per-user ordering for correct windowing.- Backpressure handling and scaled parallelism controlled by Kafka partition count.Validation and consistency between offline and online:- Include compute_version and watermark in both offline records and online writes.- Daily reconciliation job: for a sample or full set, read offline materialized CTR for a given hour and compare to online store values for same window and compute_version. Compute error metrics (mean absolute error, % drift). Alert if drift > threshold.- Shadow/batch recompute: periodically run a batch job that re-aggregates clicks from Kafka (or raw events in the data lake) for recent windows and diff against online store. Use reconciliation IDs and checksums (e.g., hash(user_id + window + value)) to detect disagreements quickly.- End-to-end test harness: inject synthetic events (test user ids) into Kafka; assert that offline materialization and online store converge to expected CTR within SLA.- Canary and versioning: when changing compute logic, run new compute_version in parallel and compare before promoting to production.Scalability and trade-offs:- Use Kafka partitioning and keyed state to scale horizontally. RocksDB state scales with operator parallelism.- Using a cache for enrichment reduces KV load; eventual staleness acceptable if profile changes are infrequent—if not, add change-data-capture pipeline to update cached profiles.- Exactly-once to data lake via transactional sinks costs complexity; acceptable to guarantee training data correctness. For online store, idempotent writes + versioning are simpler and performant.This design yields low-latency per-user CTR for serving, durable offline feature materialization for training, robust fault handling, and explicit reconciliation to ensure consistency between offline and online feature values.
HardSystem Design
43 practiced
Design an access control model for a feature registry that includes: read-only discovery for most users, write/modify permissions for feature owners, and export permissions for compliance teams. Describe policy enforcement, audit logging, and how you'd implement row/column level restrictions for PII-sensitive features.
Sample Answer
Requirements & constraints:- Functional: allow read-only discovery for most users, write/modify for feature owners, export for compliance team.- Non-functional: low-latency reads, strong auditability, fine-grained data masking for PII, scalable to thousands of features and users, policy consistency across UI/API/CLI.High-level architecture:- Policy Store (centralized): stores RBAC roles, resource policies, attribute-based rules (ABAC) for PII tags.- Authorization Service (sidecar / gateway): evaluates every request against Policy Store and returns Allow/Deny + obligations (masking instructions).- Feature Registry DB: metadata (feature definitions, owners, tags), optionally separate secure vault for raw PII fields.- Audit Logging Service: immutable append-only log (WORM) feeding SIEM and compliance UI.- Data Masking/Filtering Layer: applies column/row filters based on obligations from Authorization Service before returning data.Policy model:- Combine RBAC + ABAC: - Roles: Reader (default), Owner, ComplianceExport. - Resource attributes: feature_id, owners[], pii_level (NONE/LOW/HIGH), env (prod/test), sensitivity_tags. - Policies examples: - Reader: allow read if resource.pii_level == NONE OR user.hasAttribute('can_view_pii') == true AND user.department == resource.owner_department. - Owner: allow write/modify if user.id ∈ resource.owners. - ComplianceExport: allow export if user.role == ComplianceExport AND export_request.reason ∈ approved_reasons AND export_destination is whitelisted.Policy enforcement:- All API requests pass through Authorization Service which: - Authenticates user (OIDC/JWT). - Fetches applicable policies (cache with TTL). - Evaluates and returns decision + obligations (e.g., mask columns X,Y; remove rows where pii_flag = true). - The service or a trusted data-masking layer enforces obligations before returning results.- Use an externalized policy engine (e.g., OPA/Rego) for expressive ABAC rules and caching for performance.Audit logging:- Every decision and action: user_id, resource_id, action, decision, policy_id, obligation, timestamp, request_payload_hash, response_hash.- Exports produce additional records: export_destination, approval_id, signed consent, TTL for exported data.- Logs written to immutable store (e.g., append-only Kafka → object storage with encryption + retention policies). Provide automated alerts for suspicious patterns (large exports, repeated deny attempts).Row/column-level PII controls:- Tag features with pii_level and column-level sensitivity metadata.- For reads: - Authorization Service returns obligations: column_mask(column_name, mask_type) or row_filter(predicate). - Masking types: nullify, hash, redact, tokenization. Prefer tokenization for reversible needs (store tokens in secure vault). - Row-level: for records associated with PII, apply filters (e.g., only show rows where user.id == owner OR user.hasAttribute('can_view_pii')).- For exports: - Require explicit approval workflow recorded in audit log; for HIGH PII require multi-party approval; enforce export transforms (e.g., differential privacy, aggregation only).- Implementation: implement masking in data access layer close to data (database views or materialized masked views) and enforce via DB row-level security (e.g., Postgres RLS) combined with app-layer masking for analytics-backed stores.Scalability & trade-offs:- Cache policies heavily; push common decisions to gateway to reduce latency.- Trade-off: DB-level enforcement (RLS) gives strong guarantees but is harder to manage cross-datastore; centralized Authorization + obligations is flexible but requires trusted enforcement points.- Plan: use DB RLS for primary store and Authorization Service + masking for downstream systems.Monitoring & governance:- Periodic policy audits, automated tests for policy coverage, and a compliance dashboard exposing recent exports, failed denials, and owners for each feature. Regularly rotate keys for tokenization and secure vault access logs audited.This design ensures least-privilege discovery, owner-controlled modification, and tightly governed exports with strong auditability and fine-grained PII protection.
MediumSystem Design
24 practiced
Describe how you'd design a CI/CD pipeline for data pipelines and feature computation code. Include unit tests, integration tests (with synthetic data), canary runs, schema checks, and automated rollbacks. Also explain how you'd version and release feature code and metadata.
Sample Answer
Requirements:- Fast, repeatable CI for feature computation code and metadata- Safety gates: unit tests, integration tests with synthetic data, schema checks- Staged rollout: canary runs, monitoring, automated rollback- Reproducible versioning for code + feature schemas/definitionsHigh-level architecture:- Repo(s): feature-code (Python/SQL), feature-specs (YAML/JSON for metadata)- CI system: GitHub Actions / GitLab CI / CircleCI- Test infra: lightweight runner + sandboxed container images; integration harness uses synthetic data generation service- Orchestration/Deployment: Argo Workflows / Airflow + feature-serving infra (Feast or in-house)- Monitoring/metrics: Prometheus + alerting; data quality with Great Expectations or DeequPipeline stages:1. PR / Unit tests: run pytest, linters, type checks; mock external deps2. Static validation: validate feature-specs against schemas; semantic checks (naming, dependencies)3. Integration tests (synthetic): spin ephemeral infra, generate synthetic datasets covering edge cases, run end-to-end feature computation, compare outputs to golden assertions (tolerances for floats)4. Schema & data quality checks: run Great Expectations/Deequ to ensure output schemas, null rates, distributions within expected ranges5. Canary run: deploy feature code to staging and run feature computation on a small subset of production traffic or time-windowed historical partition; collect accuracy/regression metrics and data drift signals6. Automated verification: if metrics pass thresholds, promote; else trigger rollback and open a ticket.Automated rollback:- All deployments are blue/green or use feature flags. Canary failures trigger automatic rollback to previous release and run rollback verification tests; alerting to SRE/owner.Versioning & release:- Code versioned via git tags / semantic versioning.- Feature metadata uses deterministic versioning (e.g., feature_name:vYYYYMMDD or semantic) stored in artifact registry (Git + packaged release artifact).- Each release bundles code commit hash, container image tag, and feature-spec manifest; stored in a registry (OCI or internal).- Deployment references specific artifact versions; reproducible runs record lineage (code hash, feature-spec version, data partition).- For breaking changes: support multi-version coexistence and migration scripts; deprecate old versions with automated notices.Key trade-offs:- Synthetic tests increase safety but may not catch production edge cases → mitigated by conservative canary size and robust monitoring.- Fast CI vs thorough integration: parallelize tests and cache artifacts.This design balances safety (tests, schema checks), reproducibility (versioning, artifacts), and fast iteration (CI + canary promotion).
MediumTechnical
27 practiced
Given a large dataset of transactions skewed towards a small percentage of users (hot keys), explain three engineering strategies to mitigate data skew during offline feature computation using distributed processing (Spark or similar). For each strategy, describe trade-offs and when you'd prefer it.
Sample Answer
1) Salting / key augmentation- Idea: Append a random or hashed salt to the hot key when shuffling (e.g., (userId, salt)) so hot-user data spreads across multiple partitions; after local aggregation, remove salt and do a final reduceByKey for that user.- Trade-offs: Adds extra shuffle and a final merge step; increased intermediate state and slightly more computation. Requires careful choice of salt range to balance partition sizes.- When to prefer: Simple to implement in Spark for write-heavy hot keys where per-user aggregates are associative/commutative (counts, sums). Good when hot keys are few and you can tolerate one extra pass.2) Separate handling (skew-aware split)- Idea: Detect hot keys (pre-sampling or stats), process them separately with a dedicated workflow: load hot-user data into a broadcast variable or compute per-user features on a single heavy executor or via map-side combiner; process remaining keys with normal partitioning.- Trade-offs: More complex pipeline and bookkeeping; may underutilize resources if too many hot keys; careful memory management needed for broadcasting.- When to prefer: When hot keys are known and small in number and their state fits in memory, or when you want deterministic per-hot-key computation (complex UDFs) without impacting general shuffle.3) Custom partitioner + load balancing (range or frequency-based)- Idea: Use a custom partitioner that routes hot keys to multiple partitions or assigns fewer heavy keys per partition, e.g., hash(userId) but override hot user assignments to spread them evenly, or use frequency-based ranges for range partitioning.- Trade-offs: Requires maintaining and updating key-frequency metadata; less flexible for dynamic skews; custom partitioner logic adds operational complexity.- When to prefer: Production pipelines with stable skew patterns and when you need predictable partition boundaries and minimal extra shuffle (no salting final merge). Good for large clusters where rebalancing partitions gives better resource utilization.Summary: Use salting for quick wins on associative ops; separate handling for precise control or broadcastable hot-state; custom partitioners for stable, long-term skew mitigation with predictable performance.
HardSystem Design
32 practiced
Provide a short design for a cost-allocation and chargeback mechanism for your feature platform that fairly assigns storage and compute costs to teams based on usage. Describe what telemetry you would collect, how you'd attribute shared resources, and how to present this information to teams to incentivize cost-efficient designs.
Sample Answer
Requirements & constraints:- Fairly attribute storage and compute costs per team by usage (multi-tenant), support shared services, near real-time reporting, prevent gaming, and surface recommendations to incentivize efficiency.High-level architecture:- Telemetry pipeline (agents → ingest → metering DB) → attribution service → billing/chargeback DB → dashboards & alerts.Telemetry to collect:- Compute: pod/container VM id, owner/team tag, start/end timestamps, CPU seconds, GPU hours, memory GB-hours, node type (spot/reserved), pricing per unit.- Storage: volume id, owner tag(s), size GB by time (snapshots/deltas), IOPS, throughput, storage class.- Shared infra: load balancer, auth, logging: total cost + per-tenant request counts/bytes.- Context: deployment id, environment (prod/staging), region, cost rates, reservation amortization windows.Attribution rules:- Primary: deterministic team tags on resources — direct charge.- For untagged/shared resources: proportional attribution by measurable usage (e.g., requests, active connections, bytes transferred) over the billing period.- For reserved instances: amortize reservation cost across teams by their historical share of usage during reservation window.- Idle resources: charge higher idle rate or report warn-only; optional hourly minimums to discourage hoarding.- Define ownership fallbacks and automated tagging enforcement (admission controller).Presentation & incentives:- Team dashboard: daily/weekly cost trends, cost per feature, cost per request, hotspot drilldowns, estimated savings from specific actions.- Alerts: budget burn-rate, anomalous spikes, untagged resource creation.- Recommendations: rightsizing (memory/CPU), switch storage class, use spot instances, schedule non-prod shutdowns — include estimated monthly savings and playbook.- Show normative benchmarks (median cost per request for similar services) and leaderboard (careful to avoid blame).- Integrate chargeback statements into internal invoicing and engineering reviews.Governance & anti-abuse:- Enforce tagging via CI/CD and admission controllers; require cost-ownership in PRs.- Periodic audits, dispute process, and rate-limiting for sudden cost by a team.Trade-offs:- Per-second metering is accurate but more costly; aggregate hourly for lower overhead.- Proportional attribution for shared components is fair but can obscure causality — provide request-level traces for reconciliation.This design balances accuracy, transparency, and actionable guidance to drive cost-efficient engineering behavior.
Unlock Full Question Bank
Get access to hundreds of Data Pipelines and Feature Platforms interview questions and detailed answers.
