End-to-End ML System Design Questions

End-to-end design of machine learning systems, covering data collection and validation, feature engineering and feature stores, model training and evaluation, deployment and serving architectures, monitoring and incident management, retraining pipelines, data governance, scalability, security, and MLOps practices.

HardSystem Design

28 practiced

Design an ML platform to support many teams sharing features, compute, and models in a multi-tenant environment. Address feature discoverability, per-team isolation and quotas, access control and billing, shared vs private compute clusters, metadata/catalog services, reproducibility, and strategies for preventing 'noisy neighbor' effects. Explain how you would roll out such a platform incrementally.

Sample Answer

**High-level approach**Design a multi-tenant ML platform with a central Feature Store + Metadata/Catalog, per-team logical isolation, shared/private compute options, strict RBAC and quota/billing pipes, and tooling to ensure reproducibility and limit noisy neighbors. I’d prioritize discoverability, provenance, and incremental roll-out to reduce risk.

**Core components**- Feature Store + Catalog: centralized, versioned feature registry (feature schemas, owners, lineage, freshness, cost estimates). Searchable UI + API and automated feature quality checks.- Metadata & Model Registry: experiment metadata, dataset snapshots, model artifacts, evaluation metrics, container images, and provenance linked to feature versions and code commit hashes.- Compute Layer: Kubernetes-based clusters with namespaces per team; allow shared clusters (cost-effective) and private clusters (isolation, TPU/GPU) with node pools.- Access control & billing: RBAC + attribute-based access (team, role). All workloads tagged; usage exported to billing pipeline (cost per team, project, feature owner).- Quotas & noisy-neighbor controls: K8s ResourceQuota, LimitRange, PriorityClasses, PodDisruptionBudgets, cgroups/CPU shares. Node-pools with taints/tolerations. Burst queues limited via fair-share scheduler (e.g., Volcano or Kubernetes Queue) and preemptible pools for low-priority jobs.- Reproducibility: immutable experiment snapshots (git + data hashes), containerized runs, workflow engine (Airflow/Argo) that records DAG, parameters, bindings to feature & data versions.

**Feature discoverability & governance**- Feature metadata: owner, SLA, freshness, downstream consumers, cost.- Auto-generated dependency graphs and CI jobs to verify feature quality on changes.- Approval workflow for publishing features to "public" catalog with tags (stable/experimental).

**Per-team isolation & quotas**- Logical isolation: namespaces, network policies, secrete scopes.- Resource quotas per namespace and per-user rate limits.- Time-limited burst allowances and chargebacks to encourage cost-awareness.

**Shared vs private compute**- Shared clusters: general CPU workloads, low-cost training, enforced quotas, fair-share scheduling.- Private clusters: reserved for high-performance/hardware-sensitive workloads with stricter access, chargeback, and approval processes.

**Preventing noisy neighbors**- Enforce CPU/memory limits + cgroups; autoscaling with per-namespace caps.- Use QoS classes and priority preemption for critical workloads.- Limit GPU attachment per job and schedule using gang scheduling for large runs.- Rate-limit high-IO jobs; use separate storage tiers (hot/cold) and IOPS quotas.

**Incremental rollout plan**1. MVP (months 0–3): feature catalog (read-only), metadata store, simple shared K8s with RBAC and quotas, basic billing tags, and a model registry for a pilot team.2. Expand (3–6): enable feature publishing, CI checks, reproducible experiment snapshots, and wider team onboarding.3. Harden (6–12): introduce fair-share scheduler, private cluster option, advanced noisy-neighbor controls, automated cost-allocation and alerts.4. Optimize (12+): policy automation, autoscaling optimizations, SLA metrics, feedback loop from teams to refine UX and governance.

**Why this design (Applied Scientist perspective)**This balances fast experimentation (discoverable features, reproducible experiments) with production reliability (isolation, quotas, billing). As an applied scientist I’d prioritize lineage and reproducibility so research can be productized safely while minimizing interference between teams.

HardSystem Design

31 practiced

Design an end-to-end online recommendation system for a large-scale platform with 100M users and 10M catalog items. Requirements: support 5k QPS read traffic, p99 tail latency under 150ms for top-10 recommendations, embedding-based recall with ANN, freshness of results within 5 minutes of events, cold-start handling for new users and items, and efficient periodic re-embedding. Describe offline training, embedding storage and update strategies, ANN index sharding (e.g., FAISS), serving architecture, caching, consistency, and trade-offs for latency vs freshness.

Sample Answer

**Clarify requirements & constraints**- 100M users, 10M items, 5k QPS reads, p99 <150ms for top-10, ANN recall, freshness within 5 minutes, handle cold-start, periodic re-embedding.

**High-level architecture**- Offline training: batch jobs (Spark + TF/PyTorch) compute user/item embeddings using two-tower or hybrid model (behavior+content). Train daily / hourly; incremental online model for rapid updates.- Feature pipeline: stream events (Kafka) → Feeds real-time feature store (e.g., Feast) and to micro-batch jobs for incremental embedding deltas.

**Embedding storage & updates**- Store embeddings in two places: - Low-latency vector store: FAISS GPU/CPU shards backed by RocksDB for persistence. - Feature store / blob store (S3) for full re-training snapshots.- Update strategy: - Real-time deltas: compute lightweight user embedding updates on event (online retrain or SSE-based update) and write to vector store within seconds. - Periodic full re-embedding: nightly or hourly re-compute for quality; use rolling blue/green swap of FAISS shards.

**ANN index sharding (FAISS)**- Shard by item-id range or hash to balance size (~10M items -> 100 shards of 100k each) and fit memory/GPU.- Each shard holds IVF-PQ or HNSW index depending on latency/accuracy trade-off; use IVF-PQ on CPU for cost, HNSW on GPU for low-latency p99.- Query fan-out: route user query to subset of shards (top-K per shard) and merge; parallelize across GPUs/servers.

**Serving architecture**- API tier (stateless) receives requests → fetch latest user embedding from cache/feature store → fan-out ANN queries to shard pool → merge & re-rank with business model (scoring model, filters) → ranker + diversifier → return top-10.- Re-ranking uses heavier models (cross-attention) but only on top-100 candidates.

**Caching & consistency**- Multi-level cache: L1 in-app LRU for top-users; L2 Redis for user embeddings and recent top-10 results (TTL short, e.g., 30s–5m depending on freshness).- Stronger consistency for embeddings: updates written atomically; use versioned embeddings (timestamp) so serving chooses newest ≤5min.- Eventual consistency acceptable for recall; strict for hard filters.

**Cold-start**- New users: use content-based or demographic priors + session-based short-term embeddings (recent session encoder) and popularity seeding.- New items: content embeddings from metadata/vision/text model immediately indexed with lower-weight until interaction data accrues.

**Periodic re-embedding**- Use incremental re-training for active users/items; full re-embedding weekly. Use rolling index rebuilds to avoid downtime; maintain old and new indices and switch when ready.

**Latency vs freshness trade-offs**- Freshness demands push more real-time updates (higher cost). To meet p99 <150ms: - Favor GPU HNSW shards for query speed. - Use short-lived caches to avoid repetitive ANN hits. - If freshness-critical, accept smaller candidate set or simpler re-ranker to stay within latency.- Provide SLO tiers: ultra-fresh (within 1–5min) for active users; relaxed for long-tail.

**Metrics & monitoring**- Track QPS, p50/p95/p99 latency, recall@K, MRR, freshness lag distribution, index staleness, and costs. Use canary experiments for new embeddings.

MediumTechnical

33 practiced

Write a Python function that computes the Population Stability Index (PSI) between two numeric arrays representing a reference distribution and a current distribution for a single feature. The function should accept arrays or lists of numeric values and either a configurable number of equal-frequency or equal-width bins or explicit bin edges, handle zero counts safely (e.g., smoothing), and return the PSI score. Use numpy and describe the algorithmic complexity.

Sample Answer

**Brief approach**

Compute histogram counts for reference and current over shared bins (either explicit edges, equal-width, or equal-frequency/quantile bins). Convert counts to proportions, apply small smoothing to avoid log(0) or division by zero, then compute PSI = sum( (ref_prop - cur_prop) * ln(ref_prop / cur_prop) ).

**Python implementation**

python

import numpy as np

def psi_score(ref, cur, bins=10, bin_type='equal_width', eps=1e-8):
    """
    Compute PSI between ref and cur arrays.
    - ref, cur: array-like numeric
    - bins: int (number of bins) or sequence (explicit bin edges)
    - bin_type: 'equal_width', 'equal_freq' (quantile), or ignored if bins is sequence
    - eps: smoothing added to proportions to avoid zeros
    Returns float PSI.
    """
    ref = np.asarray(ref)
    cur = np.asarray(cur)
    if np.isscalar(bins):
        if bin_type == 'equal_freq':
            # compute quantile-based edges (include extremes)
            q = np.linspace(0, 1, bins + 1)
            edges = np.unique(np.quantile(ref, q))
            if len(edges) <= 1:
                # degenerate: use small perturbation around values
                edges = np.array([ref.min() - 1e-6, ref.max() + 1e-6])
        else:  # equal_width
            edges = np.linspace(min(ref.min(), cur.min()), max(ref.max(), cur.max()), bins + 1)
    else:
        edges = np.asarray(bins)

    # compute counts; rightmost edge included in last bin
    ref_counts, _ = np.histogram(ref, bins=edges)
    cur_counts, _ = np.histogram(cur, bins=edges)

    # convert to proportions with smoothing
    ref_prop = (ref_counts + eps) / (ref_counts.sum() + eps * len(ref_counts))
    cur_prop = (cur_counts + eps) / (cur_counts.sum() + eps * len(cur_counts))

    # PSI per bin and total
    psi_bins = (ref_prop - cur_prop) * np.log(ref_prop / cur_prop)
    return float(np.sum(psi_bins))

**Key points / reasoning**

- Smoothing eps prevents division by zero and extreme log terms; eps ~ 1e-8 is typical.- Equal-frequency uses quantiles of reference to keep reference mass even across bins; equal-width uses global min/max to compare ranges.- If bins provided as edges, bin_type ignored.

**Complexity**

- Counting with numpy.histogram is O(n) for scanning arrays.- If equal-frequency is requested, computing quantiles is O(n log n) typically (depends on algorithm), so overall O(n log n). For equal-width or explicit edges complexity is O(n). Memory O(b + n) where b is number of bins.

MediumSystem Design

44 practiced

Design a streaming ingestion and feature computation pipeline that handles out-of-order events, deduplication, late arrivals, and schema evolution. Specify the message bus, stream processing engine, state store, windowing strategy and watermarking, exactly-once semantics considerations, and how you would test and validate correctness while ensuring scalability and low operational overhead.

Sample Answer

**Approach overview**I would build an event-time, stateful pipeline using Kafka (message bus) + Apache Flink (stream processing) + RocksDB (state store). This combination gives strong event-time semantics, low-latency stateful ops, and mature exactly-once integrations — appropriate for an applied scientist moving prototypes to production.

**Message bus**- Kafka with topic partitioning by entity id; use compacted topics for entities and time-partitioned topics for events.- Avro/Protobuf with a Schema Registry to support schema evolution (backward/forward compatibility).

**Stream processing & state**- Apache Flink for event-time processing, asynchronous checkpoints, and exactly-once sinks.- Embedded RocksDB for large local keyed state, enabling efficient lookups for features and dedupe state.

**Windowing & watermarking**- Use event-time tumbling/ sliding windows depending on feature (e.g., 1h tumbling for aggregates, 5m sliding for recency signals).- Watermarks: adaptive watermarks with maxOutOfOrderness tuned per stream (e.g., 30s–5m) and side-output late events for reprocessing.- For very-late arrivals, write to a correction topic and run periodic backfill/merge jobs.

**Deduplication & late arrivals**- Dedup via maintained per-key dedupe table with event-id TTL keyed in RocksDB; use event timestamps to accept only first-seen within allowed window.- Late arrivals: accept if within lateness threshold; emit correction records (delta updates) and maintain idempotent feature writes.

**Exactly-once**- Use Flink’s checkpointing + Kafka transactions (two-phase commit sink) for end-to-end exactly-once to feature store (e.g., Kafka output topic or transactional DB).- For external sinks without transactions, use idempotent writes with deterministic upserts and versioning (event-time + sequence numbers).

**Schema evolution**- Enforce schema compatibility via Schema Registry; Flink side: use compiled Avro/Protobuf with fallback to generic records; handle missing fields with defaults and feature extraction guards.

**Testing & validation**- Unit tests for feature logic with simulated out-of-order streams.- Property-based tests asserting invariants (commutativity, idempotence).- End-to-end integration tests using Testcontainers Kafka + Flink mini-cluster.- Chaos tests: inject duplicates, replays, clock skews, and broker restarts.- Data quality: run continuous validation jobs comparing batch-aggregates vs streaming aggregates; monitor drift, completeness, and late-count metrics.

**Scalability & operational overhead**- Partition by high-cardinality key, autoscale Flink task managers, use RocksDB local state with incremental checkpoints to S3/GCS.- Prefer managed services (Confluent Cloud / Flink on Kubernetes with operators) to reduce ops.- Monitoring: Prometheus/Grafana, alert on checkpoint latency, watermark lag, late-event ratio, and dedupe table growth.- Cost/latency trade-offs: tune watermark delay vs correctness; recommend conservative lateness for critical features and periodic batch reconciliation for weak-consistency features.

Example: compute 1-hour active-user count — use 1h tumbling window + 5m watermark + dedupe by user-event-id TTL 2h; on very-late arrivals, emit correction deltas to downstream model retraining job.

This design balances correctness (event-time, dedupe, exactly-once), schema safety, testability, and operational scalability suitable for ML feature pipelines.

HardSystem Design

29 practiced

Design a reproducible experimentation platform and artifact pipeline for the applied science team. Include dataset versioning and provenance, experiment metadata, containerized execution environments, artifact signing and storage, ML metadata store, linking datasets to model artifacts, reproducible training pipelines, and tooling for rerunning historic experiments. Address storage costs and retention policies.

Sample Answer

**Clarify goals & constraints**- Reproducible experiments end-to-end: dataset + code + env + config → signed model artifact + provenance, cheap long-term storage, ability to rerun exactly, auditability.

**High-level architecture**- Dataset versioning: Delta Lake / Iceberg + content-addressable object store (S3) + data hashes. Small datasets or pointers tracked with DVC / Git-LFS style manifests. Every dataset version = manifest with file-level checksums and partition metadata.- Execution env: containerized images (Docker/OCI), built from Dockerfile + deterministic base image + dependency lockfiles. Images pushed to private registry (ECR/GCR).- Orchestration: Kubeflow Pipelines / Argo Workflows to define reproducible DAGs as code. Pipelines accept dataset manifest + hyperparams.- ML metadata store: ML Metadata (MLMD) or Vertex/MLflow for experiment metadata (inputs, params, metrics, artifact URIs, lineage pointers).- Artifact store & signing: models and artifacts stored in object store with content-addressable keys; sign artifacts using cosign/sigstore and store signatures alongside artifacts and in MLMD.- Provenance linking: every run writes a run record linking dataset manifest id, container image digest, pipeline version (git commit), hyperparams, code snapshot (git commit or artifact), and output artifact hash.- Rerun tooling: CLI and UI to instantiate historic run by fetching manifest, image digest, code commit and re-submitting pipeline. Use reproducible build steps: verify dataset checksums, verify image signatures, optionally run in ephemeral namespace.- Cost & retention: tiered storage lifecycle policies (hot = 30d, warm = 1y, cold/archival = Glacier/Archive). Keep MLMD and minimal manifests always (cheap DB). Retain signatures + manifests indefinitely; large model binaries moved to cold/archive after N months. Allow policy overrides per experiment (e.g., research vs. production).- Security & governance: IAM controls, immutable logs, artifact signing, checksum verification before rerun.

**Why this works**- Content-addressable storage + manifests ensure bit-for-bit reproducibility.- Container digests + signed artifacts guarantee execution integrity.- MLMD provides queryable lineage to answer “which dataset+code produced artifact X”.- Tiered retention balances auditability vs storage cost; cheap manifests + metadata permit reconstructing runs without keeping all binaries hot.

Example: training run record stores: dataset_manifest_id = d12345, dataset_hashes.json, container_digest = sha256:..., code_commit = abcdef, pipeline_id = v2.1, artifact_uri = s3://models/sha256:..., signature_uri = s3://signatures/..., metrics = {...}. Rerun CLI pulls those and re-submits pipeline, verifying signatures and checksums first.

Unlock Full Question Bank

Get access to hundreds of End-to-End ML System Design interview questions and detailed answers.

Join thousands of developers preparing for their dream job.