Data Organization and Infrastructure Challenges Questions

Demonstrate knowledge of the technical and operational problems faced by large scale data and machine learning teams, including data infrastructure scaling, data quality and governance, model deployment and monitoring in production, MLOps practices, technical debt, standardization across teams, balancing experimentation with reliability, and responsible artificial intelligence considerations. Discuss relevant tooling, architectures, monitoring strategies, trade offs between innovation and stability, and examples of how to operationalize models and data products at scale.

MediumSystem Design

38 practiced

Write a short design (no code required) for a metadata search service for ML assets that supports queries like 'find features computed from purchase events with freshness < 1 hour and owner in payments team'. Describe indices, metadata fields, and access-control considerations.

Sample Answer

Requirements:- Functional: text & faceted search over ML assets (features, datasets, models, pipelines). Support queries like “features computed from purchase events with freshness <1 hour and owner in payments team”.- Non-functional: low-latency (<200ms) for typical queries, fresh metadata within minutes, RBAC-compliant.

High-level architecture:- Metadata ingestion pipeline (connectors from feature store, data warehouse, model registry) → Metadata store (primary writable DB) → Search index (Elasticsearch/Opensearch) + cache → API layer + AuthZ service.

Metadata fields (per asset):- id, name, type (feature/dataset/model), description, owner (user/team), tags, lineage_sources (events/datasets), transformation_type (batch/stream), event_topic (e.g., purchase_events), freshness_seconds, last_updated, SLA, sensitivity_level, access_policy_id, created_at.

Indices & mappings:- Text fields: name, description, tags — full-text analyzer with n-grams for fuzzy match.- Keyword fields: type, owner, team, transformation_type, event_topic — exact matches and aggregations.- Numeric/date: freshness_seconds, last_updated — range queries.- Nested: lineage_sources and provenance entries (source_type, source_id).- ACLs: store access_policy_id and precompute allowed_principals per document for fast filtering (see access control).

Query example resolution:- Translate “computed from purchase events” → filter event_topic:purchase_events or lineage_sources contains source_id matching purchase topic.- “freshness < 1 hour” → freshness_seconds < 3600.- “owner in payments team” → owner_team:payments.

Access-control considerations:- Central AuthZ service holding policies (role/team-based + attribute-based). On index write, expand policies to allowed_principals list or tags used for policy decisions.- Enforce at API layer: authenticate user → request allowed_principals → append filter to search query (allowed_principals contains user or user_team) to avoid leaking fields.- For sensitive assets, mask summary fields in index and require direct metadata DB fetch with elevated permission.- Audit logging for queries and access.

Scalability & trade-offs:- Denormalize some policy info into index for fast filtering (faster reads, more writes).- Use incremental reindexing for schema changes; periodic full sync for drift.- Cache frequent queries and aggregations at API layer.

HardSystem Design

41 practiced

Design a streaming feature computation pipeline that guarantees exactly-once semantics for a stateful, real-time model. Describe the role of Kafka (or similar), Flink or Spark Structured Streaming, state management, checkpointing, and handling of late events and replays.

Sample Answer

Requirements & constraints:- Stateful, low-latency feature computation for a real-time model.- End-to-end exactly-once semantics (input → state → output).- Support late arrivals and ability to replay historical data.

High-level architecture:- Kafka (or Pulsar) as durable ordered event log and source-of-truth for inputs and for emitting computed features.- Stream processor: Apache Flink (preferred) or Spark Structured Streaming (micro-batch with idempotent sinks).- State backend: RocksDB (embedded) with durable checkpointing to S3/HDFS.- Sinks: Kafka topics using transactional writes or idempotent/upsert stores (e.g., Kafka compacted topic, transactional DB, or feature store with upsert API).

Key components & responsibilities:1. Kafka:- Retains raw events and supports replay by offsets/partitions.- Producer sets keys to maintain partitioning for locality (e.g., entity_id).2. Stream Processing (Flink preferred):- Processes events event-time with watermarks.- Maintains keyed operator state for per-entity features in RocksDB.- Uses Flink’s two-phase checkpointing and distributed snapshots to persist state + source offsets atomically.3. State management & checkpointing:- Periodic durable checkpoints to object storage. Checkpoint includes operator state and source offsets; on failure, job restarts from latest checkpoint guaranteeing no-double-processing.- Use exactly-once sinks: Flink’s TwoPhaseCommitSink or Kafka transactions so outputs are committed only after checkpoint is complete.4. Late events & watermarks:- Define watermark strategy based on SLA and observed lateness.- Use allowed-lateness window to update state if late but within bound; emit corrections downstream (delta updates).- For extremely late events, route to side-output for manual backfill or to an out-of-order handling job that merges corrections.5. Replays:- To rebuild state, trigger a job reading Kafka from earliest offsets or from compacted snapshot; reconstruct state deterministically using same logic.- Use schema versioning and idempotent update logic so replays produce the same final state.6. Model & feature consistency:- Serve features from the same durable store used by sinks (compacted Kafka topic or feature store) for transactional visibility.- When updating feature computation logic, do staged rollout: run dual pipelines (old + new), compare, then cutover. Keep migration records.

Trade-offs & practical notes:- Flink gives stronger low-latency exactly-once semantics via distributed snapshots; Spark Structured Streaming can do idempotent/atomic writes but has higher latency.- RocksDB + S3 checkpoints scale for large state; watch checkpoint frequency vs. throughput.- Designing for allowed-lateness requires product decisions: longer windows increase complexity and state size.- Monitor lag, checkpoint duration, and transaction commit latency. Add end-to-end testing (fault injection, replays) to validate exactly-once.

Outcome:This design ensures deterministic, exactly-once feature computation by coupling durable event logs, transactional output commits, and consistent state checkpoints; late arrivals and replays are managed with bounded lateness, correction streams, and deterministic rebuilds.

HardSystem Design

44 practiced

Design a multi-region, low-latency model serving architecture that must provide median inference latency under 50ms and handle 5k QPS per region. Include model distribution, cache strategy, consistency considerations, failover, and strategies for A/B and canary testing.

Sample Answer

Requirements & constraints:- Median p95 inference < 50ms, 5k QPS per region, multi-region (geo), high-availability, safe canary/A-B.

High-level architecture:- Front door: global load balancer (GeoDNS / Cloud LB) routes to nearest region; health-aware routing.- Per-region serving tier: API gateways → inference fleet (autoscaling Kubernetes nodes with GPU/CPU pods) behind regional internal LB.- Model registry & CI/CD: models packaged as immutable artifacts (container or torchscript) stored in registry with semantic versioning.- Model distribution: control plane pushes new model artifacts to each region proactively (pull from registry to local artifact cache) and verifies checksum.

Latency & scaling:- Keep model warm: use small pool of pre-warmed replicas per model to avoid cold-starts.- Horizontal autoscale based on latency & queue depth; use fast binaries (TorchServe/Triton) with batching tuned to meet 50ms median.

Cache strategy:- Two-layer cache: - Local in-process LRU cache for hottest inputs (microsecond lookup) for idempotent requests. - Regional Redis/Memcached for shared cache across nodes (TTL + size limits). Cache keys include model-version and input hash.- Cache invalidation: TTL + model-versioned keys to ensure simple strong isolation when rolling models.

Consistency:- Use model-versioned APIs (/predict?model=vX) so clients select exact model; control-plane supports atomic switch by updating traffic routing instead of overriding existing version.- Accept eventual consistency for caches; critical correctness uses model-versioned lookup to avoid stale-model inference.

Failover & resilience:- Health checks and circuit breakers; if regional capacity exceeded, global LB shifts traffic to next region with capacity.- Graceful degradation: fallback to cheaper model (smaller, CPU) or cached responses when latency spikes.- Store logs/telemetry in regional sinks and replicate important signals to central monitoring; enable rapid rollback via control plane.

A/B & canary testing:- Canary: deploy new model to small % of instances in each region (or single region). Use traffic-splitting at API gateway (weighted) and shadow mode (send copies to canary without affecting responses) to compute offline metrics.- A/B experiments: route users via feature-flag service; collect metrics (latency, accuracy, business metrics) with statistical tests (power analysis) and monitor for regressions.- Automated rollback: metric thresholds (latency increase, error rate, degradation in business KPIs) trigger immediate rollback and alert.

Monitoring & SLOs:- Track median, p95 latency, error rates, model quality (drift, accuracy), and traffic per model-version. Use tracing for tail latency analysis.

Trade-offs:- Local caches reduce latency but increase memory; model-versioning simplifies consistency but increases deployment complexity.- Cross-region failover increases latency for affected users but maintains availability.

This design ensures sub-50ms median by regional, warmed, autoscaled fleets plus local caches; safe rollouts and strong observability allow reliable A/B and canary experimentation.

MediumTechnical

32 practiced

Describe a data governance policy to minimize PII exposure for ML teams. Explain how to implement least-privilege access controls, masking or tokenization, and an audit trail for dataset access while maintaining analyst productivity.

Sample Answer

Situation: At my last role we needed ML features that used customer data while reducing PII exposure and meeting compliance.

Policy overview: Define a data governance policy that classifies data (PII, sensitive, non-sensitive), enforces purpose-based access, and mandates deidentification/tokens and full auditing before any dataset leaves the secure environment.

Implementation — least-privilege access:- Role-based + attribute-based access: create coarse roles (researcher, engineer, analyst) and ABAC rules (project, dataset sensitivity, time-limited access).- Use short-lived credentials (IAM sessions) and require just-in-time approvals for sensitive datasets.- Example tech: AWS IAM + Lake Formation or GCP IAM + Data Catalog to grant table-/column-level permissions.

Masking / tokenization:- Use field-level deterministic tokenization for keys (preserves joins) and irreversible hashing/salting for true PII where joins aren’t needed.- Apply format-preserving encryption when downstream systems require realistic formats.- Implement transformation pipelines in a secure ETL (e.g., Glue, Dataproc) that produce de-identified feature tables labeled with lineage metadata.

Audit trail & monitoring:- Log dataset access at row/column level, record user, purpose, query, and time using centralized audit logs (CloudTrail + Lake Formation/GCP Access Transparency).- Automate alerts for unusual access patterns and require quarterly access reviews.

Maintaining analyst productivity:- Provide sanitized feature stores and synthetic datasets for exploration; offer self-service deid pipelines and a tokenization library so analysts can work without hitting approvals for every experiment.- Document approved data patterns and provide templates, notebooks, and a sandbox with pre-masked samples.

Result: This approach enforces least privilege, reduces PII exposure via tokenization, and preserves productivity by giving analysts safe, self-service data while retaining full auditability and compliance.

HardTechnical

37 practiced

Case study: A regulatory audit requires that every model prediction that affected a customer within the past year be explainable and reproducible. Design storage and retrieval for per-request explainability artifacts, a compute plan to regenerate explanations on demand, and how to ensure privacy and cost constraints are respected.

Sample Answer

Requirements & constraints:- Must retrieve any prediction made in past year with its explainability artifacts and be able to reproduce the prediction (same inputs, model version, random seeds) while minimizing cost and preserving privacy.

Approach (high-level):- Store minimal canonical artifacts at inference time to guarantee reproducibility, plus lightweight explainability artifacts; support on-demand regeneration if heavy artifacts weren’t stored. Enforce strict access control, encryption, retention/anonymization policies.

Storage & data model:- Inference event record (immutable): {request_id, timestamp, model_id+git_sha, model_container_image, model_weights_hash, feature_schema_version, input_hash (HMAC), encrypted_input_pointer, decision, probability, random_seed, runtime_env_metadata, explainability_manifest_pointer, provenance_signature}- Explainability manifest (small JSON): pointers to stored artifacts (saliency maps, SHAP values summary, feature contributions), summary explanation (text), storage flags (full vs summary)- Object store for heavy artifacts (S3/GCS) with lifecycle rules. Use compressed, possibly delta-encoded artifacts. Name objects by content-addressable hashes to avoid duplication.- Metadata index in low-latency DB (DynamoDB/Postgres) keyed by request_id and keyed secondary on customer_id + time for audit queries.

Compute plan for regeneration:- Two-tier strategy: 1. At inference: compute cheap, deterministic explanations (feature attributions using same deterministic algorithm with fixed seed, top-k features, natural-language summary). Persist manifest + small artifacts. 2. On-demand regeneration: if auditor requests full artifacts (or heavy visualizations) regenerate in an isolated, reproducible environment: - Use model image + weights hash + feature schema + input (either stored encrypted or reconstructed from input_hash via secure retrieval). - Run in ephemeral, versioned compute (Kubernetes job or serverless with pinned container and fixed seed). - Recompute explanation with configured algorithm/version; produce signed artifact and store with audit trace.- Cache regenerated artifacts for subsequent requests with TTL and size-based eviction.

Privacy & security:- Do not store raw PII in plaintext. Options: - Encrypt raw inputs at rest with KMS; store only encrypted pointers and HMACs for indexing. - When possible, store privacy-preserving feature vectors instead of raw data (tokenization, hashing, irreversible transforms) while ensuring reproducibility — keep mapping keys in secure vault. - Access governed by RBAC and just-in-time decryption with audited KMS access. All reads produce audit logs.- For auditor needing raw input, require approved workflow and record purpose; if disallowed, provide reconstructed synthetic inputs that match feature distributions while maintaining deterministic reproduction properties where allowed.

Cost controls:- Store minimal artifacts by default (summaries, top-k attributions). Use lifecycle rules: keep full artifacts for short window (e.g., 30–90 days) then delete or compress to cheaper tier; retain manifests and hashes for full year per requirement.- Deduplicate artifacts using content-addressable storage; compress and delta-encode artifacts across similar requests (e.g., identical inputs).- Prioritize deterministic, cheap explainers at inference. Regeneration is billed to audit requests; require justification/approval; batch regeneration where possible.- Use autoscaled spot/preemptible instances for heavy recompute with checkpointing to save cost.

Reproducibility & auditability:- Record exact model artifact hash, container image, random seed, dependency snapshot, and OS/runtime metadata in the immutable event record.- Sign records with system key to prevent tampering.- Include provenance chain: who requested, who executed regeneration, timestamps, compute environment IDs, logs, and artifact hashes.- Provide an API for auditors: query by customer_id+date range -> returns manifests + links -> allow regeneration with approvals; responses include signed assertions that the reproduced output matches original (or explain differences with deterministic diff).

Trade-offs & risks:- Storing raw inputs eases reproduction but increases privacy risk and cost — mitigate via encryption and strict access controls.- Full storage of heavy explainability artifacts for every request is expensive; two-tier strategy balances immediate auditability with cost.- Deterministic explainers limit stochastic methods; where randomized explainers are used, capture seed and run multiple repeats to ensure stability.

Example flow for an auditor request:1. Auditor queries request_id -> system returns manifest and whether full artifacts exist.2. If full artifacts exist, decrypt and provide with signed provenance.3. If not, auditor triggers regeneration; system checks approval, spins reproducible job using recorded model image + input (decrypted under audit policy), computes explanations, stores artifacts, returns results and records audit trail.

This design meets regulatory needs: every affected prediction has a retrievable, reproducible trail; privacy is protected via encryption and access controls; cost is managed via summarization, caching, deduplication, and controlled regeneration.

Unlock Full Question Bank

Get access to hundreds of Data Organization and Infrastructure Challenges interview questions and detailed answers.

Join thousands of developers preparing for their dream job.