AI and Machine Learning Background Questions

A synopsis of applied artificial intelligence and machine learning experience including models, frameworks, and pipelines used, datasets and scale, production deployment experience, evaluation metrics, and measurable business outcomes. Candidates should describe specific projects, roles played, research versus production distinctions, and technical choices and trade offs.

HardTechnical

83 practiced

Write an optimized ANSI SQL query (and describe the approach) to compute the 95th percentile of session durations per user over the past 90 days from a sessions table containing billions of records: sessions(session_id, user_id, start_ts, end_ts, region). Explain indexing, partitioning, and approximation algorithms (for example t-digest or approx_percentile) you would use to make this computation performant in production and how you'd handle late-arriving sessions.

Sample Answer

Approach (brief): compute per-user 95th percentile of session_duration = end_ts - start_ts over last 90 days. For correctness show exact ANSI SQL using PERCENTILE_CONT (window aggregate) and then a production-friendly approximate version using an aggregate APPROX_PERCENTILE/TDIGEST (common in Snowflake/BigQuery/Presto/Spark). In production, pre-aggregate daily per-user histograms / t-digests, partition the base table by start_date (and optionally region), index on (user_id, start_date), and run incremental rollups to avoid scanning billions of rows.

Exact ANSI SQL (heavy; not recommended at scale):

sql

WITH recent AS (
  SELECT user_id,
         EXTRACT(EPOCH FROM (end_ts - start_ts)) AS session_seconds
  FROM sessions
  WHERE start_ts >= CURRENT_DATE - INTERVAL '90' DAY
)
SELECT user_id,
       PERCENTILE_CONT(0.95) WITHIN GROUP (ORDER BY session_seconds) AS p95_seconds
FROM recent
GROUP BY user_id;

Approximate / production pattern (faster; uses engine approx fn or t-digest UDAF):

sql

-- Example for engines supporting approx_percentile or tdigest_agg
WITH recent AS (
  SELECT user_id,
         EXTRACT(EPOCH FROM (end_ts - start_ts)) AS session_seconds
  FROM sessions
  WHERE start_ts >= CURRENT_DATE - INTERVAL '90' DAY
)
SELECT user_id,
       approx_percentile(session_seconds, 0.95) AS p95_seconds
FROM recent
GROUP BY user_id;

Production design and optimizations:- Partitioning: partition sessions by start_date (daily) and optionally by region to limit scans. Use partition pruning for the 90-day filter.- Indexing / clustering: clustered index or sort key on (start_date, user_id) so queries for recent dates and per-user groups are efficient. In columnar stores, use clustering keys on user_id and start_date.- Pre-aggregation: compute daily per-user sketches (t-digest or quantile sketch) and store as a lightweight table: daily_user_digest(day, user_id, tdigest). To compute 90-day p95, merge 90 digests and extract 0.95 — merge is O(size of sketch) not O(rows).- Approximation algorithms: t-digest or DDSketch are ideal for skewed distributions and merging across partitions. Many engines provide approx_percentile() or TDIGEST_AGG() UDAFs. Choose compression parameter to trade accuracy vs size (e.g., delta ~ 100-200).- Resource/parallelism: compute per-user aggregates in distributed engine (Spark/Presto) and write results to a materialized table refreshed daily/hourly.

Handling late-arriving sessions:- Watermarking & SLA window: allow a small lateness window for streaming ingestion (e.g., 24h). For batch, reprocess / upsert the pre-aggregated daily digests for the affected event_date.- Idempotent rollups: produce immutable daily files (by ingestion partition + attempt id) and maintain a reconciliation job that detects and recomputes days with corrections.- Backfill / re-materialize: keep a rolling re-compute job that re-merges digests for the last N days (e.g., 7-14 days) to capture late events; for longer tail, periodically full-recompute or run targeted backfills.- Audit & monitoring: record row counts and watermark tests; if late volume exceeds threshold, trigger re-aggregation.

Trade-offs:- Exact percentile is precise but expensive on billions of rows.- Approximate (t-digest) yields sub-percentile errors with big performance gains and easy merge semantics — recommended for production per-user quantiles at scale.

MediumSystem Design

74 practiced

Design an ML training pipeline orchestration using Airflow (or an equivalent orchestrator) that retrains a model weekly: include tasks to materialize training features, run data/feature validations, start training with a locked dataset snapshot, compute validation metrics, register the model in a registry, and trigger a canary deployment if metrics meet thresholds. Describe DAG structure, idempotency, lineage capture, and failure/retry handling.

Sample Answer

Requirements (clarify): weekly retrain; reproducible snapshot of training data; validate data/features; compute metrics; register model; trigger canary deployment only when metrics pass thresholds; capture lineage; be idempotent and resilient.

High-level DAG- schedule: cron weekly- DAG tasks (linear with parallelizable validation): 1. materialize_features (Spark/Beam job using Dataflow/EMR/K8s): generate feature parquet files into versioned path s3://features/{date}/{run_id}/ 2. validate_data (Great Expectations / TFDV): run on raw and feature outputs; fail fast on schema/drift break 3. snapshot_dataset (atomic copy or manifest creation): compute dataset hash, create immutable snapshot pointer (manifest.json) and write to versioned storage; record snapshot_id 4. train_model (KubernetesPodOperator or EMR step): runs training using snapshot_id; use container image pinned by digest 5. evaluate_metrics (unit tests + holdout eval): compute metrics, sign with snapshot_id and model artifact id 6. register_model (MLflow or Model Registry): transactional register with metadata (model version, snapshot_id, metrics) 7. canary_gate (BranchOperator): if metrics >= thresholds -> trigger canary_deploy; else -> end with alert 8. canary_deploy (trigger CI/CD job like Argo Rollouts/Spinnaker): roll out to small % and attach monitoring pipeline 9. post_deploy_monitoring (async): monitor SLI; if regressions -> rollback trigger

Idempotency & determinism- Every run has run_id and snapshot_id (dataset hash + timestamp). Tasks write to paths derived from these ids. Operators are idempotent: feature materialization writes to temp path then atomic rename; snapshot creation fails if manifest exists for snapshot_id.- Training uses immutable data snapshot and pinned image + seed for RNG to ensure reproducible outputs.- Use upsert semantics for registry (check exists by fingerprint before creating new version).

Lineage & metadata capture- Emit OpenLineage/Marquez events at each task boundary: inputs, outputs, job run, dataset versions, schema.- Persist metadata to catalog (Data Catalog / Glue / BigQuery) with links: snapshot_id, feature paths, model_version, metrics.

Failure / retry handling- Airflow retries with exponential backoff for transient failures; set sensible retry counts per task.- Validation failures are terminal for pipeline: fail fast and notify owners via Slack/email; store failing artifact in quarantine area.- Long-running steps use heartbeat and SLAs; on_failure_callback triggers a runbook and creates PagerDuty incident if critical.- For downstream idempotency after partial success: tasks check for existence of their outputs (manifests, model artifacts) and skip if completed and signatures match.- Canary failure rollbacks handled by CI/CD (automated rollback policy) and a manual override path.

Operational considerations- Secrets via Vault/KMS; logs/metrics to centralized observability (Prometheus/Grafana, ELK).- Tests: smoke tests in canary, shadow traffic for further validation.- Security/compliance: immutable snapshots for audit, retention policies for old snapshots/models.

This design balances reproducibility, safety (validation + canary), observability (lineage/metrics), and operational robustness (idempotency and retry semantics).

MediumTechnical

70 practiced

You must compute rolling model performance metrics (daily precision and recall) for multiple models across cohorts and publish them to dashboards. Explain the data modeling and aggregation pipelines you would implement, how you would handle late-arriving labels, and how you would compute accurate time-series metrics with lookback windows. Include how you would validate metric correctness.

Sample Answer

Approach summary- Keep prediction and label events as immutable time-series facts (prediction_fact, label_fact) with event_time, model_id, model_version, cohort_keys, prediction_score, prediction_threshold, ingest_time.- Compute daily precision/recall by event_date (based on prediction event_time) so metrics reflect when the model acted.

Data modeling & pipelines1. Raw ingestion: append-only parquet/Delta tables partitioned by event_date and model_id. Store both prediction and label streams with ingest_time.2. Enrichment/joins: A nightly Spark job (or streaming join) joins predictions to labels using a prediction_id or primary keys; use event_time for the prediction and label_time for labels. Persist joined records to a metrics_fact table with fields: prediction_date, label_date, model_id, cohort, prediction_binary, label, ingest_time, last_update_time.3. Aggregation: Aggregate metrics_fact into daily_model_cohort_metrics (precision, recall, counts, denominators) with windows keyed by prediction_date + cohort + model_version. Use incremental upserts (Delta/BigQuery MERGE) so re-computations replace prior values.

Handling late-arriving labels- Use event-time joins with watermarks for streaming (Spark Structured Streaming) to handle typical lateness (e.g., allowed lateness = 7 days). For labels arriving after watermark: - Persist late labels to a corrections table and trigger a backfill/recompute job for affected prediction_date(s). - Make metrics upserts idempotent: when recomputing a prediction_date window, MERGE new aggregates to overwrite prior values.- Maintain an as_of or revision column on metrics so dashboards can show finalized vs provisional metrics and the revision timestamp.

Accurate time-series with lookback windows- Define lookback semantics explicitly: e.g., recall@7d = fraction of positives observed within 7 days of prediction_date. Implement by filtering labels where label_time <= prediction_time + lookback and label_time >= prediction_time.- For each prediction_date compute multiple rolling metrics (1d, 7d, 14d) by aggregating label arrivals within those windows. Store separate columns for each lookback to avoid re-compute on dashboard.

Implementation notes- Use Spark batch for daily finalization; Structured Streaming for near-real-time with watermarks and update mode writing to Delta.- Partition metrics by prediction_date and model_id; keep small cardinality for cohorts or use pre-aggregation if many cohorts.

Validation & correctness- Unit tests for aggregation logic with synthetic data (edge cases: duplicates, missing labels).- Backtest: run a full-batch recompute for a sample period and compare with incremental/stream outputs; diff should be zero after backfill.- Shadow/Canary runs: publish metrics to internal dashboard before public release.- Monitoring: alerts for data gaps, sudden metric changes, or high correction volumes. Track label latency histogram and percent finalization by day (what % of day T is finalized after N days).- Spot checks: sample prediction ids, recompute truth offline to confirm precision/recall.- Auditable lineage: keep raw facts and aggregated SQL for reproducibility.

This design yields correct, auditable rolling metrics, handles late labels by controlled backfills and versioning, and exposes provisional vs finalized metrics so downstream consumers know when values are stable.

HardSystem Design

110 practiced

Design a multi-region ML data pipeline where raw PII must remain in the user's home region but aggregated or anonymized models can be globally accessed. Explain data partitioning, federated training vs central training with differential privacy, feature aggregation across regions, and how to safely distribute global model artifacts while satisfying data residency and latency requirements.

Sample Answer

Requirements & constraints- Raw PII must never leave user's home region (data residency).- Global access only to aggregated/anonymized features or model artifacts.- Low-latency inference regionally; periodic global updates.- Compliance, auditability, key management, and secure distribution.

High-level architecture1. Per-region data plane: ingestion → local raw store (encrypted) → regional feature pipeline → local model trainer / aggregator.2. Global control plane: orchestrates federated rounds, aggregates updates, stores global model artifacts (only non-PII).3. Secure model distribution CDN with signed artifacts and ABAC/SCIM for access control.

Data partitioning- Logical partition: data lives and is processed in the user's home region cluster (e.g., AWS region accounts or GCP projects).- Use tenant- and region-keyed buckets/tables (region_id, tenant_id) to enforce isolation.- Metadata index (no PII) replicated globally (schema, dataset availability, lineage pointers) to enable discovery without moving raw data.

Federated training vs central training with DP- Prefer federated learning (FL) for strict residency: each region trains local model on local PII, produces model weight deltas or encrypted gradients. - Local steps: data preprocessing and feature extraction run on regional Spark/Kube jobs; local trainer (PyTorch/TensorFlow) performs epochs and outputs model update. - Secure aggregation: use secure aggregation (multi-party) so server sees only aggregated update; combine with Homomorphic Encryption or MPC where needed. - Add Differential Privacy (DP) at region level (per-update clipping + noise) to bound contribution of any individual.- Central training with DP is alternative when some regions permit export of pseudonymized aggregates: regions export pre-aggregated, differentially-private feature statistics to central store; central trainer trains on aggregated stats, reducing complexity but losing per-sample fidelity.- Trade-offs: FL + secure aggregation + DP maximizes privacy/residency but higher complexity and communication. Central DP is simpler but requires trusting DP guarantees and that aggregates are sufficient.

Feature aggregation across regions- Shared feature schema & feature store metadata globally (no raw values). Implement regional feature stores (Feast-like) that register online/offline features with global registry.- For cross-region aggregated features (e.g., global counts), compute locally and publish DP-noised aggregates to a global aggregated store. Use time-windowed aggregation and provenance tags.- Use consistent feature transforms: package preprocess code as versioned containers/functions deployed per region so transforms are identical.

Safe distribution of global model artifacts- Artifacts contain only model weights and non-PII metadata. Sign artifacts with CI/CD keys and store in an artifact repository + regional mirrors (CDN + S3 cross-region replication of encrypted objects).- Enforce access controls: only region compute roles can pull global model. Use short-lived credentials, mutual TLS, and artifact verification (signature + checksum).- For low-latency inference, push model into regional serving endpoints (KFServing/TF-Serving) from the nearest mirror. Optionally quantize/compile for edge.- For online personalization, keep small regional fine-tuning layers that adapt global model to local distribution—trained only on local data.

Compliance, security, and operations- Encryption: KMS per region with separate keys. Global aggregator never has access to region KMS keys.- Audit/logging: Immutable audit logs in-region; replicate non-PII audit indices for compliance reporting.- Data governance: Policy engine (OPA) enforces who can start training rounds, request aggregates, and distribute artifacts.- Monitoring: model drift, contribution metrics, anomaly detection on updates (to detect data poisoning).- Testing & verification: replay/repro tests in sandbox regions; DP parameter tuning with privacy accountant to certify epsilon.

Failure modes & mitigations- Stragglers: asynchronous aggregation or tolerate partial rounds with adaptive weighting.- Poisoning: robust aggregation (median/Krum), anomaly detection, and quarantine of suspicious regions.- Network partitions: local serving continues with last-known global model; queue updates.

SummaryKeep raw PII and preprocessing inside each region; share only DP-noised, aggregated statistics or securely aggregated model updates. Use a global registry and artifact signing for safe distribution. Choose FL + secure aggregation + DP when strict residency; central DP aggregation can be an operationally simpler fallback where allowed.

MediumTechnical

77 practiced

Compare three model serving architectures for inference at scale: (1) single-model REST microservice per model, (2) multi-model server (e.g., TorchServe, Triton) hosting many models, and (3) serverless functions (e.g., AWS Lambda). For each approach, discuss cold-start behavior, autoscaling characteristics, model loading time, resource utilization, operational complexity, and the implications for a data engineer responsible for feature delivery and monitoring.

Sample Answer

Short summary: each architecture trades isolation, startup latency, and operational burden differently. Below I compare on the requested dimensions and close with practical implications for a data engineer.

1) Single-model REST microservice per model- Cold-start: high if container needs to spin up; warm services have near-zero inference latency. Can use long-lived pods to avoid cold starts.- Autoscaling: scales per model (horizontal pods); predictable but many scaling groups if many models.- Model loading time: fast once service is running (model baked into image or loaded at startup); can be slow on first deploy if large model included.- Resource utilization: potentially inefficient if many rarely-used models each hold memory/CPU; good isolation avoids noisy neighbors.- Operational complexity: higher operational overhead managing many services, CI/CD for each model, routing.- Data-engineer implications: must ensure feature pipelines map to model endpoints, coordinate deployments with model schema, track per-model metrics; may need cataloging and routing layer.

2) Multi-model server (Triton, TorchServe)- Cold-start: better — server runs continuously; loading a new model might be asynchronous and reasonably quick, but first inference after load suffers.- Autoscaling: scale server instances rather than models; need orchestration logic to decide when to scale up/down and when to pre-load models.- Model loading time: variable (seconds–minutes) depending on model size and framework; some servers support lazy loading and model-versioning.- Resource utilization: efficient if many models share runtime and memory; can pack GPUs/CPU across models but introduces contention.- Operational complexity: moderate — one platform to manage, but requires logic for model placement, versioning, routing, and resource isolation.- Data-engineer implications: easier single integration point for feature delivery and monitoring; must manage model lifecycle, ensure model-to-feature compatibility, and monitor shared resource contention and tail latency.

3) Serverless functions (Lambda / Functions)- Cold-start: high and unpredictable for large models; cold starts include container + model load, often unacceptable for low-latency inference unless using small models or provisioned concurrency.- Autoscaling: automatic and elastic per request, excellent for bursty workloads; can be cost-efficient for infrequent traffic.- Model loading time: must either package model in function (increases cold start) or fetch from storage at init (slower); limited / ephemeral local storage complicates large models.- Resource utilization: very efficient for spiky, low-throughput use; poor for sustained high-throughput due to duplicated hot memory across many concurrent instances.- Operational complexity: low infra ops (managed), but higher complexity around packaging models, permissions, and warming strategies.- Data-engineer implications: need to design feature delivery with low-latency caches, ensure secure model storage/access, implement warming/provisioned concurrency strategies, and instrument distributed logs/metrics across ephemeral instances.

Practical recommendation for a Data Engineer:- If you operate many models with steady traffic, use a multi-model server to centralize deployment, simplify telemetry, and optimize resource sharing.- If strict isolation and predictable latency per model are required, use single-model services despite higher operational cost.- For bursty or low-traffic models, serverless can be cost-effective but requires engineering for cold-start mitigation and feature caching.Operational priorities: implement centralized model & feature registry, consistent schema/versioning, end-to-end metrics (latency, error-rate, feature drift), and automated alerts for model load failures and resource contention.

Unlock Full Question Bank

Get access to hundreds of AI and Machine Learning Background interview questions and detailed answers.

Join thousands of developers preparing for their dream job.