Cloud Machine Learning Platforms and Infrastructure Questions
Knowledge of cloud hosted machine learning and artificial intelligence platforms and the supporting infrastructure used to develop, train, deploy, and operate models at scale. Candidates should be familiar with major managed offerings such as Amazon SageMaker, Google Cloud artificial intelligence platform, and Microsoft Azure Machine Learning and understand capabilities including pretrained models, managed training jobs, managed inference endpoints, model registries, and managed pipelines. Key areas include differences between cloud and local training, distributed and hardware accelerated training options, cost trade offs including spot and preemptible instances, serving patterns such as serverless inference, hosted endpoints and batch processing, autoscaling strategies for inference, model versioning and rollout strategies including canary and blue green deployments, integration with data storage, feature stores and data pipelines, and model monitoring, logging and drift detection. Candidates should also be able to explain when to use managed services versus self hosted or on premises solutions, discussing trade offs around productivity, operational overhead, control and customization, vendor lock in, security, data residency and compliance, as well as operational practices such as continuous integration and deployment for models, testing and validation in production, observability and cost optimization.
HardTechnical
87 practiced
Perform a threat model for a cloud ML platform that exposes an inference API and allows customers to upload training data and models. Identify likely attack vectors (model extraction, membership inference, poisoning, data exfiltration, privilege escalation) and propose mitigation strategies such as rate-limiting, differential privacy, model watermarking, input validation, RBAC and audit logging.
Sample Answer
Threat model summary — assets, actors, entry points- Assets: customer training data (sensitive PII), trained models (IP), inference API, model artifacts, keys/credentials, telemetry/logs.- Adversaries: external attackers, malicious tenants, curious insiders, compromised CI/CD, supply‑chain attackers.- Entry points: model upload, data upload, inference API, admin/UIs, build pipelines, storage buckets, metadata endpoints.Likely attack vectors & mitigations1) Model extraction (oracle access + adaptive queries)- Attack: query API to reconstruct model or mimic behavior.- Mitigations: per-tenant rate-limits and query budgets; anomaly detection on query patterns; limit output granularity (top‑k vs softmax); introduce controlled noise/response randomness; API-level fingerprinting and watermarking of model outputs; enforce paywall or tokenized access for high-volume use.- Trade-off: utility vs protection — too much noise degrades accuracy.2) Membership inference / privacy leakage- Attack: infer whether a specific record was in training data.- Mitigations: training-time differential privacy (DP‑SGD) tuned to acceptable epsilon; post-hoc output clipping/thresholding; per-request privacy budgets; returning fewer confidence details; provide privacy guarantees in SLA.- Trade-off: DP reduces model utility; pick epsilon based on risk and data sensitivity.3) Training-data/model poisoning- Attack: malicious uploads or poisoned data subtly manipulate model or create backdoors.- Mitigations: validate and sandbox user uploads; static and dynamic malware scanning; data provenance checks; anomaly detection during training (outlier detection, influence functions); model validation with holdout clean test sets; model signing and artifact immutability; human review for high-impact models.- Additional: isolate multi-tenant training environments (namespaces, VMs), limit shared pretraining.4) Data exfiltration (storage/API/backup)- Attack: read S3-like buckets, backups, logs to steal PII or models.- Mitigations: encrypt data at rest and in transit; KMS with tenant-specific keys; strict IAM/RBAC for storage and APIs; network controls (VPCs, private endpoints); disable public listing; rotate keys and use short-lived tokens.- Audit: comprehensive access logging, automated alerting on unusual downloads.5) Privilege escalation / lateral movement- Attack: leverage misconfigured IAM or vulnerable admin UI to gain broader access.- Mitigations: principle of least privilege, RBAC with separation of duties, MFA for admin accounts, just-in-time access, hardened bastion hosts, regular IAM reviews and automated policy scanning, container runtime hardening.- Monitor: abnormal activity detection and fast revocation workflows.Platform-wide controls- Input validation and sanitization on uploads to prevent adversarial payloads and code execution.- Sandbox execution for model training/inference (resource limits, seccomp, network egress control).- CI/CD security: sign builds, verify third-party dependencies, use reproducible builds.- Audit logging + tamper-evident logs (append-only, remote storage); retain per-tenant telemetry for forensics.- Incident response playbooks and red-team exercises focused on model attacks.- Customer-facing transparency: privacy/security settings, usage quotas, explainable security guarantees.Residual risk & metrics- Define acceptable epsilon for DP, maximum QPS per token, detection precision/recall targets for anomaly models.- Continuous monitoring: model extraction score, unusual confidence shifts, distribution drift, spike downloads.- Regular threat modeling updates as new attacks (e.g., advanced model inversion) emerge.SummaryCombine defensive-in-depth: mitigate query-based threats with rate-limits, watermarking and anomaly detection; mitigate data privacy with DP and output control; prevent poisoning with provenance, validation and sandboxing; protect storage and control plane with RBAC, KMS, and audit logging. Balance usability and security via configurable tenant policies and clear SLAs.
MediumTechnical
56 practiced
Design a model monitoring pipeline on a cloud platform to detect data drift and performance degradation for a binary classification model. Specify architecture, which statistical tests or metrics to compute (e.g., PSI, KL divergence, feature histograms), how to store reference vs current distributions, alerting rules, and an automated retraining trigger policy that minimizes false positives.
Sample Answer
Requirements & constraints:- Continuous monitoring for data drift and label/performance degradation on a binary classifier in cloud (e.g., GCP/AWS/Azure).- Low false positives, scalable, auditable, and supports automated retraining.High-level architecture:- Inference service → Event stream (Kafka / Pub/Sub) capturing features, prediction, metadata, timestamp → Metrics pipeline (Flink/Dataflow or Spark Streaming) → Feature store / object storage (BigQuery / S3 / Delta Lake) maintains reference and windowed current distributions → Monitoring jobs (batch or streaming) compute stats → Alerting & orchestration (CloudWatch/Stackdriver + PagerDuty + Slack) → Retrain pipeline (CI/CD) gated by policy → Model registry.What to compute:- Per-feature: PSI (Population Stability Index) over windows (7d, 30d); KL divergence for continuous features; categorical chi-square / JS divergence; feature histograms and summary stats (mean, std, percentiles).- Label/prediction: model accuracy, AUC-ROC, precision/recall, calibration (Brier score), confusion matrix over labeled windows; prediction distribution shifts.- Concept drift detectors: ADWIN or Page-Hinkley for streaming detection on loss or prediction probabilities.Storing reference vs current:- Reference snapshot stored immutably in model registry or feature store (parquet/Delta with version tag) including histograms and serialized summary metrics.- Current distributions computed over sliding windows and stored in a monitoring DB (time-partitioned BigQuery / ClickHouse) for auditing and trend analysis.Alerting rules (minimize false positives):- Tiered thresholds: warning (PSI > 0.1) and critical (PSI > 0.25); require sustained deviation over N consecutive windows (e.g., 2 daily windows or 1 weekly) before alerting.- Combine signals: trigger alert only when both data drift metric and at least one downstream performance metric degrade beyond thresholds (e.g., AUC drop > 2% absolute or precision drop > 5%).- Use rolling baselines and seasonality correction (compare to same weekday window) to avoid expected shifts.Automated retraining trigger policy:- Soft-trigger: when critical alert conditions met, start validation pipeline that: 1) Samples recent labeled data (minimum sample size with power calculation) and runs cross-validation and holdout evaluation. 2) Compares candidate model performance vs production using statistical tests (paired bootstrap for AUC, McNemar for classification errors) to ensure significance (p < 0.05) and practical uplift (e.g., AUC +0.01 or error rate -2%). 3) If candidate passes, push to staging rollout (canary 5-10% traffic) and monitor for 24-72 hours; promote to prod when metrics stable.- Hard constraints: do not auto-deploy without sufficient labeled data, statistical significance, data lineage checks, and human review for high-impact models.Additional best practices:- Explainability drift: monitor SHAP value summaries to detect feature importance shifts.- Logging & lineage: store raw inputs for repro and regulatory needs; tag events with model version.- Retraining cadence: schedule periodic full retrain (e.g., monthly) even if no drift alerts.- Governance: maintain playbooks, runbooks, and rollback procedures.
EasyTechnical
61 practiced
What is a feature store and what problems does it solve in production ML? Explain the difference between online and offline feature stores, consistent feature engineering between training and serving, serving semantics (freshness & latency), and integration points with data pipelines and model training.
Sample Answer
A feature store is a centralized system for creating, storing, and serving ML features consistently for both training and production. It solves key production problems: feature reuse, single source of truth, consistent transformations between training and serving, low-latency access for online inference, and governance/auditing of feature lineage.Key capabilities and why they matter:- Consistency: Same feature definitions used for training and serving prevents training/serving skew.- Discoverability & reuse: Cataloged features speed development and reduce duplication.- Freshness & latency controls: Guarantees required SLAs for real-time predictions.Online vs Offline:- Offline store: Batch-oriented dataset storage (e.g., Parquet/BigQuery). Used for model training, backfills, validation. High throughput, not latency-sensitive.- Online store: Low-latency key-value store (e.g., Redis, DynamoDB). Used by serving stack for real-time inference; optimized for sub-10ms reads.Consistent feature engineering:- Persist pre-computed features (feature values or transformation code) in the store.- Ensure identical transformation logic: run the same feature pipeline used to generate training labels when materializing online features; include versioning and deterministic joins on entity keys and timestamps to avoid label/feature leakage.Serving semantics (freshness & latency):- Freshness: how up-to-date features are (e.g., real-time, microbatch, daily). Define SLAs per use case (fraud: seconds; recommendation: minutes/hours).- Latency: read latency from online store; design trade-offs between compute-on-read vs materialize-on-write. Materialize-on-write gives low read latency and predictable freshness; compute-on-read reduces storage but increases inference latency.Integration points:- Batch ETL: feature computation jobs write to offline store and materialize to online store.- Stream processors: update online features in real time.- Training pipelines: read features from offline store (or materialized snapshots) to build datasets with the same keys/timestamps used in serving.- Serving stack: model queries online store via entity key to fetch features.Example: A user lifetime-value feature is computed daily in batch for training, materialized to the online store after aggregation, and updated in real time for recent purchases via a Kafka stream to meet fraud detection freshness requirements. Versioning and lineage ensure retraining uses the exact feature definitions used in production.
MediumTechnical
57 practiced
Compare canary and blue-green deployment strategies for ML model serving. For each approach describe the deployment steps, how to route traffic and validate model quality during rollout, rollback triggers, monitoring metrics to evaluate, and trade-offs when the model is stateful (session-based) vs stateless.
Sample Answer
Start with a short definition then compare by categories.Definitions:- Canary: Gradually shift a small percentage of production traffic to the new model version while the rest stays on the current one.- Blue-green: Stand up a full new environment (green) alongside the current (blue) and switch all traffic to green in one step after validation.Deployment steps:- Canary: 1. Deploy new model to a small subset of serving instances. 2. Configure progressive traffic-splitting (e.g., 1% → 5% → 25% → 100%) with a service mesh or load balancer. 3. Run validation checks and increase traffic if healthy.- Blue-green: 1. Deploy full green environment (identical infra/config) with new model. 2. Run smoke tests / shadow traffic validation. 3. Cutover all traffic to green (DNS or load balancer swap) when ready.Routing & validation during rollout:- Canary: Use weighted routing; run A/B metrics comparing requests served by canary vs baseline. Use shadowing for deterministic validation (send same input to both but return baseline response to user). Validate latency, accuracy, calibration, business KPIs, payload compatibility.- Blue-green: Use shadow traffic and pre-cutover test suites. Perform end-to-end checks, canary-like experiments inside green before full cutover.Rollback triggers:- Significant degradation in core metrics (accuracy/precision/recall beyond SLA), increased error rate, latency SLO breaches, data schema mismatches, or data drift detected by population statistics. For canary you can instantly reduce weight to 0; for blue-green you switch back to blue (fast if infra kept warm).Monitoring metrics to evaluate:- Model correctness: accuracy, AUC, confusion matrix segments, business KPIs (revenue, conversion).- Reliability: 4xx/5xx error rates, inference failures.- Performance: P95/P99 latency, throughput, CPU/GPU utilization, memory.- Data/Model health: input feature distribution (drift), prediction distribution, confidence calibration, slice metrics (per-user, geography).- For online learning: model weight changes, training vs serving drift.Stateful (session-based) vs stateless trade-offs:- Stateless: Easier—requests can be routed arbitrarily; canary traffic splitting and load balancing are straightforward; rollbacks simple.- Stateful/session-based: Harder—session stickiness required. Canary: must route a user’s session consistently to same version (session affinity or sticky cookies); otherwise inconsistent behavior and poor UX. Blue-green: safer when sessions are long—perform cutover at session boundaries or drain sessions from blue before switching. Stateful models also require state migration/compatibility (session data formats) and careful coordination to avoid split-brain or data loss. Monitoring must include session continuity metrics and end-to-end user flow correctness.Trade-offs summary:- Canary: lower blast radius, continuous validation, more granular rollback, but needs robust traffic-splitting and careful sampling to ensure statistical power.- Blue-green: simple atomic switch, simpler validation in an isolated environment, but requires double capacity and a well-tested cutover/rollback plan; larger blast radius on mistakes.Best practice: combine approaches — run a canary inside a green environment (deploy green, route a small percentage, validate, then cutover) and ensure automated alarms + automated rollback procedures keyed to business and engineering metrics.
EasyTechnical
52 practiced
Compare the core capabilities and typical use-cases of the three major managed cloud ML platforms: Amazon SageMaker, Google Vertex AI, and Microsoft Azure Machine Learning. For each platform describe: 1) managed training and hyperparameter tuning features, 2) inference serving options (serverless, hosted endpoints, batch), 3) model registry and pipeline offerings, and 4) key limitations where you might prefer a self-hosted solution (portability, custom networking, very custom GPUs, compliance).
Sample Answer
High-level summary: SageMaker, Vertex AI, and Azure ML all provide end-to-end managed ML but differ in UX, integrations, and edge cases. Below I compare them by the four requested areas for an AI Engineer.Amazon SageMaker1) Managed training & HPO: Fully managed training jobs, built-in algorithms, managed distributed training (MPI, Horovod, DeepSpeed). Hyperparameter tuning via Bayesian, random, and early stopping (Automatic Model Tuning).2) Inference serving: Supports serverless inference for small models, multi-model endpoints, provisioned/auto-scaling hosted endpoints, and batch transform for large offline workloads.3) Model registry & pipelines: SageMaker Model Registry plus SageMaker Pipelines (CI/CD for ML) with step functions-like orchestration and integration to CodePipeline/CI.4) Limitations: Less portable (deep AWS integrations), custom networking/GPU setups can be constrained; for nonstandard hardware, strict compliance or vendor-neutral portability you may self-host.Google Vertex AI1) Managed training & HPO: Managed training with custom containers, built-in AutoML and Vizier-based HPO (Bayesian optimization, early stopping); good TPU support for training.2) Inference serving: Endpoint service for online prediction (autoscaling), serverless inference for supported model types, and batch prediction for large jobs.3) Model registry & pipelines: Vertex Model Registry and Vertex Pipelines (KFP-based) with strong MLOps integration to BigQuery and Dataflow.4) Limitations: Best-in-class if you use GCP stack; portability concerns with TPUs, custom networking constraints, and specialized compliance needs may push to self-host.Microsoft Azure Machine Learning1) Managed training & HPO: Managed compute clusters, built-in estimators and distributed training; HyperDrive for HPO (random, grid, Bayesian).2) Inference serving: Real-time endpoints (AKS-backed), managed online endpoints (serverless preview/history), and batch inference via Pipeline or Batch Endpoints.3) Model registry & pipelines: Robust Model Registry and ML Pipelines (SDK/CLI) with GitOps and Azure DevOps integrations.4) Limitations: Strong Azure tie-ins; custom GPUs, strict on-prem network/control/compliance, or desire for cloud-agnostic deployment can favor self-hosting.When to self-host (common across all): need full portability (Kubernetes + KServe), specialized/custom hardware or drivers, strict VPC/isolation or bespoke networking, or regulatory/compliance mandates requiring on-prem control. Trade-offs: managed services speed up iteration and ops, self-hosting increases control at the cost of engineering overhead.
Unlock Full Question Bank
Get access to hundreds of Cloud Machine Learning Platforms and Infrastructure interview questions and detailed answers.
