Cloud Machine Learning Platforms and Infrastructure Questions
Knowledge of cloud hosted machine learning and artificial intelligence platforms and the supporting infrastructure used to develop, train, deploy, and operate models at scale. Candidates should be familiar with major managed offerings such as Amazon SageMaker, Google Cloud artificial intelligence platform, and Microsoft Azure Machine Learning and understand capabilities including pretrained models, managed training jobs, managed inference endpoints, model registries, and managed pipelines. Key areas include differences between cloud and local training, distributed and hardware accelerated training options, cost trade offs including spot and preemptible instances, serving patterns such as serverless inference, hosted endpoints and batch processing, autoscaling strategies for inference, model versioning and rollout strategies including canary and blue green deployments, integration with data storage, feature stores and data pipelines, and model monitoring, logging and drift detection. Candidates should also be able to explain when to use managed services versus self hosted or on premises solutions, discussing trade offs around productivity, operational overhead, control and customization, vendor lock in, security, data residency and compliance, as well as operational practices such as continuous integration and deployment for models, testing and validation in production, observability and cost optimization.
MediumTechnical
46 practiced
Case study: Design an end-to-end managed pipeline using Vertex AI Pipelines or SageMaker Pipelines that automates data validation, training, model evaluation, human approval gating, deployment to staging, and promotion to production. List components, triggers, artifact stores, and failure handling strategies.
Sample Answer
Requirements:- Automated end-to-end CI/CD for ML: data validation, training, eval, human-approval gate, deploy to staging, smoke tests, promote to production.- Auditability, reproducibility, artifact storage, rollback, alerts.High-level pipeline (applies to Vertex AI Pipelines or SageMaker Pipelines):1. Ingest & Validation - Component: Data ingestion step (Cloud Storage / S3 source) → DataValidation (TensorFlow Data Validation or Great Expectations) - Output artifacts: validated dataset, validation report (schema, metrics)2. Feature engineering (optional) - Component: Transform (Beam/Batch or Script) - Artifacts: featureset, feature metadata3. Training - Component: Trainer (containerized training job on Vertex Training / SageMaker Training) - Artifacts: model checkpoint, model artifact, training metrics4. Evaluation - Component: Evaluator (test set inference, bias/fairness checks, metrics) - Artifacts: evaluation report, model card5. Human Approval Gate - Component: ManualApproval or human-in-the-loop step (Vertex: Pipeline UI approval; SageMaker: Model Registry approval) - Trigger: only if evaluation meets automated thresholds; else auto-reject6. Deploy to Staging - Component: Deploy step to staging endpoint (Vertex Endpoint / SageMaker Endpoint or serverless) - Post-deploy: Run smoke tests and canary evaluations7. Automated Promotion - Component: If staging tests pass and SLA met, promote via Model Registry version promotion or update prod endpoint (gradual rollout)8. Production Monitoring & Rollback - Component: Continuous monitoring (prometheus/logging, Vertex Model Monitoring or SageMaker Model Monitor), automated rollback on drift/latency/regression.Triggers & Orchestration:- Source trigger: New data arrival in storage or scheduled cron (Cloud Scheduler / EventBridge)- Pipeline orchestration: Vertex Pipelines (Kubeflow pipelines) or SageMaker Pipelines- Model Registry triggers promotion flows; CI hooks for code changes.Artifact Stores:- Model artifacts & checkpoints: Cloud Storage / S3 and Model Registry (Vertex Model Registry or SageMaker Model Registry)- Metadata & lineage: Vertex Metadata / SageMaker Experiments & Lineage- Validation/eval reports: BigQuery / Athena or object store + metadata entries- Container images: Artifact Registry / ECRFailure handling strategies:- Step-level retries with exponential backoff, circuit breakers for training infra limits- Validation fail: stop pipeline, notify data engineers, create incident with failing dataset snapshot- Training fail: capture logs, auto-collect diagnostics, retry with different instance type or fail-fast to save cost- Post-deploy anomalies: auto-roll back to previous registered model version; route small % traffic (canary) before full cutover- Alerts & SLAs: Integrate with PagerDuty, Slack, and create runbooks; automated ticket creation- Audit & reproducibility: store input data hash, code commit, container image digest, hyperparameters in metadata for each run.Why this works:- Clear separation of responsibilities, auditable artifacts, human-in-the-loop for risk control, robust monitoring and rollback to minimize production impact while enabling automated promotion when safe.
HardSystem Design
53 practiced
Design an inference architecture for an ensemble of models that must serve 5k QPS with p99 latency under 200ms while minimizing cost. Discuss request routing, selective ensemble evaluation, caching, batching, hardware selection, and graceful degradation strategies during high load.
Sample Answer
Requirements & constraints:- 5k QPS sustained, p99 < 200ms, cost-sensitive, ensemble of heterogeneous models (some heavy NN, some lightweight).High-level architecture:- API Gateway → Router/Orchestrator → Preprocessor → Ensemble Manager → Model Workers (grouped by type) → Aggregator → Cache → Postprocessor → Client.- Use autoscaling groups per model type behind gRPC endpoints; orchestrator coordinates per-request plan.Request routing & selective evaluation:- Use a fast gating model (cheap classifier or feature-based rules) to decide which ensemble members are likely to change the final decision. Example: linear model or small transformer producing confidence/utility estimate. Only invoke expensive models when gate indicates improvement over threshold.- Maintain model importance weights; run lightweight models always, heavy models on-demand.- Implement per-user/request routing: if recent history indicates stable prediction, skip full ensemble.Caching:- Two-layer cache: L1 in-process LRU for hot items, L2 distributed cache (Redis) for cross-instance hits.- Cache key includes input fingerprint + model versions. Serve from cache for identical inputs to eliminate compute and meet latency.Batching:- Use micro-batching (e.g., 8–32) inside model workers for GPU efficiency, but schedule batch windows short (<10ms) to respect latency SLO. Use dynamic batching libraries (Triton, TorchServe) with latency-aware batch sizing.- For CPU models, prefer request-level processing or small batches.Hardware selection:- GPU instances (A10/A100 or cloud inference GPUs) for heavy neural models with batching; CPU (c6i/c7a) for lightweight models and gating. Consider GPU-sharing via multi-tenant Triton or NVIDIA MIG for cost efficiency.- Use mixed precision (FP16/INT8) and model quantization/pruning to reduce inference time/cost.Graceful degradation:- Priority queue with SLA-aware admission control. Under overload, drop or defer low-priority requests, skip non-critical ensemble members (fall back to base model), or return cached/approximate result.- Circuit breakers per model to prevent cascading overload; autoscale faster for gating and aggregator tiers.- Provide progressive responses: immediate base-model result, then patch/update when heavier models finish (if clients support async).Monitoring & ops:- End-to-end tracing, p99 latency per component, cache hit rates, gate false negatives. Continuous A/B testing to validate selective evaluation quality. Versioning and canary deploys to ensure stability.Example numbers:- Gate reduces expensive model calls by 80% → heavy GPU budget drops to ~1k QPS; with batching size 16, effective GPU throughput = 16k inferences/s per GPU—one or two GPUs per heavy model cluster suffice, lowering cost while keeping p99 headroom.
HardTechnical
44 practiced
Problem: You must run hyperparameter tuning across multiple instance types and regions to minimize monetary cost while achieving a target accuracy. Propose an orchestration strategy and algorithmic approach that prunes poor configurations early, schedules promising runs on faster/expensive resources, and accounts for spot availability.
Sample Answer
Requirements & constraints:- Target accuracy threshold A within minimal monetary budget across multiple instance types (vCPU/GPU/FP32/TF32), regions, and spot availability; time-to-result secondary.- Must support early-pruning, preemption-resilience, and dynamic scheduling.High-level orchestration:1. Controller service (Kubernetes/gRPC) with a Cost-Aware Scheduler, Trial Manager, Global State DB (Redis/Postgres), and Spot Monitor per region.2. Worker pool: heterogeneous node pools (on-demand + spot) per region; each worker runs containerized trials with checkpointing to shared object store (S3/GCS).Algorithmic approach:- Use multi-fidelity Bayesian Optimization + asynchronous successive halving (BO+ASHA): - BO proposes hyperparameter configs; each config is evaluated at low fidelity (short epochs / subset data) to get a cheap performance estimate. - ASHA/Hyperband schedules promotion: keep top fraction based on intermediate metrics. - BO’s surrogate model is trained on multi-fidelity observations (Fidel-aware GP or TPE with fidelity feature) to predict full-run accuracy and cost.- Cost-aware acquisition: modify acquisition function to maximize Expected Improvement per Dollar (EI / expected_cost) to favor cheap promising configs.- Instance mapping policy: predicted promising trials (high EI-per-cost) are queued for faster instances (GPUs, faster regions) using a priority queue; low-priority or exploratory trials run on cheaper/spot pools.- Spot handling: use short checkpoint intervals + preemption hooks; Spot Monitor tracks spot interruption frequencies and adjusts expected_cost in acquisition and instance selection. If spot risk high, promote mission-critical promotions to on-demand.- Autoscaling: scale node pools by queued expected-value demand; ensure capacity buffer for promotions.Data flow & decision loop:- BO proposes → Trial Manager launches low-fidelity trial → intermediate metrics & cost logged → Controller updates BO and ASHA → decisions to promote/terminate → scheduler assigns instance type/region considering spot risk and budget.Practical considerations & trade-offs:- Checkpoint frequency vs overhead: choose cadence that bounds rework cost under typical spot lifetimes.- Cold-start BO: use low-fidelity Latin Hypercube samples; warm-start with prior experiments.- Monitoring: track performance per region/instance for continual calibration of expected_cost and spot reliability.- Failure modes: if regions lose capacity, scheduler backfills with cheaper instances or delays promotions.Why this works:- Multi-fidelity + ASHA prunes poor configs early, saving money.- BO + cost-aware acquisition steers exploration toward best accuracy-per-dollar.- Priority-based scheduling maps resources efficiently while spot-aware policies exploit cheap capacity without risking critical runs.
MediumTechnical
49 practiced
Explain differences between A/B testing, canary deployments, and blue-green deployments for model rollouts on cloud ML platforms. For each approach explain traffic splitting mechanics, risk profile, rollback complexity, and ideal scenarios with examples.
Sample Answer
A/B testing, canary, and blue‑green are distinct strategies for rolling out models; choose by your goal (experimentation vs safe release) and risk tolerance.A/B testing- Purpose: Controlled experiment to compare model variants by business/ML metrics (CTR, precision).- Traffic splitting: Randomized user-level or session-level assignment (often 50/50 or weighted), persists to avoid cross-contamination.- Risk profile: Medium — acceptable because exposure is limited and metric-driven, but can harm UX for variant users.- Rollback complexity: Low — stop sending traffic to losing variant; must ensure logging and attribution preserve experiment integrity.- Ideal scenario: Evaluating two ranking models’ CTR over two weeks across cohorts.Canary deployment- Purpose: Gradual production rollout to detect regressions (latency, correctness) on a small slice of real traffic.- Traffic splitting: Incremental percentage ramp (e.g., 1% → 5% → 25% → 100%) either by request hashing or traffic routers.- Risk profile: Low-to-medium — limits blast radius; catches production-only bugs early.- Rollback complexity: Low — revert routing to previous model or set percentage back to 0; must handle state/migrations carefully.- Ideal scenario: Deploying new feature extractor that might increase latency—monitor infra/metrics during ramp.Blue‑Green deployment- Purpose: Replace entire production environment atomically to ensure zero-downtime switch and easy rollback.- Traffic splitting: Binary—all traffic routed to blue (current) or green (new); switch via load balancer or DNS.- Risk profile: Low for sudden catastrophic failures (easy instant rollback) but higher if new model requires long-running migrations or gradual validation.- Rollback complexity: Very low for stateless models—flip back to old environment; higher if you changed schemas or downstream expectations.- Ideal scenario: Major model version upgrade with compatibility tested in a staging green environment; quick rollback if errors.Trade-offs summary:- Use A/B for hypothesis testing and metric-driven model selection.- Use canary for cautious, incremental validation under real load.- Use blue‑green for atomic cutovers and quick disaster recovery when environments are interchangeable.Operational notes: ensure consistent routing keys for user stickiness, robust monitoring (latency, error rates, business metrics), automated health checks and feature-flagging to simplify rollbacks.
MediumTechnical
48 practiced
What runtime metrics, logs, and traces should you collect for a production ML endpoint to ensure effective observability? Cover infrastructure metrics, request and latency metrics, model-level metrics (accuracy, confidence), input and feature distributions, sampling strategies for request logs, and alerting design.
Sample Answer
High-level approach: collect metrics, logs, and traces across three layers—infrastructure, request/serving, and model—and design sampling and alerting so signals are actionable without overwhelming storage or operators.Infrastructure metrics- Host/container: CPU, memory, disk I/O, disk pressure, GPU utilization, GPU memory, temperature- Network: bandwidth, packet error rates, connection counts, load balancer health- Orchestration: pod/VM restarts, OOM kills, node drain events, autoscaler decisions, scheduling latencies- Storage/DB: latency and error rates for feature store, model registry, and artifact storesRequest and latency metrics- Throughput: requests/sec, concurrent requests, success/failure rates (4xx/5xx)- Latency buckets: p50/p90/p95/p99, end-to-end vs. model inference vs. pre/post-processing- Queue/backpressure metrics: request queue lengths, backpressure/drop counts- Cold-start counts, circuit-breaker/trip countsModel-level metrics- Performance: accuracy, precision, recall, F1, AUC where applicable (computed on labeled feedback)- Confidence & calibration: mean predicted probability, calibration error (ECE), confidence histograms- Business metrics: conversion, revenue lift, false positive cost, fairness metrics (bias by subgroup)- Drift signals: population drift, concept drift, label-delay-aware metricsInput and feature distributions- Per-feature summaries: min/max/mean/std, percentiles, missing rates, cardinality- Histograms and quantiles for continuous features; top-k frequency and new-category counts for categorical- Feature correlations and covariance changes- Population stability metrics: PSI (population stability index), KL divergence, rolling-window comparisonsTracing and logs- Distributed tracing: request id, trace id, spans for gateway → preprocess → inference → postprocess → downstream calls, with timing and error tags- Structured request logs: timestamp, request id, model version, latencies, status, anonymized or hashed key fields- Error logs with stack traces, input sample hashes, model versionSampling strategies for request logs and payloads- Rate-based: fixed % (e.g., 1%) for general sampling- Stratified: oversample errors, high-latency requests, low-confidence predictions, or specific user segments- Anomaly-triggered: always capture when feature drift, model confidence below threshold, or new categories seen- Store hashes/IDs instead of raw PII; keep small payload snapshots for debugging with privacy controls and TTLAlerting design- Define SLIs/SLOs: request success rate, p95 latency, model quality SLI (e.g., weekly accuracy), feature drift SLI- Alert tiers: urgent (p95 latency spike, error rate > threshold), high (model quality drop > delta), info (slow drift trend)- Multi-signal alerts: require both model quality drop and feature distribution change to reduce noise- Include runbooks: playbook steps, rollback model version, recreate environment, sample queries to reproduce- Safety nets: automated rollback for catastrophic failures, paged alerts for on-call, dashboards for situational awarenessPractical notes- Tag metrics by model version, deployment, region, and traffic partition (A/B)- Retention: raw request traces short-term; aggregated metrics and drift indexes long-term- Privacy/compliance: mask or avoid logging raw PII, follow retention and access controlsThis observability stack enables fast detection (latency/errors), impact assessment (model quality/business metrics), root-cause (traces + sampled payloads), and controlled response (alerts + runbooks).
Unlock Full Question Bank
Get access to hundreds of Cloud Machine Learning Platforms and Infrastructure interview questions and detailed answers.
