This topic covers the end to end practice of clarifying ambiguous problem statements, eliciting and defining functional and non functional requirements, and scoping solutions before design and implementation. Candidates should demonstrate the ability to identify target users and user journeys, conduct stakeholder interviews, ask targeted and probing clarifying questions, surface hidden assumptions and root causes, and convert vague business language into measurable technical and business requirements. They should capture acceptance criteria and success metrics, define key performance indicators, and translate requirements into testable statements and test strategies that map unit, integration, and system tests to requirement risk and priority. The topic includes assessing technical constraints and operational context such as expected scale, throughput and latency requirements, data volume and read write ratios, consistency expectations, real time versus batch processing trade offs, geographic distribution, uptime and availability expectations, security and compliance obligations, and existing system state or migration considerations. It also requires evaluation of non technical constraints including timelines, team capacity, budget, regulatory and operational concerns, and stakeholder priorities. Candidates are expected to synthesize inputs into clear artifacts such as product requirement documents, user stories, prioritized backlogs, acceptance criteria, and concise requirement checklists to guide architecture, estimation, and implementation. Emphasis is placed on scoping and prioritization techniques, distinguishing must have from nice to have features, conducting trade off analysis, proposing incremental or phased approaches, identifying risks and mitigations, and aligning cross functional teams on scope and success measures. Expectations vary by seniority: entry level candidates should reliably ask core clarifying questions and avoid solving the wrong problem, while senior and staff candidates should rapidly prioritize requirements, anticipate critical non functional needs, align solutions to business impact, and communicate trade offs and timelines to stakeholders.
MediumTechnical
51 practiced
Write four sample user stories with acceptance criteria (INVEST-compliant) for a microservice that serves ML predictions and logs features and predictions for monitoring. Each story should be small, testable, and include the data or telemetry required to validate acceptance.
Sample Answer
1) Story: As an ML Engineer, I want the prediction microservice to return model predictions via a REST endpoint so clients can get real-time inferences.Acceptance Criteria:- POST /predict accepts JSON payload with feature vector and request_id and returns 200 with {prediction, confidence, model_version, request_id}.- Response latency < 200ms for 95th percentile under test load (100 RPS).- Errors return standard 4xx/5xx with request_id echoed.Telemetry to validate:- request_id, timestamp, latency_ms, status_code, model_version emitted to monitoring.2) Story: As an ML Engineer, I want the service to log input features and prediction outputs to an observability sink so we can monitor data drift and model performance.Acceptance Criteria:- For every successful prediction, a structured log/event is emitted to Kafka (or S3) containing {request_id, model_version, features_hash, features:nullable, prediction, confidence, timestamp}.- Logs arrive within 30s of prediction; end-to-end delivery success metric >= 99.5% in test batch of 10k requests.Telemetry to validate:- counts: emitted_events, delivery_failures, avg_delivery_latency; sample payloads stored for audits.3) Story: As an ML Engineer, I want the service to emit per-minute aggregation metrics so we can monitor model health and detect performance regressions.Acceptance Criteria:- Service exposes Prometheus metrics: predictions_total, predictions_by_label, avg_confidence, pctile_latency_95, model_version as label.- Metrics update within 60s and reflect load from a synthetic test generating 1k predictions/min (predictions_total increases accordingly).Telemetry to validate:- Prometheus scrape shows metrics with correct labels and expected deltas after synthetic load.4) Story: As an ML Engineer, I want automatic alerting when key monitoring signals cross thresholds so incidents are detected proactively.Acceptance Criteria:- Alert rules created: (a) error_rate > 1% over 5m, (b) avg_confidence drop > 15% vs. baseline for 10m, (c) data_delivery_failures > 0.5% over 10m.- Alert triggers send to Slack and PagerDuty with request samples (up to 5) and recent metric snapshots.Telemetry to validate:- Simulated scenario that increases error_rate to 2% causes alert within 6 minutes; alert payload includes metric snapshots and sample request_ids.
EasyTechnical
56 practiced
As an entry-level MLE you are given: 'Use historical user events to predict churn.' List eight clarifying data questions you would ask to avoid solving the wrong problem. Cover: precise churn definition, labeling window, lookback periods, feature freshness, sampling biases, missing data handling, PII constraints, and validation split strategy.
Sample Answer
1) How do you define churn for this product (e.g., N days of inactivity, subscription cancellation, or low engagement)? — Definition determines the label and business action.2) What labeling window should we use (e.g., predict churn within 30, 60, or 90 days)? — Affects label balance and model horizon.3) What lookback period of historical events is available/useful (e.g., 7, 30, 180 days)? — Controls feature generation and signal strength.4) How fresh must features be at prediction time (real-time, hourly, daily batch)? — Impacts feature engineering and serving design.5) Are there known sampling biases (e.g., new-user oversampling, regional coverage gaps) or population shifts? — Guides sampling strategy and fairness checks.6) How should we handle missing or sparse event streams (imputation, special tokens, drop users)? — Defines preprocessing and model robustness.7) What PII constraints or anonymization requirements exist (hashed IDs, linkage limits, retention rules)? — Ensures privacy-compliant feature use and joins.8) How should we split data for validation (time-based holdout, rolling windows, or user-level stratified)? — Prevents leakage and simulates production performance.For each answer, ask for examples, sample queries, and expected business KPIs to align modeling choices with product goals.
HardSystem Design
52 practiced
Design a low-latency, multi-region model serving architecture to support 10,000 QPS with a 10ms median latency SLO and 99.99% availability. Address model replication, feature consistency, model update propagation (rolling/canary), cross-region failover, storage choices, and explain how scoping decisions (e.g., per-region models vs single global model) impact cost and complexity.
Sample Answer
Requirements:- Functional: 10k QPS globally, 10ms median inference SLO, 99.99% availability.- Constraints: multi-region (geo-proximity), strong feature consistency for inference inputs, safe model rollout.High-level architecture:- Global DNS + Geo-routing -> Regional Edge LB (CDN / Regional ALB) -> Regional API Gateway -> Per-region Model Serving Cluster (K8s/EC2 with GPU/CPU pods) -> Feature Store + Cache -> Central Control Plane (model registry, rollout manager, metrics).Key components & choices:1. Model replication- Maintain one active model per region (replicated binaries/artifacts from central registry to minimize cross-region RTT).- Use immutable model artifacts (S3/GCS + signed manifests). Pull by regional cluster on deployment.- Keep model weights local to serving nodes (warm containers) to meet 10ms median.2. Feature consistency- For low-latency, perform feature lookup locally: use regional Feature Store (Redis/Memcached for hot features) populated via async CDC from primary datastore.- For features requiring global consistency (user profile updates), use last-write-wins with per-request version/timestamp and fall back to strongly-consistent read only when necessary (rare), accepting slightly higher tail latency.3. Model update propagation (rolling/canary)- Central control plane triggers staged rollout: - Canary: deploy to small % of nodes in one region; run shadow traffic and A/B eval. - Metrics-driven promotion: automated MLOps pipeline checks latency, accuracy, resource usage, and rollback on anomaly. - Rolling updates across pods then regions to prevent simultaneous multi-region risk.- Use traffic-splitting at the API gateway for canary.4. Cross-region failover- Active-active regions with geo-routing; failover via health checks + DNS TTLs (short TTL ~30s) and client-side retry with region fallback.- Replicate state (models, feature snapshots) asynchronously; ensure idempotent requests and sticky session avoidance.5. Storage choices- Model artifacts: object store (S3/GCS) + CDN for quicker pulls.- Feature store: regional Redis clusters for hot reads, regional OLAP for cold features (Bigtable/Spanner/Cassandra) with CDC pipeline.- Metadata: central SQL/NoSQL (highly available) for registry and rollout state.Scoping trade-offs (per-region vs global model)- Per-region models (region-specific tuning): lower latency, better locality, compliance benefits; higher cost (multiple training/validation runs, storage, ops complexity) and model divergence risk.- Single global model: simpler ops, consistent behavior, lower training cost, but may increase latency if inference routed cross-region or require larger model to cover all distributions.Recommendation: hybrid — single global model replicated per-region for serving; optionally region-adapted fine-tuning overlays (small adapter layers) to balance cost and accuracy.Scaling & reliability:- Provision capacity with headroom (10k QPS -> per-region partitioning, autoscale pods using HPA/metrics), use request batching where possible, GPU pooling for heavy models.- SLOs achieved by local replicas, warm pools, aggressive caching, and circuit-breakers for downstream feature stores.Metrics & monitoring:- Track per-region p50/p99 latency, error rates, feature freshness, model drift, rollout KPIs. Automated rollback when safety thresholds breached.This design prioritizes low-latency and availability by keeping serving and hot features regional, while centralizing control for safe, observable rollouts and cost-effective training.
EasyTechnical
66 practiced
List five common hidden assumptions stakeholders make when they request 'real-time scoring' from a model. For each assumption explain its possible impact on scope, cost, or design—for example, differing interpretations of 'real-time' latency, data availability at request time, or expectations about stateful features.
Sample Answer
1) "Real-time" means sub-100ms latency.Impact: If stakeholders expect very low latency but data pipelines, model complexity, or network hops can't meet it, scope expands to include model simplification, optimized serving (e.g., C++ gRPC, model quantization), edge deployment, or caching — all increasing cost and engineering effort.2) All input features are available at request time.Impact: If some features are only available in batch, you must redesign inputs (use imputation, lookup services, or change feature set). That can reduce model accuracy or require additional engineering for synchronous feature stores, raising cost and timeline.3) The model can be stateful (keeps session/user history locally).Impact: Stateful services need sticky sessions, storage, and consistency guarantees. This increases infrastructure complexity, scaling difficulty, and cost vs. stateless inference where history is passed as input.4) Real-time scoring implies low compute cost per request.Impact: Complex ensembles or large transformers violate this; supporting them requires autoscaling, GPU-backed inference, or batching trade-offs. Budget and SLOs must be revisited; sometimes approximate/ distilled models are needed.5) Real-time scoring results are final and deterministic.Impact: Stakeholders may expect single-shot decisions. In reality you may need confidence scores, fallbacks, human review, or asynchronous re-scoring. Designing for retries, audits, and monitoring affects scope, latency guarantees, and compliance cost.For each assumption, clarify SLA (latency), data availability, state requirements, cost limits, and error-handling up front to avoid scope creep.
MediumTechnical
67 practiced
You must estimate cost and time to build a new ML feature. What inputs do you request from PM and engineering, how would you produce a best-case/worst-case estimate, and how do you explicitly account for technical debt, unknowns, and discovery work in your estimate?
Sample Answer
Inputs I request- From PM: clear success criteria (metrics, latency/throughput SLAs), target launch date, user segments, acceptance tests, business priority, rollout plan (canary vs full).- From Engineering: current infra, data availability & pipelines, existing feature flags, deployment requirements, monitoring/observability, team capacity and skill sets.- From Data Science/Analytics: sample data size/quality, label availability, feature computations, baseline models and performance.Producing best-/worst-case estimates1. Break work into phases: Discovery (2–4w), Data prep (1–6w), Modeling/prototyping (1–8w), Productionization (2–6w), Testing & rollout (1–4w).2. For each phase estimate optimistic, likely, pessimistic durations based on past projects or expert judgment. Example: Data prep: best 1w, expected 3w, worst 6w.3. Sum per-scenario to give best-case/likely/worst-case timelines and costs (use team hourly rates + infra/cloud estimates).4. Present range and probability (e.g., P50/P90).Accounting for technical debt, unknowns, discovery- Explicitly include a Discovery/Spike task (timeboxed 1–2 sprints) to reduce unknowns and produce refined estimate.- Add contingency buffer: +20% for low-risk, +50% for high-risk features; or use P90 for budgeting.- Create a risk register mapping unknowns to mitigation actions and cost of failure (e.g., poor labels → add labeling budget).- Separate tech-debt items as explicit tasks with estimates (refactor data pipelines, add feature stores) so they’re visible and not hidden in buffer.- Recommend milestone-based gating: reassess estimate after discovery with go/no-go decision.Example summary (high level):- Best-case: 6 weeks, $40k (small data cleanup, reuse infra)- Likely (P50): 12 weeks, $90k (standard data work + prod adjustments)- Worst-case (P90): 24 weeks, $180k (extensive labeling, infra changes, refactor)This gives stakeholders clear ranges, visible assumptions, and actions to reduce uncertainty.
Unlock Full Question Bank
Get access to hundreds of Requirements Elicitation and Scoping interview questions and detailed answers.
