Approaches system design from a program and delivery perspective. Candidates should explain how they clarify requirements and constraints up front, decompose complex systems into deliverable components and milestones, and plan schedules that account for technical complexity and dependencies. Describe how to involve and align engineering teams on architecture decisions, translate technical trade offs for stakeholders, identify and mitigate risks, set acceptance criteria, and plan for capacity, testing, deployment, and operational readiness. Include how program planning accounts for cross team coordination, technical debt, release coordination, and measurement of success.
MediumTechnical
43 practiced
Given conflicting requirements such as higher accuracy, lower latency, and reduced cost, describe a principled prioritization approach you would use at program level to decide which model features or optimizations to implement. Include stakeholder mapping, a scoring rubric or framework, and governance for revisiting decisions.
Sample Answer
Situation: On a program balancing higher accuracy, lower latency, and reduced cost, I apply a principled, repeatable prioritization process so engineering work aligns with business value and constraints.Approach (overview):1. Clarify objectives & constraints — translate high-level goals into measurable KPIs (e.g., +2% precision, <50ms P95 latency, <$0.10 infer cost per 1k requests), regulatory limits, and launch timelines.2. Stakeholder mapping — list stakeholders, their goals and constraints: - Product (conversion / UX) - ML research (accuracy / model innovation) - SRE/Infra (latency, reliability, cost) - Finance (budget) - Legal/Compliance (fairness, explainability) - Customers/ops (SLA expectations)3. Scoring rubric / decision matrix — define dimensions, weights (example): - Business impact (revenue/engagement): 35% - User experience (latency/throughput): 25% - Cost delta (infra + ops): 15% - Technical risk & effort: 15% - Compliance & fairness: 10% Score each candidate feature/optimization (0–5) per dimension, compute weighted score, and rank.Example: Quantization gives +0.5% accuracy loss (score 2 on business impact), big latency cut (score 5), cost reduction large (score 5), low risk (score 4) → high weighted score => prioritize.Governance & revisiting decisions:- Decision record: store rationale, assumptions, experiments, and expected KPIs in a lightweight PRD.- Experiment-first policy: require A/B or canary evaluation with telemetry (accuracy slices, latency P50/P95/P99, cost) before full rollout.- Review cadence: quarterly re-evaluation and after major infra or data changes; ad-hoc when metrics deviate.- Rollback & mitigation: clear rollback criteria, automated alerts, and runbooks.- Continuous learning: capture outcomes, update weights/criteria based on retrospective (if a decision under-performed, adjust costs/weights accordingly).This framework makes trade-offs explicit, aligns engineering effort to measurable value, and creates governance so decisions evolve as data and business priorities change.
HardSystem Design
46 practiced
Design an operational readiness plan and runbook that covers safe model rollback, data rollback strategies (for example: feature-flagging, compensating transactions), and chaos testing for ML services at scale. Include owner roles, pre-release drills, rollback decision criteria, and KPIs to validate readiness before a major release.
Sample Answer
Requirements:- Safe, auditable model and data rollback at scale with minimal customer impact- Automated and manual rollback paths, feature-flag and traffic-split support- Compensating transactions for downstream stateful systems- Chaos testing to validate rollback and recovery under failure modes- Clear owners, drills, decision criteria, and KPIsHigh-level design:- CI/CD + Model Registry + Serving Cluster + Feature Flag Service + Orchestration + Observability + Incident Platform- Model artifacts and schema versions stored in registry with signed metadata and canary configs- Traffic routing via feature flags and load-balancer weights (Envoy/Service Mesh)- Data-change journaling and idempotent compensating transaction jobs- Chaos tooling (chaos mesh/Gremlin) integrated with staging and canary environmentsCore components & responsibilities:- ML Engineer (model packaging, canary config, rollback playbook)- SRE (serving infra, traffic routing, chaos tests)- Data Engineer (data migration scripts, compensating transaction logic, backups)- Product/PM (business rollback signoff, customer comms)- Incident Lead (coordinates rollback execution and postmortem)Pre-release drills (quarterly + pre-major-release):- Tabletop: simulate model/data failure, run through decision tree and comms- Staging canary: run full canary with production-sized traffic replay + chaos inject (latency, node kill, DB fail)- Live dark launch: route 1–5% traffic to new model, compare metrics- Rollback rehearsal: perform automated rollback in staging and record time-to-recoverRunbook (step-by-step):1. Detect: alert triggers based on KPIs (below)2. Triage: Incident Lead assembles owners within 10 minutes3. Short-term mitigation: reduce traffic via feature flag to canary or previous model (traffic-weight rollback)4. If data mutation occurred: - Quarantine affected records using audit logs - Run compensating transactions (idempotent) or restore from point-in-time snapshot to staging for verification - If immediate data rollback risk is high, flip feature flag to fail-safe path or frozen-mode5. Full rollback: deploy previous model version from registry, validate smoke tests, slowly re-increase traffic6. Post-rollback: run data integrity checks, notify stakeholders, begin postmortemRollback decision criteria:- Severity thresholds: - Critical: user safety/regulatory breach or >X% revenue impact → immediate full rollback - High: model-driven error rate > threshold OR >Y% negative business KPI delta for 10 minutes → escalate and reduce traffic to 0–5% - Medium: metric degradation detected but no immediate harm → prolong canary, run targeted A/B analysis- Evidence: reproducible error on canary, correlated telemetry, no downstream mitigation available- Authorization: Incident Lead + ML Engineer + Product owner must approve full rollback; SRE can perform traffic rollback immediately if CriticalData rollback strategies:- Preventive: write-through logs and versioned writes; use append-only event store for state changes- Compensating transactions: idempotent compensators that reverse business effects (examples: refund job, reverse label writes)- Feature-flagged schema migrations: behind flags, roll forward/backward safe- Short TTL soft-write: apply changes as soft flags until post-validation- Point-in-time DB snapshot + selective restore for non-live-critical datasetsChaos testing plan:- Staging: run systematic chaos scenarios daily during canary windows (node kill, network partition, delayed responses, DB fail)- Production-safe chaos: run on small percentage of canaries (0.5–1% traffic) during low hours with automatic abort on KPI breach- Validate rollback automation under chaos: ensure flag flips and traffic rebalancing complete within target RTOKPIs to validate readiness before release:- Pre-release: - Canary stability: latency P95 within SLAs, error rate < baseline + delta (e.g., +0.5%) - Data pipeline latency and schema compatibility checks: 0 schema mismatches - Rollback rehearsal success: automated rollback completes and restores baseline in <= target RTO (e.g., 5 min)- Live guardrails: - Model correctness: precision/recall / business metric delta thresholds - Customer-facing errors: < X per 100k requests - Time to mitigation (TtM): flag flip or traffic reduction < 2 minutes on alert - Time to recovery (TtR): full rollback + validation < SLA (e.g., 15 minutes) - Data integrity: post-rollback reconciliation mismatches = 0 (or acceptable bound)Trade-offs:- Conservative: slower rollout and frequent canaries increase safety but delay delivery- Aggressive: fast rollout with rollback automation reduces time-to-value but requires robust tests and observabilityExample quick commands (conceptual):- Feature flag: set model-version to previous in flag store → traffic shift via service mesh weight- Compensator job: run /jobs/compensate?since=2025-12-01&model=v2This plan ensures safe, auditable rollbacks, minimizes customer impact, and validates readiness through drills and measurable KPIs.
HardTechnical
54 practiced
Two teams are in conflict: the Platform team wants to freeze infra changes for stability to meet SLAs, while the Product team needs infra changes to improve model accuracy. As the program lead, describe the decision framework you would use to balance stability and product needs, temporary mitigations you might authorize, how you would document the decision, and how to communicate it to both teams and executives.
Sample Answer
Situation: Two teams disagree — Platform requests an infra freeze to protect SLAs; Product needs infra changes to improve ML model accuracy. As program lead I must balance availability risk and ML-driven business value.Decision framework (stepwise):1. Clarify objectives & constraints — required SLA targets, business impact of model accuracy uplift (revenue, retention, safety), compliance windows, release timelines.2. Quantify risks & benefits — estimate expected accuracy improvement, conversion or cost impact, probability and impact of infra instability, rollback cost.3. Define acceptance criteria — safety gates (error-rate thresholds), performance budgets (latency, memory), observability requirements.4. Prioritize via cost-benefit and risk tolerance — use a risk matrix and expected-value calculation (Benefit × Probability − RiskCost) and an SLA impact threshold that cannot be exceeded.5. Decision rule — approve change if expected business benefit outweighs quantified risk and mitigation plan reduces residual risk below threshold; otherwise defer or scope down.Temporary mitigations I’d authorize:- Canary/gradual rollout with traffic steering (1% → 10% → 100%) and automated abort on SLA breach.- Blue-green or shadow (dark) deployments so model inference runs in parallel without affecting production traffic until validated.- Feature flags to gate model behavior per customer cohort.- Short-term SLA exception window during low-traffic times (with exec sign-off) and rollback playbook.- Extra monitoring dashboards, alerting, SLO burn-rate alarms, and a dedicated on-call presence during rollout.- Resource isolation (quota/cgroups) to limit blast radius of infra changes.Documentation (what & where):- Create a concise RFC with: summary, business impact, technical design, risk assessment (risk matrix), acceptance criteria, rollback plan, mitigation steps, metrics to monitor, owners, timelines, and RACI. Store in shared repo (Confluence/GitHub PR).- Add a Risk Register entry and change ticket referencing the RFC and approval trace.- Include test results from staging/canary and runbook checklists.Communication plan:- To teams: Run a 30–45 min alignment meeting with Platform and Product to walk the decision framework, trade-offs, agreed acceptance criteria, and roles. Share RFC and a one-page checklist. Daily standups during rollout and a dedicated Slack channel for live updates.- To executives: Send an executive summary (1–2 bullets on decision, expected business impact, key risks, mitigation, and ask — e.g., approval for limited SLA exception). Provide scheduled briefings only if thresholds breach, with clear metric snapshots (SLA, latency, accuracy delta, traffic percentage).- Post-release: Share outcome report — metrics vs. targets, incidents, lessons learned, and next steps.Outcome & follow-up:- If rollout succeeds, bake successful mitigations into standard release playbook and update the freeze policy with exception criteria.- If not, perform a blameless postmortem, update RFC and risk register, and adjust the decision thresholds.This framework makes trade-offs explicit, uses data to decide, limits blast radius, and ensures transparent documentation and communication to both engineers and execs.
EasyTechnical
75 practiced
Describe a pragmatic approach to estimate initial compute (GPU/CPU hours), storage, and network needs for training and serving a model, given dataset size, a rough model class (e.g., logistic-regression, CNN, transformer), and expected traffic. Include benchmarking strategy, conservative assumptions, how to iterate estimates, and what early measurements you would collect during an initial PoC.
Sample Answer
Start by converting inputs into concrete numbers:- Dataset: bytes, number of samples, average sample size.- Model class: param count and per-sample FLOPs (rough values: logistic-regression << CNN << transformer).- Traffic: qps, batch sizes, latency SLOs.Estimate storage:- Raw + processed + replicas + checkpoints. Example formula: storage = raw + processed*(1.2) + checkpoints*(model_size * 10) + logs. Use conservative multiplier (×1.5–2) for growth.Estimate training compute:- Compute FLOPs ≈ samples_per_epoch * FLOPs_per_sample * epochs. Convert FLOPs → GPU-hours using GPU TFLOPS (account for utilization 40–70%). Add data-loading/IO overhead (20–40%). Example: transformer 1e15 FLOPs / (100 TFLOPS * 0.5) ≈ 5 GPU-hours.Estimate serving compute & network:- CPU/GPU per request = per-inference FLOPs / device TFLOPS; batch to amortize cost. Memory = model_size + runtime overhead. Network = avg request size * qps * 3600 s.Benchmarking strategy:- Small PoC microbenchmarks: single-node training on subset (1–5% data) to measure samples/sec, GPU utilization, IO throughput, memory. Inference benchmark: latency/p99, throughput at different batch sizes, cold vs warm start.- Run scaling tests: increase data and replicas to extrapolate linearly/nonlinearly.Conservative assumptions:- Use lower utilization (40%), higher epochs, 2× storage buffer, and network headroom (1.5×).Iterate estimates:- From PoC, compute empirical samples/sec and bytes/sec; recalc total GPU-hours = (total_samples / samples_per_sec) * wall_time_per_step. Update storage/network using measured compression and checkpoint frequency. Re-run benchmarks when changing model size, batch size, or data pipeline.Early PoC measurements to collect:- samples/sec, steps/sec, GPU/CPU utilization, GPU memory usage, disk and network IO, time per epoch, checkpoint time/size, end-to-end inference latency distribution, throughput vs batch size, cold start time, error rates.Use these measurements to produce a cost model (per training run and per month serving) and iterate before full-scale runs.
MediumTechnical
54 practiced
Draft a roadmap to integrate model monitoring, data-drift detection, and alerting into an ongoing ML program. Include a minimal viable monitoring set for the first release, how to scale monitoring coverage in later phases, alerting thresholds and owners, and how monitoring outcomes should trigger retraining or rollback actions.
Sample Answer
Phase 0 — Clarify requirements & constraints- Define SLAs (latency, accuracy), business KPIs, data sources, model types, and rollback authority.- Identify stakeholders: ML engineers (owner), data engineers, product manager, SRE, and business owner.Release 1 — Minimal Viable Monitoring (MVM)- Metrics (per model): prediction quality (online metric like accuracy/CTR or proxy), input feature distributions (mean, std), basic data integrity (schema, null rate), throughput & latency, error rate.- Drift signals: population-level PSI/KL for key features; label-delay-aware label rate monitor.- Alerts & thresholds: critical alerts (data pipeline failure, schema change) — immediate page to on-call SRE/ML owner; degradation alerts (drop in online metric >5-10% absolute or PSI >0.2) — email + Slack to ML owner and product owner.- Ownership: ML engineer owns model alerts; data engineer owns data pipeline alerts; SRE owns infra.Phase 2 — Expand coverage- Add per-segment metrics, feature-level drift (univariate + multivariate via MMD), calibration, prediction distribution shifts, feature importance changes, and adversarial input detectors.- Implement cohort monitoring, explainability drift (SHAP summary shifts), and label-quality monitors.- Automate dashboards (Grafana/Looker), lineage metadata, and weekly drift reports.Phase 3 — Automation & Actions- Implement decision rules: - Warning: single metric breach → investigate within 48h. - Action: sustained breach for 3-5 intervals or multi-metric breach → trigger retrain pipeline (staging) and A/B test. - Critical: sudden severe degradation (accuracy drop >15% or catastrophic pipeline failure) → automatic rollback to last validated model and create incident.- Retrain logic: gated CI for retrain; validation must pass offline metrics + shadow testing before promotion.- Auditing: log all alerts, actions, model versions; store datasets for reproducibility.Operational details- Use tooling: Prometheus/Grafana, Feast for features + lineage, Evidently/WhyLabs/Errant for drift, pagerduty/slack for alerts, MLflow/DVC for model/versioning.- Threshold tuning: start conservative, use historical data to set baselines and incorporate cooldown windows and suppression to reduce noise.- SLOs: define alert budgets and periodic review cadence (monthly).- Governance: define who can approve retrain vs rollback; require postmortem for critical incidents.Outcome: phased approach gives quick protection (MVM) then extends to robust, automated responses with human-in-loop safeguards and reproducible retraining/rollback.
Unlock Full Question Bank
Get access to hundreds of Program Level System Design interview questions and detailed answers.
