This topic covers the end to end practice of clarifying ambiguous problem statements, eliciting and defining functional and non functional requirements, and scoping solutions before design and implementation. Candidates should demonstrate the ability to identify target users and user journeys, conduct stakeholder interviews, ask targeted and probing clarifying questions, surface hidden assumptions and root causes, and convert vague business language into measurable technical and business requirements. They should capture acceptance criteria and success metrics, define key performance indicators, and translate requirements into testable statements and test strategies that map unit, integration, and system tests to requirement risk and priority. The topic includes assessing technical constraints and operational context such as expected scale, throughput and latency requirements, data volume and read write ratios, consistency expectations, real time versus batch processing trade offs, geographic distribution, uptime and availability expectations, security and compliance obligations, and existing system state or migration considerations. It also requires evaluation of non technical constraints including timelines, team capacity, budget, regulatory and operational concerns, and stakeholder priorities. Candidates are expected to synthesize inputs into clear artifacts such as product requirement documents, user stories, prioritized backlogs, acceptance criteria, and concise requirement checklists to guide architecture, estimation, and implementation. Emphasis is placed on scoping and prioritization techniques, distinguishing must have from nice to have features, conducting trade off analysis, proposing incremental or phased approaches, identifying risks and mitigations, and aligning cross functional teams on scope and success measures. Expectations vary by seniority: entry level candidates should reliably ask core clarifying questions and avoid solving the wrong problem, while senior and staff candidates should rapidly prioritize requirements, anticipate critical non functional needs, align solutions to business impact, and communicate trade offs and timelines to stakeholders.
MediumTechnical
52 practiced
Design an experiment and monitoring plan to detect model drift after deployment. Include what signals to collect, how often to evaluate, statistical tests or thresholds for drift, action thresholds to trigger retraining, and rollback criteria for severe regressions.
Sample Answer
Situation: After deploying a predictive model, we must detect data and concept drift early and act safely.Experiment & monitoring plan (high-level):- Signals to collect: - Input feature distributions (per-feature histograms, percentiles) - Model outputs (score distributions, confidence/entropy) - Ground-truth labels when available (latency-aware) - Key business KPIs (conversion, CTR, revenue per user) - Latency, error rates, and sample counts - “Golden set” of labeled examples reserved for regression tests- Evaluation cadence: - Real-time streaming for counts/alerts (per-minute) - Daily feature and output distribution snapshots; weekly detailed analysis with labels - Monthly full retrain validation if no triggers- Statistical tests & metrics: - For continuous features: Kolmogorov–Smirnov (KS) test or Population Stability Index (PSI). Flag if PSI>0.2 (moderate) and >0.25 (severe). - For categorical: Chi-squared or PSI for category frequency shifts. - For model output vs. expected: KL divergence or two-sample t-test on scores. - For label-based performance: bootstrap confidence intervals on metric (accuracy/AUC/F1); use Bonferroni or Benjamini-Hochberg when testing many features. - Minimum sample size: ensure n sufficient to detect a practical effect size (power analysis, e.g., detect AUC drop of 0.03 with 80% power).- Action thresholds / automated responses: - Warning tier: Any feature PSI in (0.1–0.2) or metric drop >2% absolute — notify data scientists; increase label collection. - Retrain candidate: PSI>0.25 for any critical feature OR label-based metric drop >5% absolute or outside 95% CI vs. baseline — trigger automated retraining pipeline (shadow evaluation). - Canary deployment: Deploy retrained model to 1–5% traffic for 24–72 hours; monitor model and business KPIs.- Rollback criteria (severe regression): - If canary model causes business KPI degradation beyond predefined tolerances (e.g., revenue down >1% or CTR down >3%) or label-based metric is significantly worse (p<0.01, effect size meaningful), immediately roll back to previous production model. - If retrained model fails golden-set checks (drop >2% absolute), reject and investigate. - Implement automated circuit breaker to route traffic back if latency/error spikes exceed SLOs.Practical notes:- Use sliding windows (baseline: last N weeks) vs. recent window (last M days) to compute drift; keep multiple baselines.- Automate alerts with contextual info (which features, magnitude, p-values, sample size) and run root-cause diagnostics (feature importance change, partial dependence).- Log everything for audit and reproducibility; include human-in-the-loop sign-off for production retrain if business impact is high.
HardTechnical
59 practiced
How would you quantify and operationalize 'model fairness' as an acceptance criterion for a lending decision model? Provide specific fairness metrics, required data collection and sampling strategies, automated tests, thresholds for acceptance, and a mitigation plan if thresholds are violated in production.
Sample Answer
Situation: We're deploying a credit scoring / lending decision model and must make "fairness" an explicit acceptance criterion.1) Fairness objectives and metrics (what to measure and why)- Disparate Impact (DI) / Ratio (selection rate minority / majority) — detects disproportional denial rates (regulatory shorthand). Target: DI ≥ 0.8.- Equal Opportunity Difference (EOD) — difference in true positive rates between groups (favours minimizing Type II errors for protected groups). Target: |EOD| ≤ 0.02.- False Positive Rate Difference (FPR diff) — controls wrongful approvals/denials imbalance. Target: |FPR diff| ≤ 0.02.- Calibration within groups (Brier or calibration slope) — ensures score meaning is consistent across groups (max calibration error ≤ 0.03).- Predictive parity (if required by regulation/business) — examine precision per group and trade-offs.Rationale: combination covers group-level rate parity, error parity, and score meaning; provides multiple lenses because single metric can hide harms.2) Data collection & sampling- Collect protected attributes required by regulation (race, gender, age) where legal; else collect via surveys, third-party data with consent, or use proxy monitoring with care and clear governance.- Ensure stratified sampling by protected attribute and outcome to achieve statistical power (power calc targeting 80–90% to detect metric diffs ≥ 0.02).- Maintain a held-out fairness evaluation set representative of application population and time windows (recent 6–12 months) plus synthetic stress subsets (low-income, newly-employed).- Log model inputs, scores, decisions, and outcomes (repayment) with timestamps for longitudinal monitoring.3) Automated tests (pre-deploy + continuous)Pre-deploy CI tests:- Compute DI, EOD, FPR diff, calibration error on validation and holdout sets; fail build if any metric violates thresholds.- Statistical significance tests (bootstrap CIs) for differences; fail if upper/lower CI violates thresholds.- Adversarial subgroup scans (smallest group with largest disparity).Production monitoring (daily/weekly):- Drift detection on input features and score distributions (KL divergence, population stability index).- Rolling-window fairness metrics with alerting if metric crosses threshold or trend worsens.- Explainability checks: feature importance shifts per group; sudden changes trigger investigation.4) Acceptance thresholds (example, adjustable by risk/regulation)- DI ≥ 0.80 (80% rule)- |EOD| ≤ 0.02 and |FPR diff| ≤ 0.02- Calibration error per group ≤ 0.03- Metrics not statistically significantly different beyond thresholds (95% CI)If any fail -> block deployment.5) Mitigation plan (if thresholds violated pre-deploy or in production)Immediate (short-term):- Roll back to previous safe model or engage human-in-loop for decisions for affected cohorts.- Throttle automated approvals for flagged subgroups.Root-cause & remediation:- Data fixes: re-balance training data, remove label bias, correct feature leakage or proxy features.- In-processing: reweighting (group-aware sample weights), constrained optimization (e.g., equality-of-opportunity constraint), or fairness-aware loss.- Post-processing: equalized odds post-processing or calibrated score adjustments for groups.- Model redesign: change architecture / features to remove discriminatory proxies; add causal features.Validation:- Re-evaluate using held-out fairness set and run CI tests.Governance & long-term:- Document decisions, add to model card, update stakeholders and legal counsel, retrain schedule, and periodic audits.6) Trade-offs and reasoning- Expect performance-fairness trade-offs; choose thresholds in consultation with compliance and product stakeholders. Prefer transparent mitigations (post-processing or constrained training) and maintain human oversight when automated remedies reach limits.This operationalization ensures measurable acceptance gates, continuous monitoring, and a practical remediation path balancing accuracy, legal risk, and social harm.
HardTechnical
59 practiced
Design a requirement-driven incident response playbook for post-release model behavior regressions. Include detection triggers, stakeholders/roles to notify, decision criteria for rollback vs hotfix, runbook steps, post-mortem requirements, and a sample customer communication template for an externally visible issue.
Sample Answer
Requirements & scope:- Incident: post-release model behavior regression (accuracy, safety, latency, distribution shift, hallucinations, privacy leaks) that degrades customer-visible SLA/contract or causes safety/PII/legal risk.- Goal: detect quickly, contain impact, restore safe baseline, learn and prevent recurrence.Detection triggers (automated + manual):- Automated: drop in key metrics vs production baseline (accuracy/F1 > X% relative drop, toxic content rate increase > threshold, perplexity/LL drop, latency > SLA), alert from canary/ shadow traffic, anomaly in input distribution (covariate shift), surge in user complaints or error rate.- Manual: high-severity customer report, security/legal escalation.Stakeholders & roles (notify in order):- Incident Lead (AI Eng on-call) — owner- ML Platform/SRE — infra/root cause- Model Owner / ML Researcher — model internals- Product Manager — customer/priority decisions- Trust & Safety / Compliance — safety/legal- Customer Success / Sales — external comms- Engineering Manager / CTO (for P0)Decision criteria: rollback vs hotfix- Rollback if: regression causes safety/legal violation, widespread user impact (>X% traffic), no quick confident patch available, or root cause tied to model artifact/version; rollback expected to restore SLA within defined RTO.- Hotfix if: localized/regional issue, fixable by config/tuning or small model patch (retrain on small dataset, filter rules) deployable within RTO without introducing risk; rollback would lose critical new functionality or cause additional downstream issues.- Use risk matrix: impact severity (safety/legal > correctness > perf) × probability of hotfix success × time to mitigate → choose option with lowest expected customer harm.Runbook (step-by-step)1. Triage (15 min): Incident Lead confirms trigger, assigns severity (P0/P1) using impact matrix, opens incident channel + doc.2. Containment (30–60 min): Shift traffic to canary/previous model or apply rate limit; enable safe-guards (blocking filters, response throttling); capture request/response logs, model inputs, system metrics.3. Investigation (concurrent): ML Engineer reproduces regression on stored inputs; SRE checks infra changes; check data pipeline, feature drift, model version, A/B config, tokenization, library updates.4. Mitigation decision (60–120 min): Apply rollback or hotfix per decision criteria. If rollback: deploy previous model with smoke tests. If hotfix: apply config rule, blacklist, or targeted retrain; run validation on production-simulated data.5. Verification (post-deploy): Monitor metrics for reversion; run canary for 30–60 minutes, then incrementally restore traffic.6. Communication: internal updates every 30 min; external comms prepared if customer-visible.7. Close: declare incident resolved when metrics stable for agreed window.Post-mortem requirements (blameless)- Timeline of events, root cause analysis (5 whys), contributing factors (process, test gaps), impact quantification (users affected, duration, business impact), remediation plan with owners and deadlines (short/medium/long), verification criteria, lessons learned, monitoring/test improvements (canary tests, data drift detectors), and follow-up review in 2 weeks.Sample external customer communication (template)Subject: Incident update — Model behavior regression detected and mitigatedBody:We detected a regression in [model name] on [date/time UTC] affecting [scope: e.g., response accuracy / safety] for [X%] of requests. Immediate action: we rolled back to the previous stable model at [time], restored expected behavior, and are monitoring. Impact: customers may have experienced [incorrect/harmful] responses between [start]–[end]. Next steps: conducting a full post-incident review; targeted remediation (owner) and timeline: [actions + ETA]. If you saw a problematic response, please reply with sample and request ID. We apologize for the disruption and will share the post-mortem when complete. Contact: [CS rep email] for urgent issues.Key preventive controls to include after incident:- Automated canary with production-like traffic and safety checks- Input distribution drift alerts- Pre-deploy safety gate with adversarial/unit tests and toxic-content filters- Versioned rollback playbook and one-click revert- Runbook drills and run retrospective improvementsThis playbook balances speed, safety, and customer trust; tune thresholds, RTOs, and stakeholders per org scale.
EasyTechnical
73 practiced
Write a user story (As a..., I want..., So that...) for an AI system that flags potentially fraudulent transactions. Include acceptance criteria, success metrics, and one non-functional requirement. Make the story suitable for a sprint planning conversation.
Sample Answer
As a fraud-operations analyst, I want an AI system that flags potentially fraudulent transactions in real time so that I can prioritize high-risk cases for investigation and reduce financial loss and manual review time.Acceptance criteria (MVP):1. The model scores each transaction with a fraud probability (0–1) and returns a top-3 risk reason tags (behavioral anomaly, velocity, device mismatch).2. Transactions with score ≥ 0.85 are marked "High Risk" and appear in the investigators' queue within 2 seconds of transaction completion.3. The system provides an audit log linking model score, input features, and reason tags for every flagged transaction.4. False positive alerts can be triaged and labeled by analysts; labels are stored for retraining.Success metrics (measurable, 30-day baseline):- Precision for "High Risk" alerts ≥ 75%- Recall for confirmed frauds ≥ 60%- False positive rate for High Risk alerts ≤ 5% of total flagged transactions- Mean time to first review for High Risk alerts ≤ 1 hour- Reduction in manual review workload for low-risk transactions by 30%Non-functional requirement:- Explainability: For any flagged transaction, generate human-readable rationale (top 3 contributing features and SHAP-like attribution) within the 2-second latency budget to support analyst decisions.Notes for sprint planning:- Scope this sprint: implement model inference endpoint, risk thresholding, reason-tag mapping, basic audit logging, and a simple UI queue showing High Risk items. Training pipeline, model retraining automation, and analyst labeling UI are deferred to subsequent sprints.
MediumTechnical
65 practiced
You have 6 weeks and one engineer to deliver an MVP: automatic summarization of incoming customer emails into bullet points for agents. Propose an MVP scope, success metrics (quantitative and qualitative), a three-phase roadmap (weeks 0-6 and post-MVP), and at least three mitigations for the top risks you see.
Sample Answer
MVP scope (6 weeks, 1 engineer)- Deliver an end-to-end pipeline that ingests incoming customer emails (IMAP/webhook), runs an automatic summarizer, and exposes results via a simple UI or API for agents.- Features: extractive+concise abstractive summary (3–5 bullet points), detected intent tags (e.g., refund, technical issue), confidence score, audit link to original email, manual “edit & approve” workflow.- Use a pre-trained LLM (few-shot prompting) or a fine-tuned lightweight transformer to save time.Success metrics- Quantitative: - Coverage: >= 85% of emails processed within SLA (<= 3s per email) - Utility: agent-accepted summaries >= 70% in A/B pilot - Precision of intent tags >= 80%- Qualitative: - Agent satisfaction score >= 4/5 in post-trial survey - Feedback: summaries judged “actionable” by agents in >60% sampled casesThree-phase roadmap- Week 0–1 (Discovery & infra): - Define schema, gather 200–500 labeled example emails, select model (LLM API vs local fine-tune), design simple UI/API, set success criteria.- Week 2–4 (Build & iterate): - Implement ingestion, prompt templates or fine-tune model, create summarization pipeline, add intent classifier, confidence scoring, logging, and unit tests. - Internal testing with synthetic and historical emails; iterate prompts/models.- Week 5–6 (Pilot & measure): - Deploy feature-flagged pilot for small agent group, collect acceptance decisions and qualitative feedback, fix UX issues, measure metrics.- Post-MVP (weeks 6+): - Improve model via supervised fine-tuning on agent edits, add multi-language support, scale infra, integrate into CRM, add RLHF/feedback loop, and advanced routing rules.Top risks and mitigations1) Model hallucination / incorrect facts: - Mitigations: prefer extractive+constrained abstractive prompts; show source snippets with each bullet; add confidence thresholds and require human approval below threshold.2) Data/privacy concerns (PII in emails): - Mitigations: PII redaction pre-processing, encryption at rest/in transit, use on-prem or VPC-hosted model if required, delete training data after fine-tuning unless consented.3) Insufficient labeled data for fine-tune: - Mitigations: start with prompt engineering on LLM APIs, bootstrap labels from heuristic extractive rules, run rapid human-in-the-loop labeling during pilot to collect edits.4) Latency / cost at scale: - Mitigations: cache summaries, batch processing for low-priority emails, fallback to lightweight local model when API cost/latency spikes.This plan focuses on fast delivery, human oversight, measurable signals, and clear paths to improve quality and safety post-MVP.
Unlock Full Question Bank
Get access to hundreds of Requirements Elicitation and Scoping interview questions and detailed answers.
