AI and Machine Learning Background Questions
A synopsis of applied artificial intelligence and machine learning experience including models, frameworks, and pipelines used, datasets and scale, production deployment experience, evaluation metrics, and measurable business outcomes. Candidates should describe specific projects, roles played, research versus production distinctions, and technical choices and trade offs.
HardTechnical
84 practiced
Estimate the monthly cost (compute, storage, and egress) of running inference for an LLM-based feature expected to serve 10M requests per day with an average response of 200 tokens. State your assumptions (model size, tokens per second, cloud unit prices), propose caching strategies and batching optimizations, and estimate potential cost reduction from those optimizations.
Sample Answer
Assumptions (clear, high-level)- Traffic: 10M requests/day → 300M requests/month (30 days).- Avg response: 200 tokens per request.- Model: 13B parameter LLM (medium-sized) served on GPU with ~50 tokens/sec throughput per GPU (conservative).- Prompt tokens negligible relative to response.- Cloud unit prices (example, round figures): GPU instance (NVIDIA A10-like) $2.50/hour; vCPU/CPU-only inferencing for small models $0.20/hour (not primary here). Storage: $0.02/GB-month. Egress: $0.08/GB.- Token size: assume 4 bytes/token → 200 tokens ≈ 800 bytes ≈ 0.0008 MB. Add protocol overhead -> use 1 KB/request ≈ 0.000001 GB/request.Compute baseline- Throughput per GPU: 50 tokens/sec → 0.25 requests/sec (200 tokens each) → 900 requests/hour per GPU.- Requests/month per GPU: 900 * 24 * 30 ≈ 648k.- GPUs needed = 300M / 648k ≈ 463 GPUs.- GPU cost/month = 463 * $2.50/hr * 24 *30 ≈ $836k/month.Egress baseline- Data egress/month = 300M requests * 0.000001 GB ≈ 300 GB.- Egress cost = 300 GB * $0.08 ≈ $24/month (negligible).Storage- Model storage (13B): ~20 GB + replicas & logs => 100 GB total => $2/month. Logging/user data maybe 1 TB => $20/month.Total baseline monthly ≈ $836,046 ≈ $836k.Optimizations1) Caching:- Cache identical or similar prompts (e.g., common queries). Assume 10% cache hit reduces requests to compute by 10%.- Cost reduction: ~10% → ~$83.6k saved.2) Batching & micro-batching:- If server can batch to raise throughput: batch size 4 increases tokens/sec aggregate; effective requests/hour per GPU could rise by ~3x (depends on latency SLA). Conservative: 2x improvement.- GPUs needed halved → saves ~50% of compute → ~$418k saved.3) Model distillation / small specialized models:- Replace with distilled 3B model for 70% of requests (where high fidelity not required). 3B can run on cheaper GPUs or even optimized CPU instances. Assume cost per 3B-equivalent is 25% of 13B cost.- Savings: for 70% of requests, compute cost reduces from $2.50/hr equiv to $0.625/hr → overall compute drop ~ (0.7*0.75)=52.5% of baseline compute cost saved → ~$438k.Combined realistic optimizations (cautious stacking)- Apply caching 10%, batching 2x, and distillation for 70%: approximate compute cost becomes: baseline $836k * (1 - 0.10 cache) * 0.5 (batching) * (0.3 + 0.7*0.25) ≈ compute ~$836k * 0.9 *0.5 * (0.3+0.175)=836k*0.45*0.475≈$178k.- Add egress & storage ≈ $50 → total ≈ $178k/month.Takeaways for PM- Baseline LLM inference at this scale can be ~ $800k/month for medium-sized models.- Practical optimizations often cut costs 4–5x; combined strategies can reduce to ~$150–250k/month.- Prioritize: (1) measure cache hit potential (analytics), (2) invest in batching & serving infra, (3) evaluate distilled/smaller models for lower-cost tiers, (4) monitor latency vs cost trade-offs and rollback where quality matters.
MediumTechnical
64 practiced
Compare evaluating third-party vendor ML APIs versus building in-house for an image moderation capability. As PM, list evaluation criteria including accuracy, customization, latency, data privacy, cost, SLAs, and vendor lock-in; then walk through a concise cost-benefit analysis and go-to-market timeline for each option.
Sample Answer
Evaluation criteria (what I’d score and why):- Accuracy / precision & recall: false positives hurt UX; false negatives risk compliance. Measure on holdout labeled set and target thresholds (e.g., >95% precision for nudity).- Customization: ability to add classes, tune thresholds, create domain-specific rules.- Latency: tail P95/P99 response times for real-time flows (target <200ms P95).- Data privacy / residency: support for private hosting, no-retention agreements, SOC2/GDPR compliance.- Cost: per-call + storage + annotation + infra (TCO 1yr & 3yr).- SLAs & support: availability, incident response time, model update cadence.- Vendor lock-in & portability: exportable models/rules, SDKs, ease of switching.- Observability & tooling: explainability, audit logs, retraining pipelines, false-positive review UI.Concise cost–benefit- Third-party API: - Benefits: fastest GTM, minimal infra, continuous model updates, predictable per-call costs. Good for MVP or low-moderation volume. - Costs/risks: recurring cost at scale (e.g., $0.01–$0.10/image), limited customization, data residency risk, potential vendor rate spikes and lock-in.- In-house build: - Benefits: full customization, control over data, lower marginal cost at high scale, avoid vendor dependence, tailored explainability. - Costs/risks: high upfront engineering, labeling/annotation (~$50–$200k depending on scope), ops burden, slower model refresh cadence.Go-to-market timeline (rough)- Third-party: 4–8 weeks - Week 1: vendor evaluation + legal review (privacy/SaaS contract) - Weeks 2–3: integration + feature flags + pilot on 1% traffic - Weeks 4–6: tune thresholds, build review queue, analytics, SLA verification - Week 8: full rollout- In-house: 20–36 weeks - Weeks 1–4: requirements, dataset collection plan, annotation kickoff - Weeks 5–12: model development + baseline evaluation - Weeks 13–20: infra, monitoring, human-in-loop tooling - Weeks 21–30: productionization, A/B testing, scale optimization - Weeks 31–36: compliance audit and full rolloutRecommendation heuristic:- Use third-party for MVP or if time-to-market and limited budget/engineering capacity matter, with contractual data protections and a clear migration plan.- Build in-house when scale (high volume) or strict data/control/customization needs justify upfront investment; consider hybrid: start with vendor + parallel in-house model development to switch later.
MediumTechnical
71 practiced
How would you translate a business KPI like 'increase 30-day user retention by 5%' into an ML hypothesis and measurable model metrics? Provide a concrete example mapping product metric → model objective → evaluation metric → required model performance improvement to achieve the business KPI.
Sample Answer
Framework: translate business KPI → causal hypothesis → product metric change → model objective → evaluation metric → required model lift.Example:Business KPI: Increase 30-day retention from 40% to 45% (+5 ppt).1) Causal hypothesis- Hypothesis: Personalized onboarding nudges for new users will increase first-week engagement, which causally increases 30-day retention.2) Product metric path- Leading product metric: % of new users who complete onboarding flows within 7 days (current = 50%).- Historical conversion→retention relationship: users who complete onboarding have 30-day retention 60%, others 20%. Weighted baseline retention = 0.5*0.6 + 0.5*0.2 = 0.4 (matches current 40%).3) Model objective- Build a personalization model that predicts which onboarding variant (A/B/C) maximizes completion probability and selects variant per user (contextual bandit or multi-arm uplift model).- Objective: increase onboarding completion rate among targeted users.4) Evaluation metric (model-level)- Primary: delta in onboarding completion rate attributable to model (uplift in completion % among served users).- Secondary: predicted uplift AUC / Qini for uplift model, and average treatment effect (ATE) in an RCT.5) Required model performance to hit business KPI- Let p = new overall completion rate needed to reach 45% retention.- Solve p*0.6 + (1-p)*0.2 = 0.45 → 0.6p + 0.2 - 0.2p = 0.45 → 0.4p = 0.25 → p = 62.5%.- Current p = 50%, so absolute increase needed = 12.5 ppt (relative +25%).- So model must drive a 12.5 ppt uplift in onboarding completion across new users. If model targets only 50% of new users, required uplift among targeted = 25 ppt.Implementation considerations:- Run an RCT to validate causal link and measure ATE.- Instrument cohorts: control, model-serving, and generic personalization.- Monitor attribution window (7-day completion, 30-day retention), and guard against novelty / selection bias.This mapping makes the business KPI actionable, sets a clear model goal (maximize onboarding completion uplift), defines measurement (ATE, Qini), and quantifies the required performance to achieve the 5ppt retention gain.
HardTechnical
73 practiced
Design metrics and an operational process to measure and minimize 'hallucination rate' for an LLM assistant integrated into product. Define a precise operational definition of hallucination, describe how to sample and label examples, set acceptable thresholds, and outline remediation flows when thresholds are exceeded.
Sample Answer
Operational definition- A hallucination is any assistant statement that presents false, unverifiable, or unsupported facts as true relative to a defined ground-truth source set. Operationally: given prompt P and context C, response R contains one or more assertions A where (a) A contradicts authoritative sources or (b) A cannot be traced to the allowed knowledge sources and is materially misleading to a reasonable user.Sampling & labeling- Sampling: stratified random sampling by use-case (support, recommendations, code, citations), user segment, and confidence score. Sample size: daily rolling sample of 500 responses for high-risk flows, 200 for medium-risk, 50 for low-risk.- Labeling process: two-stage human review. Stage 1: trained raters mark each assertion as “Verified”, “Unsupported”, or “False” with evidence link and severity (minor/major). Stage 2: adjudication by senior reviewer for disagreements (>20% disagreement triggers retraining).- Labeling guide: include examples, allowed sources, threshold of evidence required (e.g., primary docs, product DB, up-to-date web).Metrics- Hallucination Rate (HR) = (# responses with ≥1 False assertion) / (sample size).- Unsupported Assertion Rate (UAR) = (# responses with ≥1 Unsupported assertion) / sample size.- Severity-weighted HR = sum(severity_score per false assertion)/sample size.- Time-series: 7-day rolling HR, per-flow HR, per-model-version HR.Acceptable thresholds (initial, adjustable)- Global HR target ≤ 1% (production conservative).- High-risk flows (legal/medical/finance): HR ≤ 0.1%.- Medium-risk: HR ≤ 0.5%.- UAR target ≤ 2% overall.- Severity-weighted HR must remain below set business-impact budget.Monitoring & alerting- Automated dashboards with per-flow and per-version HR, UAR, and CI.- Alerts: paging for immediate remediation when high-risk HR > threshold or sudden delta > 3x baseline. Warning emails when medium-risk crosses thresholds.Remediation flows1. Triage (automated + human): - On-alert, collect recent flagged responses, model version, prompt patterns, and confidence metadata.2. Containment: - For critical flows, rollback to previous model or disable the assistant for that flow; enable canned-safe fallback responses.3. Root cause analysis: - Engineering and PM run RCA: prompt engineering failures, training-data drift, retrieval errors, indexing/staleness, or model hallucination.4. Fix implementation: - If retrieval issue: refresh/patch knowledge source, add retrieval filters. - If prompt flaw: change prompt templates and add guardrails (e.g., "I don't know" policy). - If model issue: fine-tune on corrected examples, add contrastive examples, or update system message.5. Verification: - Run focused A/B test with a labeled sample to verify HR reduction; require passing threshold for 7 consecutive days before full rollout.6. Post-mortem and prevention: - Update labeling guide, expand automated detectors (fact-checker modules), add synthetic tests for new failure modes, schedule retraining of raters.Governance & longitudinal improvement- Quarterly reviews tying HR to business KPIs (NPS, support load, regulatory risk).- Maintain a hall-of-fame of hard examples and incorporate into continuous training.- SLAs with engineering for remediation time: critical incidents resolved (contain+fix) within 24–72 hours.This framework balances precise, evidence-driven metrics with operational processes so product, engineering, and safety teams can measure, prioritize, and reduce hallucinations while keeping business risk and user trust under control.
MediumTechnical
61 practiced
Design a high-level measurement plan to evaluate a new search ranking model intended to increase user engagement by 10%. Specify primary metric, secondary/guardrail metrics, required sample size considerations, segmentation strategy, and how you would test for unintended harms such as reduced diversity or increased latency.
Sample Answer
Goal: increase user engagement by 10% from the search ranking change. Measurement plan below.Primary metric- Definition: Relative change in engagement rate per search session = (sessions with engagement ÷ total search sessions). Engagement = any downstream signal you care about (click-through to content, long dwell ≥30s, add-to-cart, query reformulation avoidance). Choose the most business-relevant single KPI (e.g., “sessions with long-dwell click”).- Evaluation: Compare treatment vs control uplift; target = +10% relative uplift.Secondary / guardrail metrics- CTR (first-click and overall)- Average dwell time per clicked result- Query success rate (no reformulation within 30s)- Relevance complaints / negative feedback rate- Diversity metrics: result-type entropy, unique domains per query, coverage of long-tail queries- Performance: latency p50/p95/p99, error rate, backend CPU/memory signals- Business metrics: conversions, revenue per session (if applicable)Sample size & statistical plan- Use power calculation for a proportion uplift. For baseline engagement p0, desired relative uplift r (10%), absolute uplift = p1 − p0 = p0*r.- Parameters: alpha=0.05, power=0.8. Compute required N per arm using normal approximation of proportions or simulations for small p0.- Example: if p0=0.10 and r=0.10 → p1=0.11. That small absolute change (~0.01) needs large N (~>30k–100k sessions per arm). Run computations with standard formula or A/B tool.- Adjust for clustering (user-level randomization preferred) and expected attrition; add 10–20% buffer.- Pre-register primary metric, stopping rules; avoid peeking—use sequential testing corrections (alpha-spending) if doing frequent checks.Segmentation strategy- Pre-specified segments to detect heterogenous effects: - Query intent: informational, navigational, transactional - Frequency: new vs returning users; power users - Query popularity: head vs tail queries - Device: mobile vs desktop; network conditions - Geography / locale / language- Analyze treatment effects by segment; treat segmentation analyses as exploratory unless powered; correct for multiple comparisons (Benjamini-Hochberg or Bonferroni for confirmatory checks).Testing for unintended harms- Reduced diversity: compute per-query result entropy, fraction of queries where top-3 are from same domain, long-tail coverage. Compare distribution shifts (KS test) and per-query deltas; set thresholds (guardrail triggers) for unacceptable decreases.- Personalization regressions: for users with personalized profiles, measure change in relevance for past-satisfied cohorts.- Latency and reliability: monitor p95/p99 latency and error rates in real time; establish SLOs and automatic rollback triggers if exceeded.- Fairness/quality regressions: monitor query classes affecting vulnerable groups, mis-rankings on sensitive queries.- A/B test instrumentation: collect per-query footprints (rank lists, features) to enable root-cause analysis.- Statistical approach: use both aggregate tests and per-query holdout buckets; for many per-query tests, apply hierarchical modeling or shrinkage to avoid noisy false positives.Rollout plan- Start with small % of randomized users (1–5%) for smoke test, monitor guardrails closely for 48–72 hours.- Ramp to larger sample after no red flags, then full experiment at planned sample size and duration (min 1–2 weekly cycles to capture temporal patterns).- If primary metric significant and no guardrail violations, phased rollout with continued monitoring.Deliverables for stakeholders- Pre-registered experiment plan (metric definitions, sample size, timeline, segments, guardrails)- Dashboards: primary KPI, segments, diversity, latency p95/p99, error rates, and alerts- Post-mortem with per-query impact analysis and recommended mitigations if harms found.This plan balances statistical rigor, business focus, and active monitoring to detect both uplift and unintended harms before full release.
Unlock Full Question Bank
Get access to hundreds of AI and Machine Learning Background interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.