Business Acumen and Commercial Context Questions

Situation: After deploying a new model, overall conversions dropped 10% vs baseline.

Step-by-step investigation:

1. Triage & immediate checks (first 30–60 min)

• Confirm deployment: container/image version, feature flags, rolling vs canary, infra alerts.
• Check serving logs for errors, increased latency, timeouts, failed feature fetches, or fallback code paths.
• Verify data pipeline health: feature freshness, missing partitions, schema drift, upstream failures.

2. Metrics and distributions to examine

• Global conversion rate (overall), click-through rate, session start → conversion funnel steps, and latency.
• Prediction distributions: score histogram, mean/variance, tail behavior vs previous model.
• Confidence bins/deciles: conversion rate per predicted-probability bin to detect calibration shifts.
• Feature distributions: key features’ univariate stats and missingness rates; compare population vs training baseline.
• Secondary KPIs: revenue per user, churn, error rates, CPU/memory.

3. Traffic slicing

• Compare impact by cohort: user segment (new vs returning), geography, device/OS, client app version, traffic source, time of day.
• Per-user vs per-session aggregation to detect duplicate scoring or state issues.
• If rollout used gradual percentage, check conversion delta by bucket.

4. Rollout vs control comparison

• Use A/B metrics: compute lift, confidence intervals, and p-values for primary and secondary metrics; apply sequential testing corrections.
• Visualize time series for both cohorts to detect timing anomalies.
• Examine sample-level logs for matched control vs variant pairs—same user receiving different models across requests indicates sticky sessions problem.

5. Root-cause hypotheses & validation

• Hypothesis examples: feature missingness causing default predictions; calibration shift causing under-selection; latency/timeouts dropping conversions.
• Reproduce locally/with shadow traffic: replay recent production logs through both models to compare decisions and predicted scores per request.

6. Rollback criteria

• Roll back immediately if: statistically significant negative impact on primary metric (predefined threshold, e.g., >5% drop with p<0.01) or severe customer-facing failures (errors, timeouts).
• If impact is small/ambiguous: pause further rollout, switch to canary/0% or throttled percentage, continue investigation with targeted shadowing.
• Document findings, mitigation, and run a postmortem with root cause and preventive actions.

Key principles: prioritize user impact, use cohort-level and sample-level comparisons, validate features and predictions, and prefer quick rollbacks for clear production harm while using controlled experiments or replay to pinpoint causes.

Approach: define cohort as DATE_TRUNC('week', signup_date), count all users in that cohort, then count unique users who made at least one purchase within the first 7 days (signup day inclusive → signup + 6 days). Use COUNT(DISTINCT ...) to ensure each user is counted once even if multiple purchases.

-- Postgres: 7-day cohort conversion rate by signup week
WITH cohorts AS (
  SELECT
    user_id,
    signup_date,
    DATE_TRUNC('week', signup_date)::date AS cohort_week
  FROM users
),
purch_within_7d AS (
  SELECT DISTINCT
    c.cohort_week,
    c.user_id
  FROM cohorts c
  JOIN purchases p
    ON p.user_id = c.user_id
   AND p.purchase_date >= c.signup_date
   AND p.purchase_date <  c.signup_date + INTERVAL '7 days'
)
SELECT
  c.cohort_week,
  COUNT(DISTINCT c.user_id)                      AS cohort_size,
  COUNT(DISTINCT pw.user_id)                     AS converted_7d,
  ROUND(100.0 * COUNT(DISTINCT pw.user_id) / NULLIF(COUNT(DISTINCT c.user_id),0), 2) AS conversion_rate_pct
FROM cohorts c
LEFT JOIN purch_within_7d pw
  ON c.cohort_week = pw.cohort_week
GROUP BY c.cohort_week
ORDER BY c.cohort_week;

Key points and assumptions:

• Timezones: assume signup_date and purchase_date are stored in the same timezone (ideally UTC, use timestamptz). If stored as local timestamps, normalize with AT TIME ZONE before comparisons to avoid off-by-day errors.
• Users with no purchases: included in cohort_size via LEFT JOIN; they contribute zero to converted_7d.
• Unique converted users: COUNT(DISTINCT user_id) ensures each user is counted once even if they purchased multiple times within 7 days.
• Inclusive window: uses [signup_date, signup_date + 7 days) to include purchases on signup day and up to 6.999... days later.
• Edge cases: handle cohorts with zero users (NULLIF prevents divide-by-zero). If needed, adjust week-start (Postgres DATE_TRUNC week starts on Monday).

Framework: align model objectives to the business goal (increase purchases +5% QoQ), pick measurable proxies, design an experiment that isolates effect, and ensure safe rollout with operational constraints.

Offline proxy metrics (to iterate model):

• Purchase propensity AUC / PR-AUC on timestamped holdout predicting purchase within session (primary).
• Calibration (Brier score) to ensure probabilities map to conversion rates for ranking/thresholding.
• NDCG@k and MRR on historical clicked-and-purchased ground truth to measure ranking quality for top results.
• Uplift or causal uplift estimates (if available) to prefer items that change behavior.
• CTR and add-to-cart prediction accuracy as intermediate signals.

Online KPIs (primary → secondary):

• Primary KPI: Purchase conversion rate (purchases / sessions) and absolute purchases per day; target: +5% purchases vs control.
• Secondary KPIs: Revenue per session, average order value, add-to-cart rate, gross merchandise value (GMV).

Guardrail metrics:

• Search latency P95 and P99 (must stay within SLA).
• Click-through rate drop (no more than X% degradation, e.g., -2%).
• Refund/return rate and fraud indicators.
• Revenue per result position (ensure not cannibalizing high-margin items).
• Error rate / model-serving failures.

Experiment design:

• Randomized A/B test at user-session level; stratify by device, geography, and historical purchase propensity.
• Minimum detectable effect calculation: power 0.8, alpha 0.05, baseline conversion → compute sample size to detect 5% relative lift.
• Run for at least one full business cycle (7–14 days depending on traffic) and until sample size met.
• Monitor early-stopping rules for negative guardrails and use sequential testing with corrections.

Operational constraints & instrumentation:

• Latency: model inference P95 ≤ 100ms for ranking stage; P99 ≤ 300ms. If heavy re-ranking, use async enrichment.
• Compute: prefer lightweight models (tree ensembles or distilled DNNs) for ranking; use GPU for offline training, CPU-optimized serving in containers; autoscale pods.
• Instrumentation: log full action traces (query, results, features, model score, clicks, cart, purchase, user cohort) with unique correlation IDs; store deterministic bucketing info.
• Monitoring: real-time dashboards for KPIs and guardrails, automated alerts for threshold breaches, periodic model drift checks and calibration monitoring.
• Rollout: start with small % (1–5%) canary, expand to 25%, 50%, then full if safe. Keep rollback automation.

Reasoning: offline proxies ensure we iterate quickly on models correlated with purchases; randomized online experiments with proper stratification and guardrails isolate causal impact while protecting revenue and UX; strict latency and instrumentation guarantees production safety and observability.

Model-level accuracy (e.g., classification accuracy) measures how often a model’s predictions match labels on a test set. Measurable business lift measures the real-world impact of using the model—incremental revenue, conversion uplift, cost savings, churn reduction—usually estimated via experiments (A/B tests) or causal inference. Accuracy ≠ value: a high accuracy model can produce little or negative business lift if it’s biased, misaligned with decision thresholds, or ignores cost asymmetry.

Example 1 — Imbalanced fraud detection:

• Why accuracy misleads: 99% accuracy by predicting “not fraud” still misses almost all fraud cases; little fraud caught → high losses.
• Alternate metrics to present: precision, recall (or F1) for the fraud class, ROC-AUC, and business KPIs: dollars of fraud prevented per month, false negative cost, and expected loss reduction at chosen operating point.

Example 2 — Marketing churn/propensity scoring:

• Why accuracy misleads: A model with modest accuracy might prioritize customers who are inexpensive to retain or unlikely to respond; targeting by probability rank matters more than raw accuracy.
• Alternate metrics to present: uplift or incremental conversion (measured via randomized treatment on deciles), incremental revenue per 1000 contacts (or LTV uplift), and calibration/TPR at budgeted treatment rate.

When communicating to stakeholders, tie model evaluation to expected incremental metrics from experimentation (A/B test results, incremental ROI, cost per incremental conversion) and show operating-threshold trade-offs (precision/recall vs. cost) so decisions map directly to business outcomes.

Situation: After a deployment, offline cross-entropy decreased (better training/validation loss) but online KPI (e.g., CTR, conversion, revenue) worsened. Below is a prioritized diagnostic plan with hypotheses, instrumentation to add, tests to run, and remediation actions.

Priority 1 — Reward / Objective Mismatch

• Hypothesis: Training loss (cross-entropy) optimizes a proxy that doesn’t align with business KPI (e.g., log-loss vs. long-term conversion).
• Instrumentation: Log serving predictions, action taken, downstream rewards (click → purchase), time-to-convert, cohort-level KPI mapping.
• Tests: Compute correlation between predicted probabilities and actual business outcomes; run offline metric aligned with KPI (e.g., expected revenue). Run an A/B with a small percentage reverting to old model and compare long-term metrics.
• Remediation: Reweight or re-label training objective to reflect business reward (e.g., train on conversion labels, use uplift or multi-task loss), incorporate offline surrogate metrics that correlate with KPI.

Priority 2 — Training–Serving Skew

• Hypothesis: Feature computation differences in training vs. production (staleness, normalization, missing features).
• Instrumentation: Add feature-collection sidecar to production to snapshot raw inputs, computed features, normalization stats, and model version with timestamps; add checksums and schema validation.
• Tests: Replay production inputs through the training preprocessing pipeline and compare feature distributions; run unit tests for feature transformers; compute feature drift and distribution divergence (KL, PSI).
• Remediation: Unify feature code (use same library for train/serve), freeze preprocessing, add fallbacks for missing features, add monitoring + alerting for schema drift.

Priority 3 — Sampling / Selection Bias

• Hypothesis: Training data distribution differs from serving population (e.g., training on logged data with selection from a previous policy).
• Instrumentation: Track exposure policy, logging of which items/users were eligible vs. actually served, collect contextual covariates for both train and production.
• Tests: Compare covariate distributions for train vs. served population; use inverse propensity scoring to estimate bias; run off-policy evaluation to estimate performance under current policy.
• Remediation: Re-sample or re-weight training data using propensity scores; collect more representative data (exploration traffic) or run controlled randomized experiments.

Priority 4 — Exposure / Feedback Bias

• Hypothesis: Model changed what users see, altering downstream labels (e.g., new model surfaces different items causing lower long-term engagement).
• Instrumentation: Log full impression context, alternative-rerank candidates, and holdout buckets where candidate set is fixed; track multi-step user journeys.
• Tests: Use a shadow/diagnostic experiment where model scores are used only for logging (no effect on exposure) and compare outcomes; perform counterfactual and multi-step attribution analyses.
• Remediation: Introduce exploration (epsilon-greedy, Thompson sampling) to collect unbiased labels; incorporate sequential/causal models or bandit/uplift approaches.

Quick practical steps to start:

1. Roll back small % traffic to control model for A/B comparison.
2. Deploy logging instrumentation ASAP (predictions, features, candidate sets, timestamps, downstream events).
3. Compute feature drift, reward correlation, and propensity-weighted offline estimates within 48–72 hours.
4. Prioritize fixes that are low-risk/quick (fix skew, add logging), then address modeling objective and data collection changes.

This plan identifies likely causes, gives concrete instrumentation to detect them, describes targeted tests, and lists remediation paths prioritized by impact and implementability.