Rapid Analysis and Decision Making Questions

Assesses the ability to quickly understand a problem, identify root causes, prioritize actions, and make decisions under time pressure. Candidates should demonstrate structured thinking, time boxing, quick hypothesis generation and validation, clear communication of trade offs, rapid experiment design, and sensible risk based prioritization. This evaluates how well a candidate can produce an actionable plan with limited time and information and iterate as new signals arrive.

MediumTechnical

25 practiced

You observe a sudden spike in feature importance for a recently added feature in production. What rapid checks would you run to ensure this is not feature leakage or an analytics instrumentation bug? List the tests and expected outcomes for each.

Sample Answer

Brief preface: run fast, evidence-first checks that separate (A) data/feature problems (leakage or distribution shift) from (B) instrumentation/importance-computation bugs. For each test I list the action and the expected “safe” outcome; deviations flag further investigation.

1) Compare feature distribution: prod vs training (histograms, KS/PSI)- Action: compute PSI and KS; plot time series.- Expected: PSI < 0.1 (minor), no abrupt distribution jump. Large shift suggests data change or bad source.

2) Correlation with label and time-to-label- Action: compute Pearson/ROC-AUC of feature vs target and correlation with label lead/lag; check whether feature is computed using future info (timestamp alignment).- Expected: Moderate correlation consistent with training; no strong correlation that appears only in prod, and no dependence on future timestamps.

3) Importance method sanity: recompute using permutation importance and SHAP on a recent batch- Action: run permutation importance (drop-feature) and SHAP; compare ranking to the reported spike.- Expected: Consistent ranking across methods. If only one method spikes, suspect importance-computation bug.

4) Train vs serve code parity- Action: diff feature engineering code/path (packaging, featurization functions) between training notebook and production inference service; verify versions.- Expected: Identical logic; if prod uses different code/version, root cause likely engineering.

5) Missing / default value patterns- Action: check counts of nulls, sentinel/default values in prod for that feature and for related upstream fields.- Expected: Similar null/default rates as training; a sudden sentinel concentration suggests instrumentation bug.

6) Uniqueness / cardinality check (IDs, hashes)- Action: compute cardinality, entropy; look for near-unique values or repeated sentinel.- Expected: Cardinality similar to training. Near-unique values or hashed IDs leaking row-level info is dangerous.

7) Time window / leakage replay test- Action: simulate model scoring on historical data with only information available at scoring time (causal cut); recompute importance.- Expected: Importance returns to baseline; if it stays high, feature likely legitimate signal.

8) A/B or shadow test- Action: run a shadow model that excludes the feature on a sample of live traffic, compare predictions and downstream metrics.- Expected: Predictions and downstream metrics consistent; dramatic divergence indicates feature materially affecting outputs—investigate for leakage.

9) Logging & instrumentation audit- Action: validate telemetry: Are feature values being computed/stored twice? Check pipeline joins, timestamps, and logging offsets.- Expected: Clean, single-source telemetry; duplicates or join-on-wrong-key suggest instrumentation error.

10) Quick root-cause checklist- Action: look for recent deployments, pipeline changes, schema evolutions, new upstream sources in last deploy window.- Expected: No unreviewed changes; if there are, correlate timing with spike.

If any test fails, prioritize reproducing the problem offline (replay), roll back the offending change or remove the feature from production, and create a postmortem with fixes: stricter causal checks, unit tests for featurization, and monitoring alerts (PSI, feature importance drift).

EasyTechnical

32 practiced

After an automated alert that a deployed classifier's accuracy dropped by 8%, list the top five signals you would check in the first 10 minutes (e.g., input distribution, recent code deployments). For each signal, say why it matters and what a quick pass/fail check looks like.

MediumTechnical

29 practiced

You need to reduce GPU training costs by ~50% with no more than a 0.5% absolute drop in model performance. In a 3-day focused effort, list prioritized strategies you would try (e.g., mixed precision, smaller model, gradient accumulation), why, and quick validation steps for each.

Sample Answer

Priority order (3-day focused effort). For each: why, quick experiment to validate (fast proxy + full check).

1) Mixed precision (AMP, bfloat16)Why: ~1.5–2x throughput on modern GPUs; minimal accuracy loss for most nets.Quick validation: Enable AMP and run 1–2 epoch mini-run on 10% of data; compare loss/val metric delta and throughput. If proxy looks good, run full training and verify <0.5% absolute drop.

2) Increase effective batch size via gradient accumulation / larger physical batchWhy: Better GPU utilization → fewer steps = lower wall time/cost. Larger batch can require LR tuning.Quick validation: Double effective batch size using accumulation; train 1 epoch on small subset, check training dynamics; run a short full-train with adjusted LR (linear scaling) and compare final val metric.

3) Reduce model compute (smaller width/depth, efficient blocks)Why: Directly cuts FLOPs and memory; high ROI if over-parameterized.Quick validation: Train a compressed variant (e.g., -25% params or replace blocks with mobile/efficient versions) on quick 25% schedule; measure val metric. If within budget, run full retrain.

4) Knowledge distillationWhy: Student model retains performance of teacher with fewer computations.Quick validation: Train student for few epochs on distillation loss using pretrained teacher logits; evaluate student on validation set for metric and speed.

5) Gradient checkpointing + memory optimizationsWhy: Allows larger batches/inputs without bigger GPU; reduces OOM-driven inefficiency.Quick validation: Enable checkpointing and rerun a standard iteration profile; measure throughput and peak mem. Validate final metric on short run.

6) Data and I/O optimizations (prefetch, num_workers, sharded TFRecords)Why: Avoid CPU-GPU stalls, increases GPU utilization without model change.Quick validation: Profile data pipeline; apply optimizations and measure GPU utilization and epoch time on a small run.

7) Shorter schedule via better hyperparams (cosine LR, warmup, early stopping)Why: Fewer iterations while preserving accuracy.Quick validation: Run hyperparam sweep on subset (bayesian/ASHA) for epochs budget reduced by 20–40%; compare final metric.

8) Use cheaper hardware options (spot preemptible, multi-GPU with model parallelism) and autoscalingWhy: Lower $/GPU-hour or faster wall time with more GPUs.Quick validation: Cost/run a benchmark on candidate instance types; confirm preemption recovery and throughput.

Measure & decide:- Day 1: AMP + profiling + batch/IO changes (low risk, high ROI).- Day 2: Model size tweaks, checkpointing, short retrains.- Day 3: Distillation or schedule tuning and full verification.Track: GPU hours, throughput, and validation metric; stop any path that exceeds 0.5% absolute drop.

HardTechnical

31 practiced

A vendor change introduced an upstream sampling bias that caused deployed models to fail for certain segments. The vendor is critical. Within 48 hours, how would you decide to keep the vendor, switch, or implement mitigations? Include quick diagnostics, cost/risk evaluation, and a temporary mitigation plan.

Sample Answer

Approach: treat this as an incident — fast diagnostics, triage, short-term mitigation, and decision criteria for vendor retention vs replacement. I’d run this as a 48-hour incident with clear owners, checkpoints, and escalation.

1) Quick diagnostics (0–6 hours)- Reproduce & scope: sample recent upstream inputs and label distribution by segment (geography, device, cohort). Quantify drift vs baseline (KL divergence, PSI) and identify affected segments.- Model impact: run a fast offline inference on held-out validation and recent production data to measure delta in key metrics (accuracy, AUC, calibration, false-positive/negative rates) per segment.- Pipeline trace: check vendor change logs, SDK versions, configuration; capture timestamps when bias started.Output: a short report with severity (e.g., % of traffic degraded, SLA/financial impact estimate) to stakeholders.

2) Triage & cost/risk evaluation (6–12 hours)- Risk axes: user/customer impact, regulatory/compliance, revenue, brand, technical debt/time-to-repair.- Options: a) Keep vendor + mitigate: lowest procurement disruption, moderate engineering effort; risk of recurrence. b) Switch vendor: high procurement and integration cost, longer timeline (weeks–months), lower short-term cost but higher operational risk. c) Temporary bypass (fallback): lowest risk operationally if feasible.- Decision criteria (quantitative): if >5–10% of traffic or key-segment error increases >X% or regulatory breach, lean to isolate vendor; otherwise prefer mitigations and audit.

3) Temporary mitigation plan (12–48 hours)- Immediate traffic gating: route affected segments away from vendor output using a feature flag or traffic split in the inference layer.- Fallback model: serve last-known-good model or a simpler rule-based policy for affected segments. Deploy behind a feature flag to rollback quickly.- Input validation & sanitization: add a lightweight validator that rejects or normalizes suspicious upstream samples based on schema/feature ranges.- Reweighting/harmonization: apply importance sampling or inverse-propensity weights to counter sampling bias for short-term scoring adjustments.- Monitoring & alerting: add segment-level dashboards, thresholds, and automated rollback triggers.

4) Cross-team actions & communication- Notify vendor immediately with reproducible examples and request root-cause + rollback within SLA window.- Align with legal/procurement on contract remedies and data contracts.- Inform Product/Support with a customer-facing script if user impact is visible.

5) Next steps (post-48h)- If vendor remediates quickly and recurrence risk low -> keep with added SLAs, continuous audits, and a pre-specified canary testing pipeline.- If vendor fails to remediate or systemic trust broken -> begin parallel integration evaluation of alternatives, extract lessons for automated upstream validation, and update vendor contracts.

This plan balances immediate user protection, minimal disruption, measurable decision criteria, and a clear path to either retain the vendor with stronger guardrails or replace them when justified.

HardTechnical

43 practiced

A downstream feature store introduced a bug that corrupted inputs for several models, causing wrong predictions across services. With limited logs, outline a prioritized forensic plan to identify scope, immediate mitigations to stop damage, and follow-up actions to prevent recurrence.

Sample Answer

Situation: Several production models across services began returning incorrect predictions after a downstream feature store introduced a bug that corrupted inputs. Logs are limited, so we need a fast, prioritized forensic response to stop damage, determine scope, and prevent recurrence.

Prioritized forensic plan (immediate, within hours)1. Triage & coordination (15–30 min)- Convene incident call with feature-store, model owners, SRE, and product stakeholders.- Assign roles: Forensics lead (owner), Mitigation lead, Communication lead.

2. Stop the bleeding (minutes)- Freeze downstream model deployments (disable automatic rollouts) and route new traffic to a safe path: - Switch critical services to last-known-good features (rollback feature store snapshot) OR - Activate model fallback logic (simple rules/heuristics) for high-risk decisions.- If rollback isn’t possible quickly, throttle traffic or switch to read-only feature-store mode.

3. Rapid scope identification (1–2 hours, parallel)- Identify affected feature keys and timestamps via feature lineage: feature definitions, last-change commit, and deployment time of the buggy code.- Compare distributions: pull small samples of feature-store reads vs. historical baselines (mean/std, quantiles) to detect corrupted ranges.- Instrument model inputs: capture raw input vectors for a quick audit (enable enhanced sampling/trace for 5–10% traffic).- Query downstream services for recent error spikes, degraded metrics, or anomalous outputs.

4. Evidence preservation (immediate)- Snapshot current feature-store state and any available logs; export samples to object storage with checksums.- Capture model input/output traces, request IDs, and timestamps for replay.

5. Root cause analysis (next 4–8 hours)- Reproduce corruption in a sandbox by replaying inputs through the faulty feature-store change.- Inspect code/config change that introduced bug: schema mismatch, type coercion, joins, TTL/aggregation bug.- Check upstream data source health and transformations.

Mitigations (short term)- Roll back to a known-good feature-store version or restore from snapshot.- Deploy validation gating: reject feature writes/reads that violate schema or statistical checks.- Enable model-level input validation (sanity checks, missing-value handling) and circuit breakers to prevent bad predictions from affecting downstream systems.

Follow-up actions (days–weeks)- Implement automated data quality & anomaly detection in the feature pipeline: schema checks, value-range, distributional checks, and drift detectors with alerting to on-call.- Add end-to-end feature/data lineage with traceable commits and deploy tags; enforce data contracts and CI tests for feature code.- Expand logging and observability: structured logs for feature read/write, per-request IDs, sampling of raw features for X days.- Improve deployment practices: require canary with shadow traffic that validates model outputs against expected ranges before full rollout.- Post-incident retrospective and blameless RCA: document timeline, decisions, and a clear remediation owner for each action. Track metrics to verify fixes.- Add runbooks and tabletop exercises for similar incidents; set SLAs for detection and mitigation.

Why this approach- Quickly isolates impact, preserves evidence for accurate RCA, and prioritizes user protection. Immediate rollbacks and input validation stop damage; lineage, monitoring, and stricter CI/CD prevent repeats.

Unlock Full Question Bank

Get access to hundreds of Rapid Analysis and Decision Making interview questions and detailed answers.

Join thousands of developers preparing for their dream job.