Problem Solving and Scenario Analysis Questions

Candidates are expected to demonstrate a systematic, structured approach to analyzing and resolving complex scenarios relevant to their field. This includes clarifying the problem statement, eliciting requirements, constraints, and assumptions, and identifying missing information or ambiguous areas. Candidates should decompose complex problems into logical components, prioritize tasks or evidence, generate multiple solution options, and perform trade-off evaluation that balances impact, feasibility, cost, and risk. Core skills assessed include root cause analysis, structured diagnosis of an incident or issue, and reasoning through realistic scenarios drawn from the candidate's own domain (for example, a technical migration, a process breakdown, a customer escalation, a resourcing conflict, or a policy decision). Candidates should define how they would validate a proposed solution (test cases, acceptance criteria, or success metrics), describe how they would monitor or verify the outcome after implementation, and identify opportunities for improvement, risk mitigation, or automation where applicable. Clear communication of the recommended approach, the expected outcomes, and the rationale behind trade-offs made is essential.

HardTechnical

33 practiced

A content-moderation ML pipeline is subject to adversarial evasion attempts and label poisoning. Propose detection and mitigation strategies across data ingestion, training, and inference stages to increase model robustness and detect attacks early.

Sample Answer

Start by clarifying threat models: adversarial evasion (at inference: small perturbations, paraphrases, stealthy images/videos) and label poisoning (during data collection: mislabeled or backdoor-triggered examples). Defenses should be layered across ingestion, training, and inference.

Data ingestion (prevent/detect):- Source vetting & provenance: require signed data pipelines, rate-limit new sources, track metadata (uploader, timestamp, collection method).- Automated anomaly detection: run feature-distribution checks (embedding-level distances, population stats), outlier detectors (Isolation Forest, LOF) and clustering to surface unusual batches.- Label auditing: sample-based human review prioritized by model uncertainty/novelty; use consensus labeling and inter-annotator agreement thresholds.- Backdoor trigger scouting: search for correlated rare tokens/patterns with specific labels (frequency conditioned on label); flag highly predictive rare n-grams or pixel patterns.

Training (robustification & poisoning mitigation):- Robust training: adversarial training (projected gradient methods for text via paraphrase or embedding perturbations; PGD for images), randomized smoothing for certified robustness where applicable.- Data sanitization: influence functions or TracIn to identify training points with outsized influence on validation loss; remove or relabel suspicious points.- Label-noise tolerant methods: co-teaching, noise-robust loss functions (symmetric cross-entropy), and semi-supervised learning to rely less on noisy labels.- Backdoor defenses: neuron activation clustering, fine-pruning (remove neurons associated with triggers), and training with auxiliary clean holdout to detect sudden accuracy spikes on specific patterns.- Validation: hold out a trusted validation set from vetted sources; monitor generalization gap and per-class performance shifts.

Inference (detection & runtime mitigation):- Input filtering: use anomaly detectors on incoming embeddings; block or escalate highly out-of-distribution inputs.- Adversarial example detection: apply randomized transformations (token dropout, synonym substitution, small image augmentations) and check prediction stability; use ensemble disagreement or prediction consistency as flags.- Confidence calibration & abstention: temperature scaling, conformal prediction, or abstain thresholds; route low-confidence or flagged inputs for human review.- Runtime monitoring & logging: track input distributions, per-user request patterns, and model prediction drift; alert on sudden shifts or concentrated misclassifications.

Operational best practices:- Red-team & continuous evaluation: simulate evasion and poisoning attacks (paraphrase attacks, trigger insertion) and run continuous red-teaming.- Canary and staged rollouts with shadow models and A/B monitoring to detect performance regressions early.- Explainability tools: SHAP/Integrated Gradients to identify features driving suspicious label changes.- Incident playbook: automated rollback, quarantining, root-cause analysis, and re-training with cleaned data.

Trade-offs:- Robust training and heavy vetting increase cost and latency; balance by prioritizing high-risk content flows.- Certified methods may be conservative; combine detection+mitigation to reduce false positives.

Example short flow:1) Ingest: metadata checks + embedding OOD detection; quarantine suspicious batch.2) Train: remove top-k influential suspicious points using TracIn; adversarially augment remaining data.3) Infer: run stochastic transformations; if prediction unstable or confidence low, send to human moderator.

This layered, measurable approach—detect early at ingestion, minimize poisoning during training, and defend/detect at inference—reduces attack surface while enabling operational response.

MediumSystem Design

44 practiced

Design monitoring and observability for a nightly ETL pipeline that ingests third-party CSVs, transforms them into normalized tables, and feeds a feature store. Specify which metrics, logs, traces, and data quality checks you would implement to ensure reliability and quick diagnosis of failures.

Sample Answer

Requirements:- Nightly ETL ingesting third‑party CSVs → normalized tables → feature store- Reliability, fast diagnosis, data quality, SLA (e.g., finish by 04:00), alerting

High-level architecture:- Ingest worker(s) → staging S3/bucket → validation & transform jobs (Spark/DBT) → normalized DW tables → feature engineering jobs → feature store (Feast or internal)- Orchestration: Airflow; logging/metrics collector: Prometheus + Grafana; traces: OpenTelemetry/Jaeger; logs: ELK or Cloud Logging

Key observability to implement

Metrics (time-series, for Grafana/Prometheus):- Job-level: run_count, success/failure_count, run_duration_seconds, retry_count- Data volume: files_received_count, rows_ingested, bytes_ingested- Schema/quality aggregates: pct_rows_failed_validation, null_rate_per_column, distinct_count_changes- Feature store: features_written_count, feature_write_latency, feature_backfill_progress- SLA: pipeline_completion_latency (time vs scheduled end)

Logs:- Structured logs with job_id, file_id, step, timestamp, level, error_code, sample_row- Ingest start/end, validation summary, transformation summary, DB write batches, feature write details- Store raw CSV parsing errors and stack traces

Traces:- Distributed traces across orchestration → ingest → transform → store spans showing timing per step and downstream DB calls; tag with file_id/job_id to correlate

Data quality checks (automated; fail/pass + detailed diagnostics):- Schema conformance: column names/types, required columns present- Row-level validation: missing required fields, type coercion errors, regex/constraint checks- Aggregate checks: row count vs expected (historical delta thresholds), null rates, cardinality changes, value distribution drift (KS test) and mean/std deviation- Referential integrity: foreign keys and lookup completeness- Freshness: timestamp ranges and max age of data- Feature-level checks: feature ranges, null% per feature, feature drift vs baseline- Oracles: sample checks comparing pre/post-transform aggregates

Alerting & incident flow:- Alerting rules for: job failures, >X% validation failures, SLA miss, major schema change, feature drift beyond threshold- Severity routing: pager for production-blocking; Slack for warnings- Include run_id, logs link, trace link, sample failing rows, last successful run

Dashboards & runbooks:- Dashboards: pipeline overview, per-step latency, data quality trends, feature store health- Drilldowns: file-level logs, trace waterfall, sample bad rows- Runbooks: steps to triage common failures (ingest parsing, schema mismatch, DB constraint)

Example: if nightly job fails with 40% validation errors, alert includes file_id, validation summary (columns failing, sample rows), link to trace showing transform span stuck, enabling quick fix (reject CSV, inform provider, re-run).

This combination ensures detection (metrics), diagnosis (logs+traces), prevention (quality checks), and rapid recovery (alerts + runbooks).

MediumTechnical

35 practiced

Design logging and tracing instrumentation for diagnosing latency spikes when fetching features from a feature store. Specify which events to log, correlation IDs, sampling rates, and how to aggregate metrics for SLOs and alerting.

Sample Answer

Situation: We need instrumentation to quickly diagnose latency spikes when models fetch features from the feature store.

Design summary (events, IDs, sampling, aggregation):

Events to log (structured JSON):- Request-start: request_id, model_id, environment (prod/stage), caller (online/batch), timestamp, requested_features (list), feature_store_version, trace_id- Feature-store-call: span_id, parent_span_id, request_id, feature_set_id, endpoint, method, payload_size, cache_hit (bool), start_ts, end_ts, latency_ms, response_status, backend_node- Cache-miss-load: start_ts, end_ts, latency_ms, items_loaded- Request-end: request_id, status (ok/error/partial), total_latency_ms, features_returned_count, result_size_bytes

Correlation IDs and propagation:- request_id (unique per model inference or batch job)- trace_id (for distributed tracing across microservices)- span_id and parent_span_id for each network/DB call- model_id and feature_request_id as tags for aggregationPropagate request_id and trace_id through HTTP headers (e.g., X-Request-ID, traceparent).

Sampling strategy:- Logs: write all Request-start and Request-end for production (low cardinality).- Traces: sample 10% of successful requests, 100% of error or p95+ latency requests (use tail-based sampling if supported).- Detailed spans (deep diagnostics): 1% of overall, but always for requests flagged by alerts.- Metrics: emit aggregated metrics every 1s/10s (no sampling).

Aggregations & SLOs (metrics to compute):- Latency histograms: p50/p90/p95/p99 of total_latency_ms and feature-store-call latency.- Cache hit ratio: cache_hits / total_calls.- Error rate: errors / total_requests.- Throughput: requests/sec, features/sec.- Freshness: age of features served (if applicable).

SLO suggestions & alerting:- Example SLOs: 99% requests p95 latency < 100ms; cache hit ratio > 90%; error rate < 0.1%.- Alerts: - Latency alert: if p95 > SLO for 5m -> page on-call. - Error spike: error rate > threshold sustained 3m. - Cache regression: cache hit drop > 10% in 10m.- Include runbook links in alerts and attach recent sampled traces and top N slow requests/logs.

Example structured log (JSON):

json

{
  "event":"feature_store_call",
  "trace_id":"abcd-1234",
  "request_id":"req-5678",
  "model_id":"fraud-v2",
  "feature_set_id":"user_activity_v3",
  "cache_hit": false,
  "latency_ms": 78,
  "status": 200,
  "ts":"2025-11-20T12:34:56Z"
}

Reasoning: structured logs + distributed traces let you correlate model requests to specific feature-store calls and cache behavior. Sampling balances cost vs. root-cause ability (full traces for errors/tail). Aggregated histograms and rates power SLOs and alerting; attaching sampled traces to alerts speeds diagnosis.

MediumTechnical

34 practiced

Design acceptance criteria and a testing suite for promoting a new fraud-detection model into production. Include dataset checks, metric thresholds, stress tests, guardrail metrics to avoid revenue loss, and how to run canary evaluations.

Sample Answer

Acceptance criteria (high level)- New model must improve primary fraud metric (e.g., AUC or precision@k) vs. production by a statistically significant margin and meet guardrail thresholds (no regressions on revenue signals).- Must pass data-quality checks, stress tests, fairness and robustness tests, and canary rollout with monitoring.

Dataset checks (automated pre-deploy)- Schema validation: feature names, types, missingness, ranges.- Distributional checks: population stability index (PSI) < 0.1 for key features; mean/variance shifts flagged if >3σ.- Label quality: label leakage, label delay window validation, and minimum label coverage (>=95% of expected labeled events).- Feature importance sanity: top N important features present and non-null.

Metric thresholds and evaluation- Primary model metric: e.g., increase in precision@top1% >= +5% absolute or relative improvement; AUC uplift >= +0.02 and p-value <0.05 (bootstrap).- Secondary metrics: recall at fixed false positive rate, F1, calibration (Brier score) no worse than baseline ±5%.- Business guardrails: maximum allowed increase in false positives leading to manual review cost: FP_rate increase ≤ 10% relative; chargeback/revenue-loss proxy metric ≤ baseline + $X/day (set from finance).- Fairness: no subgroup AUC drop > 3% for protected cohorts.

Stress tests- Adversarial input: evaluate under injected feature noise (10–30% missing, ±20% value shifts) and adversarial feature swaps to ensure stability.- Temporal drift simulation: evaluate on last N weeks/months to ensure robustness; dropout tests removing top features to measure degradation gracefully.- Latency/cost: inference latency under 99th percentile SLA; memory and CPU footprint within infra limits.

Guardrail metrics to avoid revenue loss- Precision at operational threshold must not decrease; expected number of false declines (legitimate transactions blocked) ≤ historical baseline + tolerance.- Chargeback rate, conversion/dropoff rate, and manual review volume monitored with alert thresholds.- Rollback triggers: sustained increase in revenue-impacting metrics beyond tolerance for T minutes/hours.

Canary evaluation plan- Split traffic: route small % (e.g., 1%) of real traffic to model for initial canary; evaluate over pre-defined window (e.g., 24–72 hours, min N events).- Metrics to track in canary: primary model metrics, guardrail/business metrics, latency, resource usage.- Statistical test: run A/B test or interleaved evaluation; require minimum sample size for power to detect the pre-specified effect.- Automated decision logic: if primary metric improvement AND no guardrail breaches -> increase traffic to 5%, 20%, then 100% with checks at each step. If any guardrail exceeds thresholds, auto-rollback to baseline and trigger incident.- Observability: dashboards (real-time), alerting (PagerDuty/Slack), and logging for per-transaction decisions for audit.

Operational runbook- Pre-deploy checklist (data freshness, model artifact checksum, feature-transform tests).- Post-deploy monitoring cadence: minute-level for latency, hourly for guardrails, daily for statistical significance.- Communication: notify stakeholders at each traffic step; run post-mortem if rollback occurs.

This suite balances statistical improvement with strict business guardrails and progressive canary rollout to minimize revenue risk while ensuring model robustness.

HardTechnical

49 practiced

Design a monitoring strategy to detect concept drift for a deployed model before business metrics degrade significantly. Include what statistics to track (feature distributions, label delay, calibration), thresholds, alert logic, and automated remediation options (retraining, rollbacks, human review).

Sample Answer

Requirements & goals: detect drift early enough to avoid business impact, minimize false positives, handle label delay, and automate safe remediation.

Metrics to track (real-time + daily):- Feature distribution drift: per-feature Population Stability Index (PSI) and two-sample Kolmogorov-Smirnov (KS) test on sliding windows (baseline = training or last stable month). Track mean, std, and quantiles (P1,P5,P25,median,P75,P95,P99). - Thresholds: PSI > 0.2 (moderate), >0.3 (high); KS p-value < 0.01 => flag.- Prediction distribution & score drift: compare predicted probability distribution (histogram) using PSI and Earth Mover’s Distance. Sudden shift in mean prediction > 0.05 absolute for calibrated probs.- Calibration: Expected Calibration Error (ECE) and Brier score on labeled windows. ECE increase > 0.02 or Brier increase > 5% => warn.- Label/target drift and performance (when labels arrive): ROC AUC, precision@k, recall, and confusion-matrix rates over time windows (7/30/90 days). Degradation >5% relative to baseline or falling below SLA triggers.- Label delay monitoring: monitor time-to-label distribution; if median delay increases >2x or tail (95th pct) exceeds allowed SLA, mark metric-latency risk and weight drift alerts accordingly.

Alerting & logic (multi-tier):- Tier 1 (info): single metric crosses warning threshold (e.g., PSI 0.15–0.2) -> notify data science slack, no action.- Tier 2 (action): multiple correlated signals (e.g., >3 features PSI>0.2 OR PSI>0.3 for 1 feature + prediction shift) OR performance drop >3% once labels available -> create incident, run automated checks (data integrity, schema).- Tier 3 (critical): business metric degradation confirmed (or performance drop >7%) -> trigger rollback/canary and paging.

Automated remediation pipeline:1. Prechecks: run data schema & sanity tests; if fail -> block new predictions, route to fallback model or rule-based system.2. Canary/Shadow: route small % (e.g., 5-10%) to new retrained model in shadow to compare.3. Automated retrain: if Tier2 persists for 48–72h and labels adequate (min N per class), trigger retrain pipeline using latest labeled window + holdout; run validation, fairness and backtest. If new model passes gates, promote to canary serving 5-10% traffic for 24–72h; auto-promote if metrics stable.4. Rollback: if new model fails or Tier3 occurs, rollback to last validated model automatically.5. Human-in-loop: all promotions and rollbacks send notifications to owners; Tier3 requires approval within 1 hour for irreversible actions.

Best practices:- Use ensemble of tests to avoid noise; smooth signals with EWMA; use seasonality-aware baselines; keep audit logs and dashboards.- Maintain minimum sample sizes for statistical tests; use bootstrapping for small-sample confidence.- Define SLA-aligned business thresholds and simulate alerts with historical data to set thresholds and reduce false positives.

Unlock Full Question Bank

Get access to hundreds of Problem Solving and Scenario Analysis interview questions and detailed answers.

Join thousands of developers preparing for their dream job.