Problem Solving and Structured Thinking Questions

Focuses on the general capacity to approach an unfamiliar or ambiguous problem in a disciplined way, independent of the underlying domain. Core skills include clarifying the actual problem and its constraints before acting, decomposing it into smaller subproblems, recognizing patterns from prior experience, choosing among competing approaches, developing and testing a solution incrementally, weighing trade offs such as cost, risk, effort and correctness, reasoning about edge cases and failure modes, and communicating the thought process clearly to others. In technical roles this often shows up as algorithmic reasoning (selecting data structures, estimating time and space complexity) and systematic debugging. In non-technical roles it shows up as issue-tree style decomposition, hypothesis-driven analysis, and structured decision frameworks under ambiguity. The topic is about the reasoning process itself, not any single domain's toolkit.

HardTechnical

108 practiced

Propose an incident response and root-cause analysis (RCA) framework tailored to ML services that depend on both model-quality signals and infrastructure. Include runbooks for common incidents (data pipeline outage, serving skew, model performance regression), required observability (feature drift, label delay, tail latency), cross-team communication steps, and how to extract actionable RCA findings that feed back into CI/CD and training pipelines.

Sample Answer

Requirements & scope:- Detect, triage, mitigate, and prevent incidents that arise from both infrastructure and model-quality issues across data pipelines, feature stores, training and serving. SLAs: detect <5m for P0, initial mitigation <30m, RCA <72h.

Framework (phases):1. Detection — automated alerts from model-quality and infra signals.2. Triage — map alert to domain (data, model, infra, config).3. Mitigation — runbook-driven immediate actions to restore safe operation.4. Deep RCA — correlate signals, reconstruct timeline, reproduce in sandbox.5. Remediation & feedback — code/pipe changes, tests, and CI gating.6. Postmortem & metrics — publish findings, owners, and follow-ups.

Required observability (minimum):- Model-quality: prediction distribution, calibration, accuracy/AUC over time, PSI/feature drift, label-delay-aware windows.- Data: feature arrival rates, schema checksums, null/entropy stats, training vs live distribution comparisons.- Serving infra: tail latency (p95/p99), error rates, CPU/memory, request queuing, autoscale events.- Business signals: downstream KPIs, user-facing errors.- Tracing: request id propagation from ingestion → feature store → model → response.- Logging: example inputs that triggered anomalous predictions (privacy-safe).

Runbooks (concise):- Data pipeline outage: - Detect: missing feature heartbeat, backlog spikes. - Immediate: fail open/serve cached features; flip to last-known-good snapshot; notify data infra. - Actions: restart ETL, check schema mismatch, replay backlog to staging, thermally throttle consumers. - Post: add schema/version checks, synthetic-data smoke test in CI.

- Serving skew (train vs serve): - Detect: PSI > threshold OR feature value clipping; sudden change in input distribution. - Immediate: switch to shadow/rollback model or apply input validation that rejects out-of-distribution inputs. - Actions: capture paired train/serve samples, run offline evaluation, determine if feature engineering drifted. - Post: add model input validation, gating test in CD that compares PSI to training baseline.

- Model performance regression: - Detect: drop in accuracy/KPI, rising false positives. - Immediate: roll back to previous model version or reduce business impact via rules. - Actions: isolate whether root is label drift, feature drift, concept drift, or bug. Re-run training with recent data; run A/B tests. - Post: add continuous evaluation pipeline with sliding-window retrain triggers.

Cross-team communication:- Pager rotation + Slack/incident channel with runbook link.- Triage meeting within 30 minutes (PM, data eng, SRE, ML owner).- Clear RACI: ML owner (model), Data eng (ingestion), SRE (infra), PM (business impact).- Daily updates during incident, written summary within 24h, full RCA within 72h.

Actionable RCA extraction & CI/CD feedback:- Produce artifacted findings: root cause files (tagged), dataset snapshot, model version, failing test cases, and repro notebook.- Gate CI: add unit/integration tests that replicate the failure (e.g., synthetic drift test, schema mismatch case).- Training pipeline: auto-ingest flagged examples into a "challenge" set for periodic retraining; add monitoring-driven retrain trigger (thresholds on PSI/label delay-adjusted metrics).- Release pipeline: require canary + shadow evaluation for 24–72h for major models, with automated rollback if KPI regressions exceed thresholds.

Metrics to measure success:- Mean time to detect/mitigate/RCA, recurrence rate, fraction of RCAs that produced CI tests, and improvement in downstream KPI variance.

This framework ensures incidents are detected holistically, mitigated fast, and converted to concrete engineering fixes that harden CI/CD and training pipelines.

EasyTechnical

70 practiced

You need to test whether a new model increases click-through rate (CTR). Design the A/B experiment: define the primary metric and guardrail metrics, choose sample size and duration (high-level calculation), explain randomization and bucketing strategy, address cold-start and novelty effects, and outline how to detect metric leakage or instrumentation bugs.

Sample Answer

Primary metric:- CTR (clicks / impressions) measured at user level (unique users) over the analysis window. Use ratio of aggregated clicks to aggregated impressions per user to avoid highly skewed sessions.

Guardrail metrics:- Engagement depth (time on page / session), conversion rate (if downstream), revenue per user, error rate, and latency. Also monitor impressions served and pageviews to detect traffic shifts.

Sample size & duration (high-level):- Compute minimum detectable effect (MDE) — e.g., 5% relative uplift. Use baseline CTR p0, desired power 80%, alpha 0.05. For proportions: n_per_group ≈ (Z_{1-α/2}√(2p̄(1−p̄)) + Z_{power}√(p0(1−p0)+p1(1−p1)))^2 / (p1−p0)^2- Translate to users: N = n_per_group. Choose duration = ceil(N / daily active users per group), ensure at least one full business-cycle week to cover weekday effects.

Randomization & bucketing:- Randomize at consistent user-id hashed into buckets (e.g., mod 10000). Sticky assignment using hashing + salt to ensure deterministic exposure across sessions and deployments. Use stratified randomization if CTR varies by cohort (region/device) to balance.

Cold-start & novelty:- Exclude new users for first X days or analyze separately; report treatment effect by tenure cohort. Run experiment for at least two weeks to let novelty decay; perform time-segmented analysis (day 1–3 vs day 4+).

Detect metric leakage / instrumentation bugs:- Sanity checks: equal traffic, equal baseline metrics pre-rollout, invariant metrics (impressions, DAU) should be balanced. Run A/A test prior to launch. Monitor sudden jumps, large variance, and per-segment anomalies. Implement logging of raw events, compare click counts in event pipeline vs aggregated metric, and run backfill reconciliation. If leakage suspected, pause experiment and run forensic checks on event deduplication, sampling, and monotonic counters.

Statistical checks:- Use pre-registered analysis plan, correct for multiple comparisons if testing many metrics, and report confidence intervals and uplift percent.

MediumTechnical

61 practiced

Your model's validation accuracy is much higher than test accuracy. Provide a structured debugging checklist to diagnose root causes across data, model, and evaluation: include checks for data leakage, different preprocessing between splits, distribution shift, improper cross-validation, data augmentation leaks, and label inconsistencies. For each suspected cause, explain how you'd confirm and remediate it.

Sample Answer

Start with a checklist grouped by Data, Model/Training, and Evaluation. For each item: how to confirm, and remediation.

Data- Data leakage (train → val): Confirm by searching for identical IDs, timestamps, or derived features present in both splits; compute overlap of unique keys and pairwise hash matches. Remediate by re-splitting by entity/time, removing derived leak features, and retrain with strict split rules.- Different preprocessing between splits: Confirm by logging/printing preprocessing outputs (scalers, tokenizers, image transforms) on sample records from each split; compare distributions. Remediate by centralizing preprocessing in one pipeline applied consistently (or save fitted transformers).- Label inconsistencies / noise: Confirm by sampling mismatches between label sources, computing label flip rates across time, and training a simple classifier to predict label from metadata. Remediate by cleaning labels, consolidating label rules, or using robust loss / label smoothing.- Distribution shift between val and test: Confirm with statistical tests (KS, Wasserstein), feature summary diffs, or training a domain classifier to distinguish splits. Remediate by collecting representative data, domain adaptation, importance weighting, or using more robust validation (time-based CV).

Model / Training- Overfitting to validation (hyperparameter leakage): Confirm via tracking experiments—if many models tuned on same val set, validation becomes optimistic. Remediate with nested CV, holdout test set never used for tuning, or use cross-validation.- Data augmentation leaks (same augmented sample in val/test): Confirm by comparing augmentation seeds/hashes and ensuring augmentations applied only to training. Remediate by applying augmentations only at training time and ensuring deterministic val/test pipelines.- Improper cross-validation (non-iid folds, leakage in fold creation): Confirm by reviewing fold-generation code for group/time leakage; compute inter-fold overlap. Remediate with group-aware or time-series CV.

Evaluation / Metrics- Metric mismatch (validation uses accuracy, test uses thresholded or different preprocessing): Confirm by re-evaluating test set with validation metric pipeline. Remediate by unifying evaluation code and thresholds.- Sampling bias in test set: Confirm by checking sampling logic and comparing class priors. Remediate by re-sampling or adjusting evaluation expectations (report per-slice metrics).

Operational checks- Random seed nondeterminism: Confirm by repeated runs; remediate by fixing seeds and documenting nondeterministic ops.- Experiment tracking gaps: Confirm missing metadata in runs; remediate by instrumenting logging (data versions, code hash, transformer artifacts).

Action plan summary1. Freeze a clean, auditable data split and preprocessing pipeline.2. Reproduce the gap with minimal model (logistic/regression) to isolate data vs model.3. Run statistical and unit tests above, fix root cause, then retrain with proper nested CV and a held-out test never used in tuning.

HardTechnical

72 practiced

Describe differences between post-training static quantization, post-training dynamic quantization, and quantization-aware training (QAT). Propose a step-by-step plan to quantize a large transformer model for mobile inference targeting less than 1% accuracy drop: include calibration dataset selection, retraining budget for QAT, validation strategy, fallback mechanisms, and rolling deployment precautions.

Sample Answer

Post-training dynamic quantization (PTQ-dynamic): weights are quantized (usually to int8) ahead of inference; activations are quantized on-the-fly per-tensor or per-channel using runtime range estimation. Fast, no calibration or retraining, good for LSTM/linear layers, moderate accuracy impact.

Post-training static quantization (PTQ-static): both weights and activations are quantized. Requires a calibration step over representative data to estimate activation ranges (min/max, or KL/percentile). Better runtime performance than dynamic and lower accuracy loss if calibration data is representative; still no retraining.

Quantization-aware training (QAT): simulates quantization effects (rounding, scale) during training so the model learns to be robust to reduced precision. Typically yields the smallest accuracy drop, at the cost of extra training and engineering complexity.

Step-by-step plan to quantize a large transformer for mobile with <1% accuracy drop

1) Baseline & constraints- Measure FP32 baseline on held-out test metrics and latency/size targets (e.g., target int8 latency/memory).- Select target backend (e.g., TFLite, ONNX Runtime Mobile, PyTorch Mobile) and supported quant schemes (per-channel weight quant, symmetric/asymmetric activations).

2) Quick wins: PTQ-dynamic- Apply dynamic quant to linear/embedding layers and evaluate. If drop ≤1% and meets latency, ship. If not, proceed.

3) PTQ-static with calibration- Build calibration dataset: 500–5,000 representative samples (for NLP: ~5k sentences covering different lengths, domains; for vision: 1k–2k images). Ensure distribution matches production traffic.- Use per-channel weight + per-tensor activation quantization; choose calibration method (min-max trimmed 99.999 percentile or KL for distributions with outliers).- Validate on dev set; if drop ≤1% accept. If not, continue.

4) QAT (if PTQ fails)- Retraining budget: fine-tune for 1–3 epochs over full training data or 5–20k steps with learning rate 0.1–1% of initial LR; use mixed-precision training and freeze lower layers initially if needed. Smaller schedules often suffice—monitor validation closely.- Training recipe: insert fake-quant ops, use SGD/AdamW with weight decay, clip gradients, possibly use knowledge distillation from FP32 teacher to preserve accuracy.- Early stopping when validation metric recovers within target.

5) Validation strategy- Multi-tier validation: calibration/dev set for tuning; held-out test for final accuracy; latencies measured on representative mobile devices/profiles (CPU, memory, thermal). Run A/B style tests on offline subsets; include robustness tests (long tail, tokenization edge cases).- Evaluate numerical fidelity (per-layer activation distributions), and run unit tests for functional equivalence on key inputs.

6) Fallback & safety nets- Keep FP32 and mixed-precision builds available. Design runtime to select model variant per-device capability.- If quantized inference exceeds acceptable error for specific inputs (e.g., low-confidence predictions), fallback to FP32 RPC or cloud call.- Instrument telemetry: accuracy drift, latency, memory pressure, error rates; sample inputs for automatic repro and rollback triggers.

7) Rolling deployment precautions- Canary rollout: 1% devices for 48–72 hours, monitor business KPIs and system metrics.- Gradual ramp: 1% → 5% → 25% → 100% with automated rollback if defined thresholds (e.g., >0.5% metric regression or latency increase).- Data capture: collect anonymized inputs/outputs (respecting privacy) for post-deploy validation; maintain reproducible evaluation pipeline to reproduce in staging.- Communication: notify stakeholders and on-call engineers; schedule deployment windows and automated rollback scripts.

Trade-offs and notes- Per-channel weight quantization + QAT yields best accuracy. QAT cost is higher but often necessary for <1% drop on large transformers.- Calibration quality matters more than size past a point; diversity is key.- Use distillation to accelerate QAT convergence and reduce accuracy loss.

This plan balances fast wins (dynamic/static PTQ) with QAT when needed, includes concrete calibration sizes, retraining budget, validation, fallbacks and safe rollout.

EasyTechnical

60 practiced

Explain Big-O notation and how you apply it when reasoning about time and space complexity in ML preprocessing and model code. Give concrete examples comparing O(n), O(n log n), and O(n^2) by referencing tokenizing a corpus, sorting feature values, and pairwise similarity computations. Describe how dataset size and hardware constraints influence algorithm choices.

Unlock Full Question Bank

Get access to hundreds of Problem Solving and Structured Thinking interview questions and detailed answers.

Join thousands of developers preparing for their dream job.