Structured Problem Solving and Decomposition Questions

Unit tests (focus: isolated functions, deterministic outputs)

1. Timestamp normalization

• Purpose: ensure timezone-aware and naive timestamps are converted to UTC at consistent format (ISO8601) and preserved ordering.
• Inputs: ["2023-03-10T15:00:00-05:00", "2023-03-10 21:00:00+01:00", "2023-03-11T02:00:00Z"]
• Assertions: all converted to UTC datetimes, formatted ISO strings, and difference between original instants preserved.
• Edge cases: DST transitions, naive strings (assume default tz), malformed strings -> raise/handle.

2. Numerical imputation

• Purpose: imputer fills missing values using configured strategy (mean/median/constant) and does not alter non-missing values.
• Inputs: [1.0, None, 3.0, None], strategy="median"
• Assertions: missing replaced with median=2.0; dtype preserved; no change to 1.0 and 3.0.
• Edge cases: all-missing column, extreme outliers, large cardinality.

3. Categorical encoding

• Purpose: encoder maps categories to integers or one-hot, handles unseen categories consistently (e.g., map to -1 or all-zero).
• Inputs: ["red","blue",None,"green"], fit on ["red","green","blue"]
• Assertions: mapping stable, None -> special token, unseen -> -1.
• Edge cases: high-cardinality, mixed types, nulls.

4. Schema and type checks

• Purpose: functions validate input columns, required dtypes, and raise informative errors.

Integration tests (focus: combined pipeline behavior, upstream/downstream effects)

1. Full pipeline on small dataframe: normalization -> impute -> encode; assert column order, no nulls, deterministic outputs given seed/config.
2. Idempotence: running pipeline twice should not change already-processed data.
3. Performance/regression: process N rows within expected time and memory budget.

Two concrete test examples (input -> expected output)

Test A — Timestamp normalization unit test
Input (single column "ts"):
["2023-11-05T01:30:00-04:00", "2023-11-05T01:30:00-05:00"]
Config: convert to UTC, output ISO format
Expected:
["2023-11-05T05:30:00Z", "2023-11-05T06:30:00Z"]
(also assert ordering preserved)

Test B — Integration: pipeline end-to-end
Input dataframe:
ts: ["2023-03-10T15:00:00-05:00", None]
num: [10.0, None]
cat: ["A", "B"]
Config:

• normalize ts -> UTC ISO
• impute num with mean
• encode cat with mapping {"A":0,"B":1}, unseen->-1
  Expected dataframe:
  ts: ["2023-03-10T20:00:00Z", None] (or if impute policy for ts is to leave None)
  num: [10.0, 10.0] (mean=10.0)
  cat: [0,1]

Notes on assertions and testing practices

• Use fixture data, deterministic seeds, and small pandas DataFrames.
• Mock external dependencies (timezone DB, remote config).
• Add property-based tests for invariants (no NaNs after imputation, UTC tzinfo).
• Validate metrics: column counts unchanged, data types stable, and meaningful error messages for schema violations.

Situation: At my last company we needed an ML-driven fraud-detection pipeline for a payments product. The dataset was large but noisy, stakeholders wanted production impact within 6 months, and engineering capacity was limited.

Task: As the ML engineer owner, I had to break the project into deliverable milestones that delivered value early, reduced risk, and enabled iterative improvements.

Action (decomposition & prioritization):

• I mapped goals to business metrics (false positive rate, detection latency, ROI) and split work into four milestones prioritized by risk and value:
  1. Data readiness (EDA, schema checks, labeling plan) — de-risk data quality.
  2. Offline baseline model (lightweight XGBoost) + evaluation against historical labels — quick signal of feasibility.
  3. Real-time feature pipeline + model serving in staging — validate latency and infra.
  4. A/B experiment and rollout with monitoring and alerting — measure production impact.
• Prioritization used a RICE-like score (Reach, Impact, Confidence, Effort) so high-impact, low-effort items (baseline model, basic features) came first.
• I led cross-functional sprints: paired with engineers to build streaming features, worked with product to define acceptable FP/TP trade-offs, and set up CI for model retraining.

Result:

• Within 8 weeks we delivered a baseline model that reduced chargeback reviews by 18% offline. By month 4 the staging pipeline sustained sub-200ms inference; A/B test at month 6 showed a 12% reduction in fraudulent losses with a 2% increase in manual reviews (within SLA). Time-to-first-production-value was under 2 months, and full rollout met business targets.

Learned:

• Prioritize early, low-effort experiments to validate assumptions.
• Make success metrics explicit and tie milestones to them.
• Invest early in monitoring and retraining automation to avoid technical debt. I apply this milestone-driven, risk-first decomposition on all ML projects now.

Iteration 1 — MVP (10 categories, fast proof-of-concept)

• Dataset: ~500–1,000 images/category (5–10k total). Balanced, clear images, single object centered, diverse backgrounds minimal. One validated label per image.
• Baseline models (low compute): MobileNetV2 or ResNet18 pretrained; simple transfer learning (freeze backbone, train head) on a single GPU or cloud instance.
• Validation criteria to promote: >75% top-1 accuracy on holdout set + per-class recall >65% for at least 8/10 classes; confusion matrix shows no systematic class collapse.
• Prioritization: 70% model (fast prototyping via transfer learning), 30% data (clean labeling and balanced classes).

Iteration 2 — Robustness & scale prep

• Dataset: 2–5k images/category (20–50k total). Add varied lighting, occlusions, viewpoints, and real-world negatives. Introduce 3 annotator agreement for noisy cases.
• Baselines (moderate compute): Fine-tune full ResNet50 or EfficientNet-B0/B3; add simple augmentations (RandAugment), mixup; train on 1–2 GPUs.
• Validation criteria: >85% top-1, per-class recall >80% for 9/10 classes, stable performance under common perturbations (±5% on augmented test).
• Prioritization: 60% data (diversity + hard examples), 40% model (regularization, better backbone).

Iteration 3 — Production-ready & scale to 100 classes

• Dataset: 1–2k images/category for 100 classes (100–200k total) OR active learning to prioritize labeling; ensure long-tail handling and hierarchical labels.
• Baselines (higher compute): EfficientNet-L2 or Vision Transformer (ViT)/Swin small fine-tuned; distillation to smaller model; handle with multi-GPU / cloud TPU.
• Validation criteria: For 10→100, target similar per-class recall scaled (overall top-1 ≥80% for 100 classes), micro/macro-F1 thresholds, latency and memory meeting deployment SLAs.
• Prioritization: 50/50 data vs model: use active learning to collect most informative samples, class-rebalancing, then scale model complexity and distill for production.

Other notes:

• Use continuous evaluation: holdout, OOD, and business-metric tests. Apply class-weighting/oversampling for long-tail. Monitor per-class drift post-deploy and iterate with targeted data collection.

Approach: prioritize low-risk, high-impact changes first (no model rearchitecture) then progressively more invasive optimizations, measuring accuracy vs 95/99th tail latency and cost.

Step 1 — baseline

• Record current model accuracy (AUC/accuracy), p50/p95/p99 latency, CPU/GPU utilization, memory, and request patterns. Define success criteria (e.g., +1% accuracy w/o p99 >100ms).

Step 2 — feature count pruning (low risk)

• Run feature ablation / SHAP to remove low-importance features or fold sparse features. Experiment: remove 5–20% least-important features in steps, retrain lightweightly. Measure accuracy drop vs latency gain (faster input preprocessing, smaller input tensors).

Step 3 — caching (low-medium risk)

• Add result and feature caching for repeated requests or precompute embeddings. Measure hit rate and effective tail latency reduction. A/B test to ensure no staleness issues.

Step 4 — batching & request shaping (medium)

• Introduce micro-batching with adaptive timers (max latency budget 100ms). Simulate traffic to tune batch size vs p99. Quantify throughput increase and any accuracy change (none).

Step 5 — quantization (medium-high)

• Apply post-training quantization (INT8) and run calibration. Measure accuracy drop; if > acceptable, try quant-aware training. Measure p99 improvements on target hardware.

Step 6 — distillation (high)

• Train a smaller student model distilled from teacher to regain accuracy if quantization hurts. Compare student size, accuracy, and tail latency.

Step 7 — architecture changes (highest risk)

• Only if above insufficient: explore lighter architectures or pruning. Run canary deployment with monitoring and rollback.

Measurement & gating:

• For each step run controlled A/B tests and synthetic load tests to capture p50/p95/p99 latencies, throughput, and accuracy. Use Pareto plots (accuracy vs p99) to pick best trade-off.
• Stop when acceptance criteria met or when further gains require unacceptable accuracy loss.

Risk mitigation:

• Canary deployments, automated rollbacks, feature flags, and conservative steps first (feature pruning, caching, batching) minimize production risk.

A symptom is an observable effect (e.g., accuracy dropped from 92% to 70%). The root cause is the underlying reason that produced that effect (e.g., a bug that applied the wrong feature encoding at inference). Distinguishing them means moving from “what happened” to “why it happened” by instrumenting and isolating layers (data → features → model → serving).

Concrete example:

• Symptom: Production model accuracy fell from 92% to 70% after a release.
• Root cause: An upstream ETL change reordered categorical columns so the one‑hot encoder in the inference pipeline mapped categories to the wrong feature positions, producing garbage inputs to the model.

Short 3-step trace process:

1. Reproduce & isolate: Run the same inference code on a snapshot of production input and compare model outputs to a known-good baseline to confirm failure path.
2. Data & schema checks: Compare schemas, feature distributions, null rates, and column order/hashes between training data, staging, and prod; look for mismatches in encodings/scaling.
3. Inspect transformation lineage & fix + validate: Audit recent ETL/CI changes, fix the transformation (e.g., enforce explicit column names or save encoder metadata), deploy a patch, and run regression tests and canary rollout to confirm metrics recover.

This approach separates immediate observable problems from causal faults and prevents recurring issues by adding schema/encoding contracts and automated data checks.