Situation: At my last company we were building an NLP pipeline to auto-generate product descriptions. Two approaches emerged under ambiguity: (A) fine-tune a large pretrained transformer (T5) with prompt-tuning for controlled length/format, (B) a smaller encoder-decoder with task-specific engineered features (product attributes + template constraints). Both had pros/cons on latency, cost, and output quality.
Task: My job was to decide which to productionize within 6 weeks, using data-driven experiments to balance quality, latency, and inference cost.
Action:
- Experiment design: I defined an A/B test over 4 weeks with stratified sampling across product categories (electronics, apparel, home). Each incoming product was randomly assigned to approach A or B; reviewers were blinded.
- Metrics & significance: Primary metric = human-rated relevance & fluency (1–5). Secondary = BLEU/ROUGE, distinctness, generation latency, and cost-per-inference. I pre-registered success: a ≥0.25 point lift in mean human score with p < 0.05 (t-test with bootstrap for non-normality).
- Implementation: Deployed both models behind a feature-flagged inference endpoint; logged inputs, outputs, latency, cost, and human ratings from a panel of 200 raters (each example rated by 3 people).
- Rollout plan: If A or B met criteria, roll to 10% traffic for monitoring (automated quality checks and user engagement signals) for 2 weeks, then ramp to 100% with rollback guardrails.
Result: After 4 weeks, A (fine-tuned T5) achieved +0.35 mean human score (p = 0.01), slightly higher diversity, but 2x inference cost and 30% higher latency. We deployed A to 10% while optimizing batching and quantization to cut cost/latency. After optimizations cost fell by 40% and latency met SLOs. The team prioritized investing in model-serving optimizations and feature work to support A; we deferred the engineered-template path to a lower-priority backlog.
Learning: Pre-registering metrics and significance helped avoid bias; including cost and latency as first-class metrics ensured we chose a practical solution, not just the highest-scoring model. The experiment also created measurable acceptance criteria that aligned engineering, product, and ML stakeholders.
