Retrospectives and Continuous Improvement Questions

Situation: After completing a 6-month generative-AI feature (customer support summarization) we ran a formal lessons-learned to capture what went well and what to change before scaling.

Task: As the AI lead I organized and facilitated the retrospective to produce concrete process improvements across scope, data, models, infra, and ops.

Action:

• Participants: product manager, two ML engineers, two data engineers, an SRE, QA, and a customer-success rep — total 8 people to cover technical and user-impact perspectives.
• Categories examined:
  • Scope: feature creep and unclear success criteria that lengthened the timeline.
  • Data: labeling bottlenecks, class imbalance, missing schema validations.
  • Models: tuning iterations, lack of model versioning, overfitting on pilot data.
  • Infra: GPU provisioning delays and inconsistent reproducible environments.
  • Ops: deployment rollback pain points, no drift monitoring, and limited telemetry.
• Facilitated a 90-minute structured session using a timeline + root-cause for top issues, then prioritized by impact vs. effort.
• Top process change implemented: a CI/CD-driven ML pipeline that combined automated data validation (Great Expectations), a model registry (MLflow), and automated canary deployments with rollback rules. This required new checks before training and an enforced model-vetting checklist (data provenance, unit tests, performance gates).
• I led the rollout, created templates, and ran a training session for engineers.

Result / Measurement:

• Post-change we tracked three KPIs over the next two releases:
  • Production incidents due to data/model issues dropped 40% (from 5 to 3 incidents over two sprints to 3 to 1).
  • Mean time from idea to validated model decreased 30% (from ~10 days to ~7 days) due to reproducible runs and fewer manual checks.
  • Production-quality metric (ROUGE-F1 for summaries) improved 2.5 points on average because models were trained on validated, balanced data.
    Those measurable improvements confirmed the change reduced risk and sped iteration, and the pipeline became the standard for subsequent projects.