Giving and Receiving Feedback Questions

Situation: (opening) Hey [Name], do you have 10 minutes? I wanted to talk about the stakeholder updates you’ve been sending.

Task: (observation + empathy) I really appreciate you keeping everyone informed — your summaries make things easier to follow. I’ve noticed some of the language is a bit high-level, and I worry stakeholders might not have the detail they need to act.

Action: (concise, nonjudgmental feedback)

• When you say things like “the model is performing well,” it’s hard for readers to know what “well” means or whether any action is required.
• A clearer way is to include specific metrics, time windows, and next steps.

Concrete example:

• Vague: “The model is performing well and we’ll keep monitoring.”
• Suggested rewrite: “Model performance on the last validation run: accuracy 88.4% (±0.3), ROC AUC 0.92, 99th-percentile latency 120ms. No regression vs. baseline; data drift monitoring flagged a slight shift in Feature X (KL=0.08). Next step: retrain scheduled in two weeks if drift persists; I’ll update with retrain results or any anomalies.”

Result / Offer: Would you like me to review your next update or help create a short template with the key fields (metrics, comparison, risk, next steps)?

1. Quickstart + expected outcome
   "Add a short 'Quickstart' section at the top with a minimal sequence of copy-paste commands to run the container and a clear expected result. Example:

• docker build -t my-model:latest .
• docker run --rm -p 8080:8080 my-model:latest
  Then show a sample curl request and expected JSON response. Why: new engineers should be able to verify the image and endpoint in <5 minutes without reading the full doc."

2. Reproducible environment & exact versions
   "List exact versions for Python, OS base image, ML framework, and key libs; include the Dockerfile snippet or a requirements.txt hash. Example:

• Base image: python:3.10-slim-bullseye
• torch==2.1.0+cpu, torchvision==0.16.0
  Also add a checksum or git commit used for the model artifact and a command to fetch it. Why: prevents 'it works on my machine' issues and speeds troubleshooting."

3. Onboarding checklist + troubleshooting
   "Create a short 'Onboarding checklist' with steps to register credentials, local port mappings, environment variables, and how to run unit/integration tests (commands and expected pass rates). Add a Troubleshooting FAQ for common failures (port in use, model load error, missing env var) with log file locations and example error messages. Why: reduces ramp time and duplicate questions, letting new hires become productive faster."

Situation: On a recent recommendation-system project I joined, a senior engineer proposed deploying a large Transformer-based ranking model end-to-end in production. It promised slightly better offline NDCG but added 3× latency, increased memory, and no uncertainty calibration—risks for real-time traffic.

Task: As the ML engineer responsible for production readiness, I needed to raise these concerns, show evidence, and align the team on a safer, performant approach.

Action:

• Prepared: I ran targeted experiments comparing the proposed model vs a distilled Transformer and a tree-based baseline on latency, throughput, A/B simulated CTR, calibration (ECE), and cost. I collected logs showing tail-latency issues and a simple cost estimate for infra.
• Structured the conversation: booked a private 30‑minute slot, started by asking permission to share observations, framed feedback around user impact (latency, availability, cost), presented data (charts and short experiment notes), and proposed concrete alternatives: model distillation, async scoring for heavy features, and adding calibration + canary rollout.
• During the meeting I listened to his constraints, iterated on a hybrid plan, and offered to pair on a distillation experiment.

Result: He agreed to a paced plan: first deploy a distilled model with calibration and strict SLOs behind a canary; full model only after validating infra changes. The canary matched accuracy within 0.8% while cutting p95 latency by 60% and reducing infra cost. We also added a checklist for production readiness.

Learning: Candid feedback is most effective when grounded in reproducible data, focused on user impact, and paired with concrete alternatives and collaboration.

Situation: In a large ML org reviews are frequent but inconsistent—code/model bugs slip to prod, reviews are slow, and engineers avoid candid critique.

Framework (high-level goals): timely reviews (SLA ≤48h), high coverage (>90% critical artifacts), scalable participation, and strong psychological safety.

Processes:

• Triage + categorization: every artifact labeled (minor/major/critical) and routed to appropriate queue.
• SLAs: minor ≤48h, major ≤72h, critical/production-blocking ≤24h with on-call reviewer rota.
• Lightweight pre-review: short design note + model card for non-prod; full PR for prod changes limited to <400 LOC or modularized.
• Rotating reviewer pools: cross-team reviewers (2 reviewers for major, 1 for minor) with monthly rotations to spread expertise.
• Mentorship loop: paired reviews for juniors; “challenge + propose” format to keep feedback constructive.
• Blameless templates: use facts, impact, and suggested fixes; require at least one empathy line on higher-stakes reviews.
• Postmortems and knowledge-sharing: review heatmaps and recurring issues discussed in monthly learning sessions.

Tooling:

• Code review: GitHub/GitLab with required templates, size checks, automated linters, and CI gating.
• Model review: MLflow/Weights & Biases for artifacts, model cards stored in repo, automated checks for data drift, explainability, fairness tests.
• Triage & metrics: lightweight dashboard (Datadog/Looker) showing queues, SLAs, review coverage.
• Async feedback: Slack/Teams integrations, and anonymous feedback option for sensitive matters.

KPIs:

• Review coverage: % of PRs/models passing review (target >95% for prod).
• Turnaround: median/95th percentile review time (median ≤24–48h, 95th ≤72h).
• Defect rate post-deploy: production incidents per 1k deploys (trend down).
• Reviewer load: avg reviews per engineer/week (balanced).
• Psychological safety proxy: pulse survey score on “I feel safe giving/receiving feedback” (target ≥4/5).

Addressing chronic non-participation:

• First, diagnose: review dashboards to identify teams/individuals, then 1:1 conversations to surface blockers (time, confidence, unclear expectations).
• Remediation steps:
  • Expectation reset: include participation in OKRs and performance goals (qualitative + quantitative).
  • Training: focused workshops on giving/receiving feedback and technical review skills.
  • Protected time: allocate 2–4 hours/week per engineer for review work.
  • Pairing & shadowing: require non-participants to pair-review with a mentor for 4 sprints.
  • Escalation: persistent non-participation after coaching triggers manager intervention, with reallocation of responsibilities if needed.
• Positive reinforcement: recognize top reviewers in reviews leaderboard, rotate “review champions” to share tips.

Why this works:

• Balances speed and safety by combining SLAs, automation, and human review.
• Scales via rotating pools, tooling, and measurable KPIs.
• Emphasizes culture through blameless templates, training, and manager accountability—so feedback stays constructive and sustainable.

Situation: Our ML team was shipping models quickly but code-review feedback often stalled, causing repeated quality issues (data leakage, missing tests, reproducibility gaps) and unclear ownership for fixes.

Process I set up:

• Tooling: PRs in GitHub/GitLab + Jira for tracking actionable items, CI (pytest, unit tests, model-data checks), Codecov, DVC for data/versioning, and Slack for notifications. A dashboard (Looker/Datadog/Confluence) surfaces open review items and metrics.
• Templates: A PR template with sections: Summary, Model changes, Dataset/Feature changes, Repro steps, Metrics, Tests added, Risks. A “Review Action” Jira template created from review comments with fields: Title, PR link, Type (bug/refactor/test/documentation), Owner, Priority, Due date, Acceptance criteria.
• Workflow & ownership: Reviewers must add either an “Approved” or a Jira-linked action item. If a comment requires work, the reviewer opens (or converts to) a Jira issue and assigns an owner—usually the PR author unless they delegate. Use GitHub-Jira integration so every PR shows linked tickets.
• SLAs: Initial review acknowledged within 24 hours on business days; substantive review completed within 48–72 hours. Action items must be assigned within 24 hours and have an estimated due date (typically within 1 sprint for non-blocking, 24–72 hours for critical).
• Measurement: Track KPIs—mean time to first review, mean time to close review action, percentage of PRs with unresolved comments at merge, recurrence rate of issue types (e.g., missing tests). Dashboard updates weekly.
• Retrospectives & continuous improvement: Monthly review-focused retro: analyze recurring action types, spike areas (data handling, tests, infra), and prioritize changes: update PR template/checklist (e.g., require a data schema diff), add CI gates (fail if model registry not updated), run focused training sessions or pair reviews, and assign a process owner to close the loop. After changes, measure impact on recurrence and MTTR; if regression persists, escalate to process improvement sprint.

Example outcome: After implementing this, our MTTR for review actions dropped from 7 to 2 days, test coverage for model code increased 25%, and recurring data-leakage incidents dropped by 80% within two months.

This approach balances clear tooling and templates, defined SLAs, measurable metrics, and a retro-driven feedback loop so action items are owned and improvements are sustained.