Professional Integrity and Ethics Questions

Requirements & goal: catch regressions and infrastructure failures early, prioritize high-signal alerts, and minimize paging noise.

Monitoring checklist

1. Metric categories (what to monitor)

• Data: input schema drift (field types, missing rate), feature distribution shifts (per-feature mean/std/KL), arrival rate, latency from source, duplicate rate.
• Model: prediction distribution (class balance, confidence/entropy), input->output ratio, calibration (Brier/ACE), degradation on production holdout (rolling golden dataset), increase in abstains/errors.
• Infra: latency P50/P95/P99, CPU/GPU/memory, pod/container restarts, queue/backlog size, throughput (requests/sec), model loading failures.
• Business: key KPIs (conversion, revenue per user, false positives cost), user-facing error rates.

2. Aggregation windows & rollups

• Real-time (30s–5m): infra health, request latency spikes, queue/backlog.
• Short-term (5–60m): prediction rate, sudden data distribution changes, CPU/GPU saturation.
• Daily (1–24h): feature drift stats, calibration, business KPIs, model performance vs baseline.
• Use multi-window checks: e.g., detect both abrupt spike (5m) and sustained drift (24h).

3. Threshold strategies

• Static thresholds for infra (CPU > 90%, P99 latency > X ms).
• Adaptive baselines: rolling-window mean ± n*sigma for feature stats and confidence.
• Anomaly detection models (seasonal-ARIMA, EWMA, or simple percentile-based) for nonstationary metrics.
• Composite alerts: require multiple signals before paging (e.g., feature drift + drop in golden-set accuracy).
• Rate-limited and suppress noisy alerts with cooldowns and deduplication.

4. Alert prioritization & routing

• P0 (page immediately): model-serving down, P99 latency impacting SLAs, data pipeline halted, producer schema change breaking parsing, major revenue drop.
• P1 (page on-call): sustained model quality regression (>threshold on golden set and business KPI), infra nearing capacity.
• P2 (ticket/Slack): early drift signals, minor latency increases, single-feature anomalies.
• Include contextual links: recent deploys, runbook, dashboards, logs, recent model config.

5. Escalation playbook (play-by-play)

• Triage (first responder within 15 min): confirm alert validity, check runbook checklist (deployments, infra, recent data commits), attach evidence.
• Mitigation steps:
  • If infra: scale replicas, rollback faulty deploy, restart pods, increase resources.
  • If data issue: reroute to backup feed, pause model writes, switch to fallback model or serve cached predictions.
  • If model quality drop: rollback to previous model, enable human-in-the-loop, serve conservative thresholded outputs.
• Communication: post to incident channel with impact, owner, ETA; update status every 15–30 min.
• Escalation: after 30–60 min unresolved, escalate to senior on-call; after 2 hours, involve engineering manager and product owner.
• Post-incident: run blameless postmortem within 72 hours, update alerts thresholds/runbooks, add automated tests (synthetic traffic, golden-set checks), and adjust suppression rules to reduce noise.

Best practices

• Store metric history, annotate deploys/changes, maintain runbooks per alert, continuously tune thresholds, and use canary/blue-green deployments plus shadow testing to reduce production surprises.

Remediation (Immediate, measurable fixes)

1. Restore reliable model scoring pipeline

• Owner: SRE (Alice) + ML Eng (Ravi)
• Deadline: 48 hours
• Action: Roll back to v1.4 serving image, validate 1000 sample requests end-to-end, deploy canary at 5% traffic.
• Success criteria: 99.5% request success rate and latency <250ms on canary for 24h; roll to 100% if met.

2. Fix missing input validation causing NaNs

• Owner: ML Eng (Ravi)
• Deadline: 72 hours
• Action: Add schema validation (using Great Expectations) at feature ingestion and model-serving adapters; reject or impute invalid rows.
• Success criteria: 0 ingestion errors in synthetic test suite; production error rate from invalid inputs <0.1% over 7 days.

3. Restore observability for model predictions

• Owner: Observability Eng (Maya)
• Deadline: 72 hours
• Action: Enable prediction-level logging (feature vector hash, model version, score) to central logging (ELK/Cloud Logging), add Grafana dashboard for model throughput, latency, error-rate.
• Success criteria: Dashboards show live metrics; alerts firing tested end-to-end.

Preventive Actions (Process + product changes)

1. Implement automated data drift & performance alerts

• Owner: ML Engineer (Ravi) + Data Scientist (Liu)
• Deadline: 30 days
• Action: Add population/stability drift tests (JS divergence, feature distribution), label-skew and calibration monitors; integrate with PagerDuty.
• Success criteria: Alerts trigger on synthetic drift scenarios; runbook created and validated.

2. CI/CD for model and infra changes with gating

• Owner: ML Platform Lead (Sonia)
• Deadline: 45 days
• Action: Require model performance tests (unit, integration, offline metrics vs baseline) and canary rollout policy in pipeline; block deploys failing checks.
• Success criteria: 100% of deploys pass automated gates for 30 days; rollback time <10 minutes for failed canaries.

3. Runbooks & postmortem practice improvements

• Owner: Engineering Manager (Carlos)
• Deadline: 14 days
• Action: Create/assign runbooks for common failure modes (data quality, serving failures); conduct a tabletop drill with SRE/ML/DS every quarter.
• Success criteria: Runbooks published in handbook; drill completed with documented findings.

4. Access & change control for model artifacts

• Owner: Security/Platform (Aisha)
• Deadline: 30 days
• Action: Enforce signed model artifacts in registry, require PR approval from ML owner for model version promotions.
• Success criteria: All promoted models in registry have signatures and approval traces.

Each item must be tracked in the project board with owner, start date, and verification task; close only after success criteria are met and verified by cross-functional reviewer.

Situation: Model accuracy dropped 15% right after a data-pipeline change. I run a structured RCA in parallel tracks (data, features, labels, code/config).

1. Quick triage (minutes)

• Confirm the drop: check monitoring dashboard, timestamps, affected cohorts.
• Rollback window: if risk high, consider hotfix/rollback while investigating.

2. Artifacts I examine

• Pipeline change diff / PR description and deploy time
• Raw input samples (pre- and post-change) — fetch a few hundred rows
• Aggregate stats: feature distributions, counts, missingness, cardinality
• Schema/typing and serialization (protobuf/Avro) changes
• Feature store snapshots and cached features
• Label distribution and recent ground-truth vs predicted
• Inference logs and model input hashes

3. Hypotheses (priority order)

• Data-shift (schema, ordering, units, timezone)
• Missing/incorrect features (nulls, default values, encoding)
• Labeling issue (new label source or lag)
• Code bug in preprocessing or feature serialization
• Model regression (unlikely if code unchanged)

4. Tests and tools I run

• Data diff: run pandas / Spark profiling comparing histograms, summary stats, PSI (Population Stability Index) and KL divergence
• Schema validator: run schema-inference (e.g., Great Expectations) against new data
• Unit test the feature pipeline: run the same preprocessing locally on sample inputs and compare outputs (hash checks)
• Replay: run model offline on stored pre-change examples through the new pipeline and compare predictions
• A/B compare: route a small % to previous pipeline for live comparison
• Label sanity: sample recent labeled examples and manually inspect
• Logging: check feature-level null rates, NaN propagation, Inf, and categorical OOV counts

5. Example quick commands

• PSI example: compute PSI per feature and flag >0.1
• SQL: SELECT COUNT(*) FROM input WHERE feature IS NULL AND event_time > deploy_time;

6. Outcome & fix approach

• If preprocessing bug: patch and redeploy; add unit tests and schema checks
• If data drift: add alerting thresholds, retrain or adjust features
• If label issue: pause training/metrics until labels fixed

Preventive actions: enforce CI for data contracts (Great Expectations), add end-to-end integration tests that run model inference on a golden dataset during deploys, and improve monitoring with feature-level alerts and automated rollback.

Framework I use: triage → evaluate options under uncertainty → pick conservative measurable action → own outcome and close loop.

Triage (first 5–15 min)

• Quickly gather facts: user impact, error rates, latency, affected cohorts, recent deploys or data pipeline changes.
• Classify severity (safety/data-loss/availability/regulatory).
• Confirm we only have partial logs and estimate confidence intervals around key signals (e.g., 70% sure inference pipeline failing for 20% of traffic).

Evaluate rollback vs mitigation

• Rollback pros: restores known-good behavior, highest chance to stop incident; cons: loses new capabilities, may disrupt experiments, rollback itself can break downstream sync.
• Mitigation pros: preserves capabilities, can be targeted (circuit breaker, rate-limit, routing); cons: may be incomplete, prolongs risk window, harder to reason about for ML feedback loops.
• Decision criteria: user-safety first, then SLO breaches, then business impact, then observability/reversibility, cost/time-to-restore. For ML-specific risks include data poisoning, label skew, feedback loop amplification.

Decision under uncertainty

• If user safety or regulatory compliance is at risk, choose immediate rollback.
• If impact is bounded, rollback is high-cost and we can implement a rapid, reversible mitigation that reduces blast radius (e.g., enable a feature flag to send 90% of traffic to fallback model, enable deterministic rule-based fallback for edge cases, or reduce model confidence threshold to abstain).
• Prefer options that are incremental, observable, and reversible (canary, traffic-split, throttling).

Concrete actions I take

• Implement mitigation via feature flag / load balancer rule or model-serving config (example: route 80% to stable model, 20% to new model for diagnostics).
• Add fast telemetry: higher-resolution logs for affected paths, synthetic tests, and shadow traffic to replicate behavior offline.
• Communicate immediately to stakeholders: incident channel, decision rationale, expected rollback/mitigation timeline.
• If rollback chosen: follow automated rollback playbook, verify data pipeline consistency and model registry rollback to tagged artifact.

Accountability and follow-through

• Log decision, rationale, trade-offs, and confidence levels in the incident ticket.
• Own the verification: monitor SLOs and custom ML health metrics (model skew, input distribution, confidence histograms) until stable.
• Post-incident: lead blameless postmortem with root-cause analysis, reconstruct missing logs where possible, update runbooks (improve logging, add canary tests, automated rollback triggers), and publish learnings.
• If mitigation was chosen, ensure a clear rollback criterion and deadline; if unmet, escalate to full rollback.

This approach balances safety, business impact, and reversibility while making the decision traceable and ensuring continuous improvement of our ML deployment practices.

An effective postmortem for a production ML incident is concise, factual, and action-oriented. Key elements I include:

• Title & Summary: one-line summary and high-level impact (who/what/when/how long).
• Timeline: ordered timestamps (discovery → investigation → mitigation → resolution) with who did what; include alerts, commits, deployments, data pipeline runs.
• Impact: measurable effects (error rates, latency, revenue loss, user-facing SLO breaches, % of predictions affected) and business/customer impact.
• Root cause analysis: distinguish proximate cause vs. systemic causes; use 5 Whys or fishbone to identify technical and process failures (e.g., stale training data, schema change, model drift, deployment rollback).
• Contributing factors: list environment, monitoring gaps, ambiguous ownership, insufficient tests.
• Remediation (short-term): steps taken to restore service (rollbacks, hotfixes), with timestamps.
• Preventive measures (long-term): automated tests, data validation, schema contracts, model drift detection, canary/gradual rollout, improved alerting, runbooks.
• Owners & deadlines: assign clear owner for each action, acceptance criteria, and follow-up dates; track in a ticket system.
• Learnings & metrics: success criteria for prevention, and retrospective notes.

Blameless yet accountable approach:

• Language: describe systems and decisions, not people. Use “the system allowed X” not “person Y failed.”
• Root cause focuses on process and tooling failures; still assign clear owners and deadlines for fixes.
• Facilitate a safe retrospective: invite relevant stakeholders, encourage facts over opinions, and require accountable follow-through via tracked actions and a follow-up review to verify fixes.