MLOps: Monitoring, Retraining, and Lifecycle Management Questions

Direct answer

An observability dashboard for a production classifier needs at least eight metrics spanning performance, data, infrastructure, and business categories, each with its own alert rule, because a single "is it healthy" number can't distinguish a model problem from a data problem from an infra problem.

Structured elaboration

Worked example

For a loan-approval model specifically: an SLO might be "p95 inference latency under 200ms, 99.9% availability, and rolling 7-day approval-rate-by-protected-group parity within 5 percentage points of the training-time baseline": the last of these ties directly into fairness monitoring, since a loan model's business-KPI dashboard without a fairness cut is incomplete for this domain. Concrete tooling: Prometheus/Grafana for the infra and latency layer (they're built for time-series metrics and alerting), a dedicated ML-observability layer like Evidently for drift/data-quality checks (since Prometheus has no native concept of a feature distribution), and the business-KPI dashboard usually lives in whatever BI tool the business already uses, joined to the model-version tag so a KPI dip is attributable to a specific deployed version.

Trade-offs & pitfalls

The most common failure in a first-pass dashboard design is putting everything in one undifferentiated wall of charts. Grouping by category and giving EACH category its own alert (rather than one giant composite health score) is what lets an on-call engineer immediately localize a problem: an infra alert routes to SRE, a drift alert routes to the model owner, a business-KPI alert routes to product: collapsing them into one signal forces someone to manually figure out which team should even be paged.

Direct answer

A stateless microservice's correctness is a function of its code and the request; redeploy the same code and you get the same behavior forever. An ML model service's correctness also depends on the training data, and that dependency is invisible in the code itself, which is what makes it operationally different.

Structured elaboration

• Determinism: a stateless service given the same input and the same code version returns the same output, always. A model given the same input can still degrade over time even with zero code changes, because the world the model was trained on has moved on. "It didn't change" is not a defense against an incident.
• Dependency on training data: a microservice's dependencies are declared (a package.json, a Dockerfile). A model's most important dependency: the training dataset: usually isn't tracked with the same rigor, so "what changed?" during an incident often has no good answer unless you deliberately version it.
• Model versioning and rollback complexity: rolling back a microservice means redeploying a previous container image; behavior is fully restored. Rolling back a model can also require reverting the feature-computation code and confirming the previous model's dependencies (a feature store schema, an embedding table) haven't drifted out from under it in the meantime.
• Reproducibility: rebuilding a microservice from source is trivial. Reproducing a training run months later requires the same data snapshot, same library versions, same hardware-dependent numerics, and often the same random seed: any one of which being unpinned makes "rebuild it and compare" impossible.

Worked example

Concretely: a stateless service incident is almost always "a deploy introduced a bug" or "a dependency broke," and the fix is a code rollback that instantly and fully restores prior behavior. A model-service incident is more often "nothing was deployed, but the input distribution moved": the on-call engineer's first question isn't "what changed in the code" but "what changed in the world," and answering that requires drift monitoring, not a deploy log. This is why an SRE used to microservice on-call needs a different first-response instinct for ML: check feature distributions and recent retrains before assuming a code regression.

Trade-offs & pitfalls

The pitfall for an SRE new to ML on-call is importing the microservice mental model wholesale: treating a model-quality alert like a service-health alert (restart it, roll back the last deploy, check for a bad dependency bump) when the actual cause is upstream data. The fix isn't to abandon SRE discipline, it's to extend it: SLOs, error budgets, and blameless postmortems all still apply, but the telemetry that matters (feature distributions, label lag, model-version-tagged metrics) is different from what a typical service dashboard shows, and the on-call runbook needs to name that difference explicitly rather than assume it's obvious.

Direct answer

An alerting strategy that balances sensitivity against fatigue needs severity tiers so not everything pages the same way, aggregation windows so a single noisy blip doesn't fire, and thresholds derived from your own historical data rather than guessed round numbers.

Structured elaboration

1. Severity tiers: informational (logged, visible on a dashboard, no page) for small drifts within normal variance; warning (Slack/ticket, next-business-day triage) for a moderate, sustained signal; critical (immediate page) reserved for signals strongly correlated with real business impact historically.
2. Aggregation windows: require a signal to persist across multiple consecutive evaluation windows (not just one) before escalating: this alone eliminates most single-window statistical noise without losing real, sustained drift.
3. Threshold selection from historical data, not guessing: pull a year of historical PSI (or whichever metric) values for a feature during periods you KNOW were healthy, and set your warning/critical thresholds using something like a control chart (mean plus 2 or 3 standard deviations of that healthy period) rather than an arbitrary round number like "0.25 for everyone": different features have genuinely different natural variance, and a single global threshold either under-alerts on a naturally-stable feature or over-alerts on a naturally-noisy one.
4. Suppression during known events: a holiday, a planned data-pipeline migration, or a marketing campaign predictably shifts distributions; suppressing or down-weighting alerts during a pre-registered known-event window avoids training your on-call team to ignore alerts that fire during Black Friday every single year.

Worked example

Three concrete example rules: (1) latency: page if p99 exceeds 500ms for more than 5 cumulative minutes in a 15-minute window; (2) prediction-distribution shift on a key feature: warn if that feature's PSI exceeds its own control-chart threshold for 3 consecutive daily evaluations, escalate to critical if it also coincides with a measured accuracy drop; (3) accuracy: page if rolling 24-hour accuracy drops more than 2 standard deviations below its 90-day trailing mean. Separately: flag inputs that look adversarial rather than merely drifted (a sudden burst of feature values sitting exactly at a decision boundary, or an unusual spike in a single user/session generating an implausible volume of requests) as its OWN alert category, since the response (rate-limit, block, investigate for abuse) is different from the response to ordinary drift (investigate, consider retraining).

Trade-offs & pitfalls

The single biggest driver of alert fatigue in practice isn't any one bad threshold, it's failing to distinguish "worth investigating" from "worth waking someone up for": routing every signal through the SAME severity path guarantees the team eventually tunes out the channel entirely. The fix is disciplined tiering plus a regular retrospective on alert precision (of the pages that fired this month, how many led to a real action?) so thresholds get tuned based on evidence, not left static forever.

Direct answer

An SLI is a directly measured signal (p95 latency, error rate); an SLO is a target for that signal over a time window (99.9% of requests under 200ms, measured weekly); an SLA is the externally-facing commitment, usually with a consequence attached, built on top of one or more SLOs.

Structured elaboration

For an online recommendation model, a reasonable SLI/SLO set:

• Latency SLI: p95 inference latency. SLO: p95 under 100ms, measured over a rolling 7-day window.
• Availability SLI: fraction of requests served successfully (not erroring). SLO: 99.9% availability over 30 days.
• Prediction-quality SLI: a proxy business metric like click-through-rate uplift over a baseline (not raw accuracy, since ground truth on "was this the RIGHT recommendation" is rarely available in real time). SLO: CTR uplift stays within 2 percentage points of its trailing 30-day average.
• Feature-freshness SLI: age of the most recent feature update feeding the model. SLO: 95% of predictions served with features no more than 5 minutes stale.

Beyond latency and error rate, ML-specific reliability metrics worth SLO-ing: model calibration (Brier score, since a well-calibrated model matters even when raw accuracy looks stable), prediction stability (the rate at which predictions for the SAME entity change between consecutive scoring runs, since a model that flip-flops erodes trust even if each individual prediction is defensible), data-pipeline freshness, label-availability lag, and slice-level performance (aggregate metrics can hide a model that's badly wrong for one important subgroup).

Worked example

Alerting and escalation tie back to these SLOs directly: a single SLI reading outside its SLO band for one evaluation window is informational; a SUSTAINED breach (several consecutive windows) escalates to a warning routed to the model owner; and an SLA-threatening breach (approaching the point where the external commitment itself would be violated) escalates to a page with an error-budget-consumption note, so the on-call engineer immediately knows not just "something's wrong" but "how much runway is left before this becomes a customer-facing commitment violation."

Trade-offs & pitfalls

The tight and loose ends of the threshold spectrum both fail: too tight (alerting on every small wobble) trains the team to ignore the channel; too loose (only alerting once you're already near an SLA breach) leaves no runway to fix the problem before it becomes externally visible. The practical calibration is setting the SLO with meaningful headroom below the SLA, so an SLO breach is a genuine early warning, not simultaneous with the moment things become a customer-facing problem.

Direct answer

A job-FAILURE alert means the pipeline itself broke, a task errored, a deadline was missed, an exit code was non-zero, and it needs an on-call engineer who can restart, retry, or fix infrastructure. A data-quality-REGRESSION alert means the job ran and completed successfully but the data it produced looks wrong, a schema changed, null rates spiked, row counts fell outside tolerance, and it needs a data owner who understands the business meaning of the data, not necessarily an infrastructure fix. Conflating the two into one generic "something's wrong" page routes both kinds of problems to whoever happens to be on-call, even when they lack the context to act on half of what they're paged for.

Structured elaboration

A useful third, intermediate category is a WARNING that doesn't clearly fall into either bucket yet (a job that's running slow but hasn't missed its deadline), which should go to a low-urgency channel rather than paging either group.

Worked example

Concretely, define two independent alert rules on the same pipeline: job_exit_code != 0 OR run_duration > deadline fires a JOB-FAILURE page to the on-call rotation, while abs(row_count - trailing_median) / trailing_median > 0.3 OR null_rate(required_field) > 0.02 fires a DATA-QUALITY page to the data owner's channel, evaluated independently of whether the job itself succeeded. This independence matters: a run can trip BOTH (the job crashed halfway through, producing incomplete data) or EITHER alone (the job succeeded cleanly but an upstream source silently sent bad data; or the job failed outright with zero data-quality signal to evaluate because no output was produced at all, in which case only the job-failure alert should fire, and the data-quality check should be skipped rather than falsely reporting "100% null rate" on a run that produced nothing).

Trade-offs and pitfalls

The pitfall in implementation is letting the data-quality check run and alert even when the job itself failed and produced no meaningful output, which generates a confusing, redundant second alert for the same underlying incident; guard the data-quality evaluation on the job having actually completed. The other common mistake is routing both alert types to the same on-call rotation "to be safe," which defeats the purpose: an infrastructure engineer paged for a data-quality regression they cannot diagnose (they don't know if a 3% null-rate spike on a specific business field is actually a problem) either ignores it or escalates it anyway, adding latency instead of removing it.