MLOps: Monitoring, Retraining, and Lifecycle Management Questions

Direct answer

Model monitoring is the practice of continuously tracking whether a deployed model is still healthy: whether its inputs still look like what it was trained on, whether its predictions still perform well, and whether the infrastructure serving it is behaving. It spans four signal categories: data/feature drift, model performance, latency/resource usage, and business KPIs.

Structured elaboration

• Data/feature drift: are the inputs the model sees today still similar to training-time inputs? Matters because a model's guarantees only hold within the distribution it was trained on; drifted inputs are the earliest warning that quality may degrade.
• Model performance: accuracy, precision/recall, calibration: measured once labels arrive. Matters because it's the ground truth of whether the model is actually doing its job, though it often lags behind drift signals by however long labels take to arrive.
• Latency and resource usage: p95/p99 inference latency, CPU/GPU utilization, memory. Matters because a model that's "accurate" but too slow to serve within its SLA is still a production failure, just a different kind.
• Business KPIs: the downstream metric the model actually exists to move (conversion, revenue, fraud losses prevented). Matters because a model can look statistically fine on every ML metric while the business impact it's meant to deliver quietly erodes: this is the category that ultimately justifies the model's existence.

Monitoring vs. observability: monitoring answers pre-defined questions ("is accuracy above X") with dashboards and alerts you built in advance. Observability is the broader capability to ask NEW questions of your system after something unexpected happens, using rich enough telemetry (not just aggregated metrics, but queryable raw signals) to investigate a novel failure mode you didn't anticipate. A concrete example: your accuracy-drop alert fires (monitoring did its job), but figuring out WHY: slicing by region, correlating with a specific upstream pipeline's timestamp, comparing feature distributions for the specific failing cohort: requires observability: the ability to drill into raw, high-cardinality telemetry that a pre-built dashboard was never designed to show.

Worked example

A team with strong monitoring but weak observability catches "accuracy dropped 5%" within minutes (a threshold alert fired) but then spends two days manually pulling logs to figure out why, because their telemetry only stores aggregated daily metrics, not per-request feature values they can slice and filter. A team with both catches the drop AND, within the same incident, filters the raw per-request logs by region and immediately sees the drop is 100% concentrated in one geography: turning a two-day investigation into a twenty-minute one.

Trade-offs & pitfalls

The trap is treating any one category as sufficient on its own: teams that only watch latency and error rate (classic infra monitoring) miss silent quality degradation entirely, since a model can serve fast, error-free, WRONG predictions indefinitely. Business-KPI-only monitoring is the opposite trap: it eventually catches real problems but with a long detection lag, since business metrics are noisy and slow-moving compared to a direct drift signal.

Direct answer

An observability dashboard for a production classifier needs at least eight metrics spanning performance, data, infrastructure, and business categories, each with its own alert rule, because a single "is it healthy" number can't distinguish a model problem from a data problem from an infra problem.

Structured elaboration

Worked example

For a loan-approval model specifically: an SLO might be "p95 inference latency under 200ms, 99.9% availability, and rolling 7-day approval-rate-by-protected-group parity within 5 percentage points of the training-time baseline": the last of these ties directly into fairness monitoring, since a loan model's business-KPI dashboard without a fairness cut is incomplete for this domain. Concrete tooling: Prometheus/Grafana for the infra and latency layer (they're built for time-series metrics and alerting), a dedicated ML-observability layer like Evidently for drift/data-quality checks (since Prometheus has no native concept of a feature distribution), and the business-KPI dashboard usually lives in whatever BI tool the business already uses, joined to the model-version tag so a KPI dip is attributable to a specific deployed version.

Trade-offs & pitfalls

The most common failure in a first-pass dashboard design is putting everything in one undifferentiated wall of charts. Grouping by category and giving EACH category its own alert (rather than one giant composite health score) is what lets an on-call engineer immediately localize a problem: an infra alert routes to SRE, a drift alert routes to the model owner, a business-KPI alert routes to product: collapsing them into one signal forces someone to manually figure out which team should even be paged.

Direct answer

An SLI is a directly measured signal (p95 latency, error rate); an SLO is a target for that signal over a time window (99.9% of requests under 200ms, measured weekly); an SLA is the externally-facing commitment, usually with a consequence attached, built on top of one or more SLOs.

Structured elaboration

For an online recommendation model, a reasonable SLI/SLO set:

• Latency SLI: p95 inference latency. SLO: p95 under 100ms, measured over a rolling 7-day window.
• Availability SLI: fraction of requests served successfully (not erroring). SLO: 99.9% availability over 30 days.
• Prediction-quality SLI: a proxy business metric like click-through-rate uplift over a baseline (not raw accuracy, since ground truth on "was this the RIGHT recommendation" is rarely available in real time). SLO: CTR uplift stays within 2 percentage points of its trailing 30-day average.
• Feature-freshness SLI: age of the most recent feature update feeding the model. SLO: 95% of predictions served with features no more than 5 minutes stale.

Beyond latency and error rate, ML-specific reliability metrics worth SLO-ing: model calibration (Brier score, since a well-calibrated model matters even when raw accuracy looks stable), prediction stability (the rate at which predictions for the SAME entity change between consecutive scoring runs, since a model that flip-flops erodes trust even if each individual prediction is defensible), data-pipeline freshness, label-availability lag, and slice-level performance (aggregate metrics can hide a model that's badly wrong for one important subgroup).

Worked example

Alerting and escalation tie back to these SLOs directly: a single SLI reading outside its SLO band for one evaluation window is informational; a SUSTAINED breach (several consecutive windows) escalates to a warning routed to the model owner; and an SLA-threatening breach (approaching the point where the external commitment itself would be violated) escalates to a page with an error-budget-consumption note, so the on-call engineer immediately knows not just "something's wrong" but "how much runway is left before this becomes a customer-facing commitment violation."

Trade-offs & pitfalls

The tight and loose ends of the threshold spectrum both fail: too tight (alerting on every small wobble) trains the team to ignore the channel; too loose (only alerting once you're already near an SLA breach) leaves no runway to fix the problem before it becomes externally visible. The practical calibration is setting the SLO with meaningful headroom below the SLA, so an SLO breach is a genuine early warning, not simultaneous with the moment things become a customer-facing problem.

Direct answer

A stateless microservice's correctness is a function of its code and the request; redeploy the same code and you get the same behavior forever. An ML model service's correctness also depends on the training data, and that dependency is invisible in the code itself, which is what makes it operationally different.

Structured elaboration

• Determinism: a stateless service given the same input and the same code version returns the same output, always. A model given the same input can still degrade over time even with zero code changes, because the world the model was trained on has moved on. "It didn't change" is not a defense against an incident.
• Dependency on training data: a microservice's dependencies are declared (a package.json, a Dockerfile). A model's most important dependency: the training dataset: usually isn't tracked with the same rigor, so "what changed?" during an incident often has no good answer unless you deliberately version it.
• Model versioning and rollback complexity: rolling back a microservice means redeploying a previous container image; behavior is fully restored. Rolling back a model can also require reverting the feature-computation code and confirming the previous model's dependencies (a feature store schema, an embedding table) haven't drifted out from under it in the meantime.
• Reproducibility: rebuilding a microservice from source is trivial. Reproducing a training run months later requires the same data snapshot, same library versions, same hardware-dependent numerics, and often the same random seed: any one of which being unpinned makes "rebuild it and compare" impossible.

Worked example

Concretely: a stateless service incident is almost always "a deploy introduced a bug" or "a dependency broke," and the fix is a code rollback that instantly and fully restores prior behavior. A model-service incident is more often "nothing was deployed, but the input distribution moved": the on-call engineer's first question isn't "what changed in the code" but "what changed in the world," and answering that requires drift monitoring, not a deploy log. This is why an SRE used to microservice on-call needs a different first-response instinct for ML: check feature distributions and recent retrains before assuming a code regression.

Trade-offs & pitfalls

The pitfall for an SRE new to ML on-call is importing the microservice mental model wholesale: treating a model-quality alert like a service-health alert (restart it, roll back the last deploy, check for a bad dependency bump) when the actual cause is upstream data. The fix isn't to abandon SRE discipline, it's to extend it: SLOs, error budgets, and blameless postmortems all still apply, but the telemetry that matters (feature distributions, label lag, model-version-tagged metrics) is different from what a typical service dashboard shows, and the on-call runbook needs to name that difference explicitly rather than assume it's obvious.

Direct answer

Data drift is a change in the distribution of the input features, $P(X)$. Label drift is a change in the distribution of the target, $P(y)$, independent of the inputs. Concept drift is a change in the relationship between inputs and target, $P(y \mid x)$: the same input now genuinely means something different.

Structured elaboration

Worked example

The practical dividing line is label availability. Data drift is detectable immediately and without any ground truth: you're just comparing today's feature distribution to a training-time baseline, which needs no labels at all. Label drift needs the label stream, but not necessarily paired with features (you can watch the marginal rate). Concept drift is the hardest: because $P(y \mid x)$ is about the relationship, you need labeled examples paired with their original features to catch it, and if labels are delayed (common in fraud, churn, or credit risk), your concept-drift signal necessarily lags reality by however long labels take to arrive. A team that only monitors input-feature drift and treats "no drift alarm" as "the model is fine" will miss concept drift entirely until the delayed labels catch up.

Trade-offs & pitfalls

The most common mistake is conflating data drift with a real problem: input distributions shift constantly in healthy systems (seasonality, new user cohorts, a marketing campaign), and most of that shift never touches $P(y \mid x)$. Alerting on data drift alone without asking "did this actually move the label relationship" is the single biggest driver of alert fatigue in production ML monitoring. The senior move is treating data drift as a leading indicator that something might need investigation, and treating measured performance degradation (once labels catch up) as the thing you actually act on.