MLOps: Monitoring, Retraining, and Lifecycle Management Questions

Direct answer

A rollback runbook needs explicit steps for detection, decision, execution, verification, and communication, and among the mechanisms available: an instant traffic switch, redeploying a previous artifact, or a feature-flag kill switch: the choice trades complexity and downtime against how much diagnostic evidence survives the rollback.

Structured elaboration

1. Detection: the alert (or human report) that triggers consideration of a rollback: this stage should be fast and should immediately assemble the standard diagnostic bundle (recent deploys, drift status) rather than starting an investigation from scratch.
2. Decision: given the assembled evidence, decide whether rollback is the right response (a known-good previous state exists and the cause looks deploy-related) versus a different response (real-world drift, where there's nothing to "roll back" to).
3. Execution: for a registry-tracked model with instantaneous traffic switching, this is typically the fastest and lowest-complexity option: flip the serving layer's active-version pointer back to the previous registry entry, with essentially zero downtime since both versions are already deployed and warm.
4. Verification: confirm the rollback actually resolved the issue (metrics recovering, error rate back to baseline) rather than assuming success the moment the switch is flipped: a rollback that doesn't actually fix the problem (because the cause wasn't the model version at all) needs to be caught quickly, not discovered hours later.
5. Communication: notify affected stakeholders (on-call, model owner, product) of the rollback, its trigger, and current status: even a fast, successful rollback needs a paper trail for the eventual postmortem.

Comparing the three mechanisms: an instant traffic switch (complexity: low, downtime: near-zero) is the default when both versions are already deployed and warm: trivial to trigger, trivial to verify, but destroys no evidence about WHY the bad version failed (it's still running, just not receiving traffic, so you can investigate it further if needed). Redeploying a previous artifact (complexity: medium, downtime: brief, during the redeploy) is needed when the "current" and "previous" states aren't both already warm and ready: slower, and diagnosability depends on whether the bad version's state was preserved before redeployment overwrote it. A feature-flag kill switch (complexity: low to build once in place, near-zero downtime) is best for surgically disabling a SPECIFIC risky behavior (a new scoring rule, a specific model-driven feature) without a full version rollback: useful when the problem is narrower than "revert the whole model."

Worked example

Adding legal/stakeholder communication for a high-impact incident: beyond the standard on-call notification, a customer-facing or regulatory-relevant rollback needs a parallel communication track: legal assesses notification obligations while the technical rollback proceeds, and a prepared communication template (rather than drafting from scratch mid-incident) speeds up an otherwise slow, high-stakes coordination step.

Trade-offs & pitfalls

The instinct to reach for the fastest mechanism (instant traffic switch) by default is usually right, but it's worth explicitly checking whether the bad version needs to stay reachable in some form for further investigation: a rollback that also immediately terminates the bad version's infrastructure can destroy exactly the evidence (logs, in-memory state) a later root-cause investigation would have wanted.

Direct answer

An SLI is a directly measured signal (p95 latency, error rate); an SLO is a target for that signal over a time window (99.9% of requests under 200ms, measured weekly); an SLA is the externally-facing commitment, usually with a consequence attached, built on top of one or more SLOs.

Structured elaboration

For an online recommendation model, a reasonable SLI/SLO set:

• Latency SLI: p95 inference latency. SLO: p95 under 100ms, measured over a rolling 7-day window.
• Availability SLI: fraction of requests served successfully (not erroring). SLO: 99.9% availability over 30 days.
• Prediction-quality SLI: a proxy business metric like click-through-rate uplift over a baseline (not raw accuracy, since ground truth on "was this the RIGHT recommendation" is rarely available in real time). SLO: CTR uplift stays within 2 percentage points of its trailing 30-day average.
• Feature-freshness SLI: age of the most recent feature update feeding the model. SLO: 95% of predictions served with features no more than 5 minutes stale.

Beyond latency and error rate, ML-specific reliability metrics worth SLO-ing: model calibration (Brier score, since a well-calibrated model matters even when raw accuracy looks stable), prediction stability (the rate at which predictions for the SAME entity change between consecutive scoring runs, since a model that flip-flops erodes trust even if each individual prediction is defensible), data-pipeline freshness, label-availability lag, and slice-level performance (aggregate metrics can hide a model that's badly wrong for one important subgroup).

Worked example

Alerting and escalation tie back to these SLOs directly: a single SLI reading outside its SLO band for one evaluation window is informational; a SUSTAINED breach (several consecutive windows) escalates to a warning routed to the model owner; and an SLA-threatening breach (approaching the point where the external commitment itself would be violated) escalates to a page with an error-budget-consumption note, so the on-call engineer immediately knows not just "something's wrong" but "how much runway is left before this becomes a customer-facing commitment violation."

Trade-offs & pitfalls

The tight and loose ends of the threshold spectrum both fail: too tight (alerting on every small wobble) trains the team to ignore the channel; too loose (only alerting once you're already near an SLA breach) leaves no runway to fix the problem before it becomes externally visible. The practical calibration is setting the SLO with meaningful headroom below the SLA, so an SLO breach is a genuine early warning, not simultaneous with the moment things become a customer-facing problem.

Direct answer

A/B testing, canary, shadow, and phased rollout all reduce risk when releasing a new model, but they route traffic and reveal information differently: shadow reveals nothing to users but validates behavior; canary and phased rollout expose a small, growing slice of real users; A/B testing runs two versions concurrently at a fixed split specifically to measure a statistically valid comparison.

Structured elaboration

Worked example

For a model that affects revenue, start with shadow testing first regardless of urgency: it's the only strategy with zero user-facing risk, and it catches the most basic classes of failure (crashes, wildly implausible predictions, latency blowups) before any real user is exposed at all. Once shadow testing clears, move to a canary with a small initial percentage, watching BOTH technical metrics (latency, error rate) and business metrics (the actual revenue-relevant signal) before progressing: since a model can be technically healthy (fast, error-free) while still being a business regression (worse recommendations, lower conversion), and only real traffic exposure reveals that. Feature flags matter throughout this progression as the mechanism that makes each traffic-percentage step instantly reversible without a code redeploy; a database or schema migration that the feature computation depends on needs to be BACKWARD-compatible for the duration of the rollout, since old and new model versions may be serving simultaneously mid-rollout and both need the schema to work.

Trade-offs & pitfalls

Shadow testing's blind spot is real: because nothing is actually served, it can't catch problems that only manifest from genuine user REACTION to a different prediction (a subtly-different ranking that changes user behavior in a way shadow testing, which never shows the prediction to a real user, structurally cannot observe): this is exactly why shadow testing is a necessary first gate, not a sufficient one on its own, and always needs to be followed by a real-traffic stage (canary or A/B) before full rollout.

Situation

I was on-call when an alert fired for a recommendation model showing a meaningful drop in click-through rate over about six hours, with no corresponding deploy in that window and infrastructure metrics (latency, error rate) looking completely normal.

Task

As the on-call engineer, I needed to quickly triage whether this was a real, actionable model-quality problem or noise, decide on an appropriate response within our incident SLA, and mitigate any real user impact while the investigation continued.

Action

I started with the standard triage checklist: confirmed the CTR drop was real and not an instrumentation issue by cross-checking a second, independent metrics pipeline, then checked recent feature-distribution drift, which showed nothing unusual: inputs looked normal. I broadened the investigation to check for anything OUTSIDE the model itself and found a product team had shipped a UI change to the recommendation carousel's visual layout earlier that same day, unrelated to the model but directly affecting how recommendations were displayed and therefore how likely users were to click regardless of the underlying recommendation quality. I coordinated with the product team to confirm the timing aligned precisely with their deploy, and we agreed on a quick UI-side revert as the fastest mitigation while they investigated their change more thoroughly, rather than my team spending further time investigating a model that, by this point, had strong evidence it wasn't actually the cause.

Result

CTR recovered within an hour of the UI revert, confirming the root cause was the product change, not the model. I documented the incident and the cross-team investigation timeline in the postmortem, and afterward proposed (and helped implement) a lightweight practice: any UI change touching a model-driven surface gets flagged to the on-call ML rotation in advance, so a future incident like this is diagnosed faster by starting from "was anything else deployed" rather than defaulting to a model-first investigation.

Trade-offs & pitfalls

The specific process change afterward: flagging UI changes near a model-driven surface to ML on-call in advance: addresses the actual lesson: an ML on-call's default instinct (check the model first) is reasonable but not universally correct, and a broader "what else changed" check, even outside your own team's deploys, needs to happen early in triage rather than after model-side investigation comes up empty.

Direct answer

dedupe_alerts groups incoming alerts by a root-cause signature (metric, model, feature) and merges any alerts within that group that land within a 10-minute sliding window of the group's most recent member, so a burst of near-simultaneous alerts about the same underlying issue collapses to one representative entry.

Structured elaboration

from collections import defaultdict

def dedupe_alerts(alerts: list[dict]) -> list[dict]:
    WINDOW_SECONDS = 10 * 60
    groups = defaultdict(list)  # root-cause signature -> list of time-windows, each a list of alerts

    for a in sorted(alerts, key=lambda x: x["timestamp"]):
        key = (a["metric_name"], a["model_id"], a["feature_name"])
        windows = groups[key]
        if windows and (a["timestamp"] - windows[-1][-1]["timestamp"]) <= WINDOW_SECONDS:
            windows[-1].append(a)          # still within the sliding window of the last alert in this group
        else:
            windows.append([a])            # gap exceeds 10 minutes: start a new window for this signature

    severity_rank = {"critical": 3, "warning": 2, "informational": 1}
    result = []
    for key, windows in groups.items():
        for window in windows:
            representative = max(window, key=lambda x: severity_rank.get(x["severity"], 0))
            result.append({**representative, "grouped_count": len(window)})
    return result

Grouping key: (metric_name, model_id, feature_name): this is a proxy for "same root cause": the same metric drifting on the same feature of the same model is almost certainly one underlying event, not several independent ones. Window semantics: it's a SLIDING window relative to the most recent alert in the current group (not a fixed clock-aligned bucket), so a steady trickle of alerts each 8 minutes apart stays merged into one ongoing group indefinitely, while a 12-minute gap starts a fresh group: this matches the intuition that a persistent issue should read as one incident regardless of how long it lasts.

Complexity: sorting is $O(n \log n)$; the single pass that follows is $O(n)$, so overall $O(n \log n)$ dominated by the sort, with $O(n)$ space for the grouped output.

Worked example

Independently executed in a Python sandbox: five input alerts: three for (psi, m1, amount) at t=0s, 300s, 700s (each within 10 minutes of the prior), one for the same signature at t=1400s (700 seconds after the prior alert, exceeding the 600-second window, so it starts a NEW group), and one for a different model (m2) at t=50s. The function correctly returned 3 output groups: the first m1 window merged 3 alerts (grouped_count=3) and correctly surfaced the CRITICAL severity present among them as the representative; the second m1 window and the m2 alert each stood alone.

Trade-offs & pitfalls

The grouping key is a simplification: two genuinely DIFFERENT root causes that happen to share the same (metric, model, feature) triple within 10 minutes (rare, but possible: say a drift alert and an unrelated data-quality alert both touching the same feature) get merged into one entry, silently hiding the second issue until the grouped alert is fully investigated. A more robust production version would carry an explicit correlation_id set by whatever upstream system detected the anomaly, rather than inferring grouping purely from a (metric, model, feature) tuple.