MLOps: Monitoring, Retraining, and Lifecycle Management Questions

Direct answer

MLOps is the discipline of running machine learning systems reliably in production, not just building models. It extends DevOps practice with the parts that are unique to ML: data changes underneath you, models decay silently, and "the code passed its tests" doesn't mean "the model is still right."

Structured elaboration

The lifecycle runs in a loop, not a line:

1. Data collection and preprocessing: sourcing and cleaning the data a model will train on, with the same rigor you'd apply to production data (schema checks, deduplication, leakage screens).
2. Training: producing a candidate model artifact from a pinned dataset and code version.
3. Validation: checking the candidate against offline metrics and a baseline before it ever sees production traffic.
4. Deployment: shipping the artifact behind a controlled rollout (canary, shadow, or blue-green), not a hard cutover.
5. Monitoring: watching data drift, prediction quality, and infrastructure health once the model is live.
6. Feedback loop: routing what monitoring finds (a metric regression, a drift alert, new labeled data) back into the next training run, closing the loop.

Stakeholders span far more than the model builder: SRE/infra own the serving platform's reliability, data engineering owns the pipelines feeding the model, product owns whether the model is actually moving the business metric it's meant to move, and privacy/legal own what data can be logged and retained. A model that "works" in a notebook and ships without coordinating these groups is how outages happen.

Worked example

Two concrete failures MLOps practice exists to prevent:

• Silent model staleness: a fraud model trained on last year's transaction patterns keeps returning predictions with no error thrown, because nothing about serving a stale model looks broken from an infrastructure point of view: the request comes in, a response goes out, the latency graph is flat. Only a drift/performance monitor watching prediction quality over time catches this; a standard uptime check does not.
• Training-serving skew: a feature is computed one way in the offline training pipeline (say, a 30-day rolling average using calendar days) and a subtly different way in the online serving path (a 30-day rolling average using a different timezone boundary). Every offline validation metric looks great because training and offline evaluation share the same feature code; only real production traffic exposes the mismatch, and it shows up as a mysterious online/offline metric gap with no code change to point to.

Trade-offs & pitfalls

The common failure mode is treating MLOps as "the tooling I bolt on after the model works." In practice the lifecycle stages constrain each other: a training pipeline that doesn't pin its data version makes monitoring's job impossible later (you can't tell if a metric moved because of drift or because the training data quietly changed), and a deployment strategy with no fast rollback path makes every retraining decision higher-stakes than it needs to be. The tools at each stage (a feature store, an experiment tracker like MLflow, a model registry, a CI/CD pipeline, an observability stack) matter less than whether the stages are wired into one loop; disconnected point tools that don't share a model-version identifier recreate the same operational blind spots MLOps is meant to remove.

Direct answer

A lineage and provenance system needs to capture linked metadata at four levels: dataset, feature, transformation, and model: store it in a form that supports both point lookups (trace this prediction back) and broad audit queries (find every model touched by this dataset), and expose it through APIs that make lineage a queryable capability, not just a passive log.

Structured elaboration

1. What to capture at each level: dataset (snapshot id, schema, source, ingestion timestamp), feature (transformation code version, input dataset references, computation timestamp), transformation (the specific code/config that ran, its inputs and outputs), model (training run id, all feature versions consumed, hyperparameters, evaluation metrics).
2. Storage model: a graph database (nodes for datasets/features/models, edges for "derived from" relationships) is a natural fit for lineage's inherently graph-shaped structure and makes multi-hop traversal queries (trace back three levels) efficient; a relational store can work too, especially if most queries are shallow (one or two hops) and you value operational simplicity over graph-native traversal performance. The choice trades query-pattern fit against operational familiarity: a team without graph-database experience may reasonably prefer a well-indexed relational schema over introducing a new storage paradigm.
3. APIs for querying lineage: a "trace forward" API (given a dataset, what models were trained on it, directly or transitively) and a "trace backward" API (given a model or prediction, what data and code produced it) cover the two directions audits and debugging actually need: building these as first-class query APIs, not ad-hoc scripts against the raw storage, is what makes lineage genuinely USABLE rather than merely recorded.
4. Performance at high volume: for high-prediction-volume systems, don't store per-PREDICTION lineage as individually-linked graph nodes (this doesn't scale): instead, link each prediction to its MODEL VERSION's lineage (which is far lower cardinality), and rely on the model version's own snapshot-time lineage record to answer "what data/code produced predictions from this version," rather than trying to trace every individual prediction's full graph independently.

Worked example

An example audit query: "which currently-active models were trained, even indirectly, on dataset X": starting from dataset X's node, traverse forward through every feature computed from it, then forward again through every model that consumed any of those features, filtering to models still marked active in the registry. This is exactly the query a regulatory or legal team would ask after discovering a data-quality problem in dataset X, and it's the kind of multi-hop traversal a graph-native storage model handles far more naturally than a relational join chain would.

For an MVP deliverable within six months: prioritize model-level and dataset-level lineage first (the two levels that answer the most common real audit questions), defer fine-grained feature-transformation-level lineage to a later phase, and build the "trace backward from a model" API before the more complex "trace forward from a dataset" one, since backward tracing from a specific known-bad model is the more common initial use case (debugging) compared to forward tracing from a dataset (a rarer, audit-driven need).

Trade-offs & pitfalls

The temptation is building the most complete possible lineage graph (every field, every transformation step, per-prediction granularity) before shipping anything: the MVP framing exists specifically to counter that instinct: ship the two levels (model, dataset) that answer 80% of real questions first, and let genuine demand for finer granularity (if it materializes) drive the next phase, rather than guessing upfront at a level of detail nobody may actually need.

Direct answer

Model monitoring is the practice of continuously tracking whether a deployed model is still healthy: whether its inputs still look like what it was trained on, whether its predictions still perform well, and whether the infrastructure serving it is behaving. It spans four signal categories: data/feature drift, model performance, latency/resource usage, and business KPIs.

Structured elaboration

• Data/feature drift: are the inputs the model sees today still similar to training-time inputs? Matters because a model's guarantees only hold within the distribution it was trained on; drifted inputs are the earliest warning that quality may degrade.
• Model performance: accuracy, precision/recall, calibration: measured once labels arrive. Matters because it's the ground truth of whether the model is actually doing its job, though it often lags behind drift signals by however long labels take to arrive.
• Latency and resource usage: p95/p99 inference latency, CPU/GPU utilization, memory. Matters because a model that's "accurate" but too slow to serve within its SLA is still a production failure, just a different kind.
• Business KPIs: the downstream metric the model actually exists to move (conversion, revenue, fraud losses prevented). Matters because a model can look statistically fine on every ML metric while the business impact it's meant to deliver quietly erodes: this is the category that ultimately justifies the model's existence.

Monitoring vs. observability: monitoring answers pre-defined questions ("is accuracy above X") with dashboards and alerts you built in advance. Observability is the broader capability to ask NEW questions of your system after something unexpected happens, using rich enough telemetry (not just aggregated metrics, but queryable raw signals) to investigate a novel failure mode you didn't anticipate. A concrete example: your accuracy-drop alert fires (monitoring did its job), but figuring out WHY: slicing by region, correlating with a specific upstream pipeline's timestamp, comparing feature distributions for the specific failing cohort: requires observability: the ability to drill into raw, high-cardinality telemetry that a pre-built dashboard was never designed to show.

Worked example

A team with strong monitoring but weak observability catches "accuracy dropped 5%" within minutes (a threshold alert fired) but then spends two days manually pulling logs to figure out why, because their telemetry only stores aggregated daily metrics, not per-request feature values they can slice and filter. A team with both catches the drop AND, within the same incident, filters the raw per-request logs by region and immediately sees the drop is 100% concentrated in one geography: turning a two-day investigation into a twenty-minute one.

Trade-offs & pitfalls

The trap is treating any one category as sufficient on its own: teams that only watch latency and error rate (classic infra monitoring) miss silent quality degradation entirely, since a model can serve fast, error-free, WRONG predictions indefinitely. Business-KPI-only monitoring is the opposite trap: it eventually catches real problems but with a long detection lag, since business metrics are noisy and slow-moving compared to a direct drift signal.

Direct answer

A model-registry SQL schema needs tables for models, model_versions, artifacts, and metrics with clear foreign-key relationships, and the "latest production version" query relies on filtering to the production stage and taking the most recent by promotion timestamp per model.

Structured elaboration

CREATE TABLE models (
    model_id       VARCHAR(64) PRIMARY KEY,
    model_name     VARCHAR(255) NOT NULL,
    owner          VARCHAR(255) NOT NULL,
    created_at     TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP
);

CREATE TABLE model_versions (
    version_id       VARCHAR(64) PRIMARY KEY,
    model_id         VARCHAR(64) NOT NULL REFERENCES models(model_id),
    version_number   INT NOT NULL,
    artifact_uri     VARCHAR(1024) NOT NULL,
    sha256           CHAR(64) NOT NULL,
    stage            VARCHAR(32) NOT NULL DEFAULT 'dev',  -- dev, staging, production, archived
    parent_version_id VARCHAR(64) REFERENCES model_versions(version_id),
    created_at       TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    promoted_at      TIMESTAMP,
    author           VARCHAR(255) NOT NULL,
    UNIQUE(model_id, version_number)
);

CREATE TABLE artifacts (
    artifact_id    VARCHAR(64) PRIMARY KEY,
    version_id     VARCHAR(64) NOT NULL REFERENCES model_versions(version_id),
    artifact_type  VARCHAR(64) NOT NULL,  -- 'weights', 'preprocessor', 'tokenizer'
    storage_uri    VARCHAR(1024) NOT NULL,
    sha256         CHAR(64) NOT NULL
);

CREATE TABLE metrics (
    metric_id    VARCHAR(64) PRIMARY KEY,
    version_id   VARCHAR(64) NOT NULL REFERENCES model_versions(version_id),
    metric_name  VARCHAR(128) NOT NULL,
    metric_value DOUBLE PRECISION NOT NULL,
    cohort       VARCHAR(128)  -- NULL for an aggregate metric, a segment name for a sliced metric
);

Latest production version per model, with its primary metric:

WITH latest_prod AS (
    SELECT model_id, version_id,
           ROW_NUMBER() OVER (PARTITION BY model_id ORDER BY promoted_at DESC) AS rn
    FROM model_versions
    WHERE stage = 'production'
)
SELECT m.model_name, lp.version_id, mt.metric_value AS primary_metric
FROM latest_prod lp
JOIN models m ON m.model_id = lp.model_id
LEFT JOIN metrics mt ON mt.version_id = lp.version_id AND mt.metric_name = 'primary' AND mt.cohort IS NULL
WHERE lp.rn = 1;

Which model version served a given request, for audit:

CREATE TABLE serving_log (
    request_id   VARCHAR(64) PRIMARY KEY,
    version_id   VARCHAR(64) NOT NULL REFERENCES model_versions(version_id),
    served_at    TIMESTAMP NOT NULL
);

Worked example

A JSON-metadata diff CLI is the natural companion tool: given two versions' hyperparameters and metrics JSON blobs, print a human-readable diff of changed hyperparameters, and flag any metric that regressed beyond a statistically-meaningful threshold (not just any numeric change): testable directly with fixed input JSON pairs and an expected diff output, no database needed for that part.

Trade-offs & pitfalls

The ROW_NUMBER()-based "latest production" query assumes exactly one production version per model at a time: if the schema needs to support genuine A/B testing between two simultaneously-production versions, stage = 'production' alone is insufficient and the schema needs an additional traffic-allocation concept (a separate table mapping version_id to a traffic percentage) rather than treating "production" as a single-version state.

Direct answer

The SQL query computes a 7-day and 30-day rolling mean per feature, converts their difference to a z-score using the 30-day window's own standard deviation, and flags any feature whose recent mean has moved more than 3 standard deviations from its longer-run baseline.

Structured elaboration

WITH recent_stats AS (
    SELECT
        feature_name,
        AVG(feature_value) AS recent_mean,
        COUNT(*) AS recent_n
    FROM features
    WHERE event_time >= CURRENT_TIMESTAMP - INTERVAL '7' DAY
    GROUP BY feature_name
),
baseline_stats AS (
    SELECT
        feature_name,
        AVG(feature_value) AS baseline_mean,
        STDDEV(feature_value) AS baseline_stddev,
        COUNT(*) AS baseline_n
    FROM features
    WHERE event_time >= CURRENT_TIMESTAMP - INTERVAL '30' DAY
    GROUP BY feature_name
)
SELECT
    r.feature_name,
    r.recent_mean,
    b.baseline_mean,
    b.baseline_stddev,
    (r.recent_mean - b.baseline_mean) / NULLIF(b.baseline_stddev, 0) AS z_score
FROM recent_stats r
JOIN baseline_stats b ON r.feature_name = b.feature_name
WHERE ABS((r.recent_mean - b.baseline_mean) / NULLIF(b.baseline_stddev, 0)) > 3
ORDER BY ABS((r.recent_mean - b.baseline_mean) / NULLIF(b.baseline_stddev, 0)) DESC;

NULLIF(b.baseline_stddev, 0) guards against a division-by-zero error for a feature that happens to be perfectly constant over the baseline window (real, if rare: a boolean flag that never varied, for instance): without this guard, such a feature would either error the whole query or produce an undefined result depending on the SQL engine's behavior.

Assumptions about independence and sample size: this z-score treats each daily observation as independent, which is often NOT strictly true for a feature with autocorrelation (today's value predicting tomorrow's, common in many real features): autocorrelation makes the effective sample size smaller than the raw row count suggests, meaning the z-score's implied confidence is somewhat OVERSTATED for an autocorrelated feature. It also assumes roughly normal within-baseline variation; for a heavily skewed feature, a z-score threshold calibrated for normality may fire too often or too rarely relative to what's actually unusual for that specific feature's real shape.

Scaling to petabyte tables: the two CTEs each require a full scan and aggregation over their respective windows: at petabyte scale, this needs the underlying tables PARTITIONED by event_time (so the WHERE clause prunes partitions rather than scanning the full table), and ideally the baseline aggregates PRE-COMPUTED and materialized on a schedule (daily) rather than recomputed fully on every ad-hoc query run, since recomputing a 30-day aggregate over petabyte-scale data on every single check is wasteful when the underlying baseline changes slowly.

Trade-offs & pitfalls

The independence assumption is the subtlest issue here and the one most likely to be silently wrong in practice: a feature with meaningful day-to-day autocorrelation will produce a z-score whose "3 standard deviations" doesn't mean what it would for genuinely independent observations, making the flagged threshold less statistically meaningful than it appears; a more rigorous version would explicitly account for autocorrelation (an effective-sample-size correction) rather than treating every row as independent evidence.