Investigation Report: Model Drift1) Executive summary- Brief description of observed drift, detection date, primary metric degradation (e.g., AUC drop from 0.82 → 0.74), user/business impact estimate, recommended next-step priority.2) Description of issue- Model name/version, serving endpoint, production cohort, baseline evaluation window (T0) and current window (T1), detection method (monitor alert: delta metric threshold, population stability index, Chi-square/K-S).3) Data snapshots — feature distributions- For each key feature: summary stats at T0 and T1 (count, mean, std, min, max), categorical frequency table, visualization (histogram, KDE, ECDF).- Example SQL (Postgres) to extract aggregated feature stats by day:sql
WITH data AS (
SELECT
DATE(event_time) AS day,
feature_numeric,
feature_cat,
model_score,
label
FROM production.events
WHERE event_time BETWEEN '2025-10-01' AND '2025-11-30'
)
SELECT
day,
COUNT(*) AS n,
AVG(feature_numeric) AS mean_fn,
STDDEV_POP(feature_numeric) AS sd_fn,
MIN(feature_numeric) AS min_fn,
MAX(feature_numeric) AS max_fn
FROM data
GROUP BY day
ORDER BY day;
sql
SELECT
DATE(pred_time) AS day,
AVG(CASE WHEN label=1 THEN model_score END) AS avg_score_pos,
SUM(CASE WHEN label=1 AND model_score>=0.5 THEN 1 ELSE 0 END)::float / NULLIF(SUM(1),0) AS daily_accuracy,
COUNT(*) AS n
FROM predictions.join_labels
WHERE pred_time BETWEEN '2025-10-01' AND '2025-11-30'
GROUP BY day ORDER BY day;
python
import pandas as pd
from scipy.stats import ks_2samp, chi2_contingency
# load snapshots
t0 = pd.read_parquet("snapshot_t0.parquet") # baseline
t1 = pd.read_parquet("snapshot_t1.parquet") # current
# numeric drift (KS test)
for col in numeric_cols:
stat, p = ks_2samp(t0[col].dropna(), t1[col].dropna())
print(col, stat, p)
# categorical drift (Chi-square)
for col in categorical_cols:
obs = pd.concat([t0[col].value_counts(normalize=False),
t1[col].value_counts(normalize=False)], axis=1).fillna(0).values
chi2, p, _, _ = chi2_contingency(obs)
print(col, chi2, p)
