Approach: produce a single Experiment Impact Score (EIS) that combines normalized primary metric lift (directional; higher is better) with penalties for guardrail breaches. Use variance-aware normalization (z-score using mean and standard error) and apply penalties multiplicatively so large breaches dominate. Use shrinkage (Bayesian/James-Stein style) to reduce noise for low-sample variants.Algorithmic steps:1. For each variant, compute primary metric lift vs control: lift = metric_variant - metric_control.2. Compute standard error (se) of lift; compute z = lift / se.3. Apply shrinkage: z_shrunk = z * (n / (n + tau)) where n = sample size for variant, tau = prior strength (tunable).4. Normalize z_shrunk to 0–1 via logistic: norm = 1 / (1 + exp(-alpha * z_shrunk)) where alpha controls sensitivity.5. For each guardrail g, compute severity_sg = max(0, (value_g - threshold_g) / threshold_range_g). Optionally weight guardrails wg.6. Total penalty P = 1 - min(1, sum(wg * severity_sg)).7. Final EIS = norm * P.Pseudo-SQL (conceptual):sql
WITH metrics AS (
SELECT experiment_id, variant, n,
metric, metric_control,
(metric - metric_control) AS lift,
-- compute se from counts/variance table
SQRT(variance_variant / n + variance_control / n_control) AS se
FROM experiment_metrics
),
shrunk AS (
SELECT *,
(lift / NULLIF(se,0)) * (n / (n + :tau)) AS z_shrunk
FROM metrics
),
normalized AS (
SELECT *,
1.0 / (1.0 + EXP(-:alpha * z_shrunk)) AS norm_score
FROM shrunk
),
guardrail AS (
SELECT experiment_id, variant,
SUM(weight * GREATEST(0, (value - threshold)/NULLIF(range,0))) AS raw_penalty
FROM guardrail_metrics
GROUP BY experiment_id, variant
),
combined AS (
SELECT n.*, 1 - LEAST(1, COALESCE(g.raw_penalty,0)) AS penalty_factor
FROM normalized n
LEFT JOIN guardrail g USING (experiment_id, variant)
)
SELECT experiment_id, variant,
norm_score * penalty_factor AS experiment_impact_score
FROM combined;
Notes on tuning:- alpha controls how z maps to 0–1; higher alpha makes score more binary.- tau controls shrinkage; larger tau reduces influence of small-n variants.- Use multiplicative penalty so severe guardrail breaches nullify high lifts.- Validate with historical experiments and calibrate weights/thresholds.