Clarify objective & constraints: we want higher signups but not at the expense of long-term retention/LTV. I’d (1) define measurable lead-quality signals, (2) build a quality score and monitor cohorts, and (3) add automated guardrails and experiment rules.1) Define quality & metrics- Leading proxies: 7/30-day active retention, time-to-first-value (TTFV), conversion to paid, engagement events per week.- Downstream business KPIs: 90-day churn, ARPU, LTV.- Secondary signals: fraud flags, geolocation, campaign source, form-completion quality.2) Measure & score- Create a Lead Quality Score using logistic regression or gradient-boosted model predicting 90-day retention (features: source, UTM, device, TTFV, initial events).- Produce daily cohorts by acquisition channel/experiment arm and compute retention & predicted LTV.Sample SQL (compute 7-day retention by experiment arm):sql
WITH signups AS (
SELECT user_id, experiment_arm, signup_date
FROM users
WHERE signup_date BETWEEN '2025-01-01' AND '2025-02-01'
),
events AS (
SELECT user_id, MIN(event_date) as first_event_date,
MAX(CASE WHEN event_date <= signup_date + INTERVAL '7 days' THEN 1 ELSE 0 END) as active_7d
FROM user_events JOIN signups USING(user_id)
GROUP BY user_id, signup_date
)
SELECT s.experiment_arm,
COUNT(*) as signups,
SUM(e.active_7d)::float / COUNT(*) as retention_7d
FROM signups s LEFT JOIN events e USING(user_id)
GROUP BY s.experiment_arm;
