Design Metrics and Impact Measurement Questions

KPI 1 — Org Activation Rate (early onboarding success)

• Definition: % of newly-created organizations that reach a defined activation state within 14 days.
• How to compute: Identify org_created events; for each org, check for activation_event (see activation definition below) timestamp <= org_created + 14d. Activation rate = activated_orgs / total_new_orgs.
• Notes: measure by cohorts (week/month), normalize by org size.

KPI 2 — Admin Engagement Depth (configuration completeness)

• Definition: Average % of mandatory admin configuration steps completed per org in first 30 days.
• How to compute: Instrument admin_config_step events with step_id. For each org, compute completed_steps / total_required_steps. KPI = mean across orgs. Also report distribution (25/50/75/100%).

KPI 3 — Multi-user Usage Density (real end-user adoption)

• Definition: % of active orgs where >= X non-admin users perform core actions weekly (e.g., core_action events) within 30/90 days.
• How to compute: For each org, count unique non-admin user_ids performing core_action in a 7-day window. Mark org active if count >= X (X could be 3 or 10 depending on typical team sizes). KPI = active_orgs_with_threshold / total_orgs.

Activation definition (recommended)

• An org is “activated” when, within 14 days of creation: (1) owner or admin has completed >= 3 key config steps (invite at least 1 teammate, configure integration/setting A, and create first project/record) AND (2) at least 2 distinct non-admin users have performed a core_action once.
• Rationale: Ensures both admin setup and at least minimal multi-user engagement; thresholds adjustable by tier/expected team size.
• Instrumentation: events: org_created, admin_config_step(step_id), user_invite_sent, user_accepted, core_action. Track roles to distinguish admin vs non-admin.
• Reporting suggestions: segment KPIs by org size, industry, and sales tier; track time-to-activation and drop-off funnel to prioritize onboarding improvements.

Event: onboarding_step_attempted

Purpose: Measure where new users experience friction during onboarding by logging each attempt to complete a distinct onboarding step and its outcome.

Event definition:

• event_name: onboarding_step_attempted
• triggered: when user attempts any defined onboarding step (UI action submit/click that advances flow)

Required properties:

• user_id (hashed, nullable) — persistent id if consented; see privacy
• anon_id — session or device id for deduping without PII
• step_id — enum: {create_account, verify_email, add_payment, connect_integration, complete_profile, first_action}
• step_name — human-readable string
• step_state — enum: {started, succeeded, failed, timed_out, cancelled}
• error_code — string (standardized error IDs) when step_state=failed
• elapsed_ms — time in ms from step start to this event
• attempt_number — integer (nth attempt on this step during session)
• platform — enum {web, iOS, Android}
• experiment_id — string (if user in A/B test)
• user_segment — e.g., free/trial/paid (if available)

Sampling rules:

• Full capture for new users (first 7 days after account creation)
• 10% uniform sample for returning users to reduce volume
• Always capture events with step_state=failed (override sampling) to retain failure signals

Privacy & security:

• Do not include PII (email, full name, payment details). Hash user_id with salted HMAC if stored.
• error_code must not leak raw error messages that may contain user data.
• Store elapsed_ms and IDs only; redact any free-text inputs.
• Comply with GDPR/CCPA: honor DNT and deletion requests; provide flag user_consent boolean.

Mapping to analytics & experiments:

• Dashboard metrics:
  • Funnel: onboarding steps ordered by step_id with conversion rate and median elapsed_ms
  • Drop-off: % users with step_state=failed/cancelled per step
  • Failure heatmap: error_code frequency by step
  • Time-to-complete per platform/segment
• Experiment integration:
  • experiment_id property enables slicing conversion and time metrics by variant
  • Use attempt_number and elapsed_ms to detect regressions from experiments
• Alerts:
  • Spike in failure rate (>5% absolute increase vs rolling 7-day baseline) triggers incident notification.

Implementation notes for engineers:

• Emit start and terminal events for each step so elapsed_ms is reliable.
• Use standardized enums and centralized schema registry; update docs when adding step_id values.
• Validate event size and rate; provide backfill plan for new properties.

Hypothesis

• Removing or delaying early monetization prompts (e.g., first 3 sessions) increases short-term retention: users exposed to fewer early paywalls will have higher Day-7 retention than users who see current prompts.

Primary metric

• Day-7 retention (D7) of new users (binary: returned on day 7). Chosen because it’s an early-stable retention signal correlated with lifetime value in free-to-play games.

Segmentation strategy

• Primary cohort: new installs only (first-time users) to avoid prior exposure bias.
• Stratify by acquisition source (organic vs paid), country (top 3 markets), and device OS (iOS/Android) to ensure balance and surface heterogeneous effects.
• Exclude internal test accounts and heavy whales (past payers) if present.

Experiment design & sample size

• Two-arm randomized A/B: Control = current monetization timing; Variant = delayed/removed early prompts.
• Minimum detectable effect: aim to detect +2 percentage points on D7 from baseline p=20% → using standard two-proportion power calc (α=0.05, power=0.8) → ~6,500 new users per arm (≈13k total). Inflate by ~20–30% for attrition, segment splits, and device/country stratification → plan for ~17k–20k total new users. Run until the last enrolled user reaches day 7; ensure temporal balance across week/weekend.

Two guardrail metrics

1. ARPDAU (or 7-day revenue per new user): ensure short-term monetization isn’t unduly harmed.
2. Pay conversion rate (percent of users making any purchase within 7 days).
   (Also monitor crash rate / engagement anomalies as safety checks.)

What constitutes a win

• Statistically significant improvement in D7 retention (p<0.05) in the variant versus control AND no meaningful negative impact on guardrails:
  • ARPDAU change within an acceptable tolerance (e.g., no >5% relative decrease) and
  • Pay conversion not reduced beyond a business-defined threshold (e.g., no >10% relative drop).
• Secondary: positive or neutral signals across key segments (no major market where variant harms retention).

Rollback criteria

• If D7 retention shows no significant improvement or worsens, rollback.
• Immediate rollback if any guardrail is violated beyond thresholds (e.g., ARPDAU drops >5% with p<0.05 or pay conversion drops >10%), or if monitoring flags severe unintended effects (large increase in crashes, major regional regressions).

Analysis & next steps

• Pre-register analysis plan: primary test on overall D7 (two-sided), pre-specified segment analyses, multiple-testing correction for secondary checks.
• If win: consider staged rollout (e.g., 10%→50%→100%) and A/B test variants (timing/different UX approaches) to optimize LTV. If neutral or mixed by segment, iterate on variant design or target specific cohorts (e.g., organic users) rather than global change.

Objective 1: Increase monetization revenue while keeping core engagement stable

• KR1: Grow ARPU from $0.50 to $0.65 by end of Q1 (+30%).
  Why measurable/meaningful: ARPU is direct revenue per user; baseline $0.50 → $0.65 yields incremental revenue of $15k/mo on 100k MAU if sustained (100k * $0.15 = $15k). Ties directly to monetization goal.
• KR2: Increase monthly paying user conversion rate from 1.5% to 2.5% (from 1,500 to 2,500 paying users among 100k MAU).
  Why: Conversion rate is actionable (product + pricing UX). With baseline ARPU, increasing conversion yields predictable revenue lift and shows product-market willingness to pay.
• KR3: Launch two new revenue features (e.g., premium subscription tier + ad-tier targeting) and reach 10% adoption among active users of those flows (measured via feature funnel).
  Why: Measures delivery and initial uptake; links feature launch to behavior change and future revenue streams.

Objective 2: Preserve or improve core engagement while implementing monetization changes

• KR1: Maintain MAU ≥ 95k (no more than 5% drop from 100k baseline).
  Why: MAU is primary engagement guardrail; ensures monetization doesn't cause churn at scale.
• KR2: Keep average weekly time spent per active user within ±5% of baseline (measure via session minutes).
  Why: Time spent reflects product value and ad inventory health; small tolerance signals preservation of experience.
• KR3: Reduce feature-induced churn rate (users who churn within 14 days of encountering a monetization flow) to <3%.
  Why: Targeted churn metric isolates negative impact of monetization changes; actionable for iterating UX.

Rationale summary: These objectives balance top-line revenue targets (ARPU, conversions, new features) with explicit engagement guardrails (MAU, time spent, churn). Each KR is numeric, tied to baseline, and metrics are instrumentable in analytics tools for quarterly tracking.

Situation: I owned a 5-product portfolio with ad‑supported, subscription, and transaction-based monetization. The company needed one cross‑product North Star to align leadership without hiding product signals.

Approach / Proposal:

• Composite North Star: Portfolio Value Delivered (PVD) = Σi (wi × Si)
  • Si = product i’s normalized value signal (captures monetization + engagement)
  • wi = strategic weight for product i

Decomposition (keep product-level signals visible):

• Define Si as a vectorized score: Si = α·RevenueGrowthIndex + β·EngagementIndex + γ·RetentionIndex − δ·CostPerAcqIndex. Each index is normalized (0–1) and tracked separately on dashboards. PVD is the single alignment metric; teams still own and report the vector components weekly.
• Publish a “PVD breakdown” heatmap that shows contribution of each product and of each component (revenue, engagement, retention, cost).

Governance:

• Metric owners: Portfolio PM owns PVD definition; each product PM owns its Si components and data quality.
• Change control: Any change to indices, weights or normalization requires ROI justification, stakeholder sign‑off, and a 60‑day parallel run.
• Cadence & incentives: Use PVD for quarterly strategic prioritization only; maintain product-level KPIs for sprint-level incentives to avoid gaming.

Prevent a single product dominating:

• Use normalization and weighting techniques so scale doesn’t swamp contribution.

Two weighting/normalization approaches:

1. Normalization-first, then equalize (fair contribution)

   • Normalize each product signal with per-product min–max or z-score over a rolling 12‑month window to remove raw scale.
   • Set wi = 1/N (or slight strategic tilt). This makes a small but high-growth product visible.
   • Example: If Product A revenue = $10M and Product B = $0.5M, z‑scoring brings their growth signals onto comparable footing.

2. Strategic-weight with capped share (business-priority)

   • Set wi proportional to strategic importance (e.g., ARR, TAM potential) but cap any product’s weight at 30% to prevent domination.
   • For scale-heavy products, apply log normalization to revenue before weighting: scaled_rev = log(1 + revenue) to compress extremes.

Additional guardrails:

• Monitor contribution drift: flag if any product’s share > cap for 2 quarters.
• Run sensitivity analysis quarterly to validate weights.
• Keep product-level OKRs separate — PVD guides portfolio tradeoffs; product KPIs drive execution.

Why this works:

• Preserves product-level signals through explicit decomposition.
• Provides a single metric for cross-functional alignment.
• Normalization and caps ensure fairness and prevent large products from masking smaller but strategic wins.