Estimation and Capacity Sizing Questions

Approach (high-level): use logged production impression data + inverse propensity scoring (IPS) to estimate how the new ranking would change CTR and downstream revenue without launching. Then simulate latency impacts and reason about trade-offs before an online experiment.

Offline estimation steps:

1. Collect logged impressions: for each impression i record context x_i, shown item a_i, position k_i, click y_i (0/1), price/rev r_i, and logging policy propensity p_log(a_i | x_i).
2. Compute new-policy propensity (up to re-ranking) p_new(a | x). For deterministic re-rankers, use a small randomized logging policy or model a stochastic ranking to get non-zero p_new.
3. IPS estimate for CTR or revenue:
   • CTR_new ≈ (Σ_i y_i * w_i) / (Σ_i w_i) where w_i = p_new(a_i|x_i)/p_log(a_i|x_i).
   • Revenue_new ≈ (Σ_i r_i * y_i * w_i) / (Σ_i w_i) or estimate expected value per impression likewise.
   • Use self-normalized IPS to reduce variance: divide by Σ_i w_i.
4. Variance & confidence: bootstrap the logged dataset or compute variance of IPS; report CIs.
5. Position-bias correction: if propensity factors include position, ensure p_log encodes position probability; alternatively use click models (PBM/DBN) to estimate position bias and derive unbiased estimates.
6. Sensitivity checks: stratify by query type, device, position; run covariate shift checks (are contexts in logs representative of expected traffic?).

Simulate latency effects:

• Model the 30ms increase across flows; measure how latency correlates with click probability historically (compute marginal effect of latency on CTR by buckets).
• Adjust CTR_new downward by estimated latency elasticity; or run a replay that injects latency in simulated client-side metrics.
• Consider revenue-per-click vs abandonment: longer latency may reduce overall impressions (higher abandonment) especially on mobile/low-bandwidth.

Trade-offs and caveats before online A/B:

• IPS needs good overlap: if p_new/p_log extreme, IPS variance explodes—consider Doubly Robust estimators or collect exploration data.
• Logging policy quality: deterministic logs can't support re-ranking evaluation reliably; you may need controlled randomized exploration (top-k swap or Plackett-Luce logging).
• Position bias mis-specification: wrong click model yields biased estimates.
• Latency distribution: average +30ms may hide worse tail increases; tail latency impacts UX more and can disproportionately reduce CTR.
• User segments: latency-sensitive cohorts (mobile, low-end devices, new users) may react differently—stratify or isolate in experiment.
• Business metrics beyond CTR: session length, conversion, revenue per user, ad fairness, and downstream costs (infrastructure CPU/ops).
• Safety and SLAs: if increased latency risks SLA breaches, gate rollout or use conservative canary with traffic caps and monitoring.

Recommended next steps:

• Run IPS + DR estimation and bootstrap CIs; simulate latency-adjusted CTR using historical latency–CTR elasticity.
• If IPS variance high, run short randomized exploration (e.g., 1–2% traffic with randomized top-k swaps) to collect proper propensities.
• Start a small canary A/B with strict SLOs, monitor CTR, revenue, latency tail, abandonment, and user engagement; ramp based on metrics and rollback thresholds.

Calculate expected uplift in absolute retained users (probability × relative uplift × users):

• Feature A: 0.10 × 0.20 × 1,000,000 = 20,000 users (absolute +2.0% of base).
• Feature B: 0.50 × 0.05 × 1,000,000 = 25,000 users (absolute +2.5% of base).

Recommendation: Prioritize Feature B — it has higher expected retention uplift (25k vs 20k) and lower variance (more reliable outcome).

Qualitative factors I'd weigh before final decision:

• Cost & time to build: if A is much cheaper/faster, its risk-adjusted ROI might be preferable.
• Downside/upside profile: A has a bigger upside if it succeeds (20% uplift), which matters if we can afford high-risk bets.
• Learnings & technical dependencies: which yields reusable platform improvements or insights for future features?
• Targeting & segmentation: can we expose A to a high-likelihood segment to raise success probability?
• Measurement readiness: ensure instrumentation and A/B test power to detect these effects.
• Strategic fit: alignment with roadmap, brand, or long-term retention initiatives.

If engineering cost and strategic fit are similar, roll out B first for reliable impact while optionally running a targeted experiment for A to validate its higher upside.

Estimate: ~8,000–9,000 users per variant (order of magnitude: single-digit thousands).

Intuition and formula (plain language):

• Detecting a 1 percentage-point absolute lift from 5% → 6% is hard because conversion outcomes are binary and rare; variance = p(1−p) is relatively large compared to the small effect.
• The standard frequentist sample-size for two proportions (per arm) is roughly:
  n ≈ [ (Zα/2·√(2p̄(1−p̄)) + Zβ·√(p1(1−p1)+p2(1−p2)))² ] / Δ²
  where p1=0.05, p2=0.06, p̄≈0.055, Δ=0.01, Zα/2≈1.96 (5% two-sided), Zβ≈0.84 (80% power). Plugging in gives ~8.1k per arm.
• Intuition: numerator captures required multiples of standard error to distinguish signal from noise; denominator Δ² shows that halving Δ (smaller effect) quadruples required N.

Pragmatic ways to reduce required sample size:

• Increase metric sensitivity: use a more frequent or continuous metric (e.g., click rate, revenue per user) or composite metric with lower variance.
• Covariate adjustment / regression: adjust for known predictors (user cohort, past behavior) to reduce residual variance.
• Restrict population: test on higher-conversion segments where relative lift is larger or variance lower.
• Use one-sided test if direction is known and acceptable (reduces Zα).
• Accept lower power or larger MDE (trade-offs with business needs).
• Sequential / adaptive designs (group sequential, Bayesian A/B): stop early if strong evidence; can reduce average sample size but needs proper correction.
• Increase effect size via bolder treatment or improved targeting.

As PM, balance statistical rigor with business urgency: pick an acceptable MDE, consider stratification and metric choice first—those often give biggest practical gains without compromising validity.

A p-value is a number that helps you judge how surprising your A/B test result would be if there were actually no difference between variants. In plain terms: it tells you the probability of seeing data at least as extreme as yours assuming the null hypothesis (no effect) is true. Statistical significance means the p-value is below a pre-set threshold (commonly 0.05), so the observed difference is unlikely to be due to random chance alone.

Why this matters for A/B tests:

• It gives a systematic way to decide whether an observed lift is likely real or just noise.
• It helps avoid acting on random fluctuations, so you make more reliable product decisions.

One important limitation to watch: statistical significance ≠ business importance. A very small effect can be statistically significant with large samples but irrelevant to revenue or UX. Always check the effect size and its practical impact (confidence intervals, ROI) before shipping changes.

Assumptions (stated):

• Monthly Active Users (MAU) = 10,000,000.
• "Average 30 sessions/month per active" → each active opens the app 30 times/month.
• "10 requests per session" → each session issues 10 requests to the messaging feature (could be API calls, pings, sends, etc.).
• 95th-percentile concurrency = 0.5% of MAU (users concurrently active at peak 95th percentile).
• Uniform request distribution across concurrent users during peak second (conservative).
• 30 days/month, 24 hours/day; peak occurs during a short high-usage window (we use concurrency to capture that).

Compute baseline monthly -> peak:

1. Total sessions/month = MAU * 30 = 10,000,000 * 30 = 300,000,000 sessions/month.
2. Total requests/month = sessions * 10 = 3,000,000,000 requests/month.
3. Average requests/second across month = 3e9 / (30243600 ≈ 2,592,000) ≈ 1,157 RPS.
   But peak is driven by concurrent users:
4. Concurrent users at 95th = MAU * 0.005 = 50,000 users concurrently.
5. If each concurrent user issues 1 request in that second on average — but given 10 requests/session and session lasts, say, 5 minutes (300s), per-user request rate during session = 10/300 ≈ 0.033 req/s → per concurrent user.
6. Peak RPS = concurrent_users * per-user peak rate = 50,000 * 0.033 ≈ 1,650 RPS.

Alternative simpler calc: if we assume those 10 requests happen burstily within 30s, per-user rate=10/30≈0.33 → peak RPS = 50,000*0.33 ≈ 16,500 RPS. So expected peak RPS range: ~1.6k — 16.5k depending on session duration/burstiness. I’d plan capacity for the higher bound with safety factor (x2) → ~33k RPS.

Validation plan with telemetry:

• Instrument request counters, latencies, and user-session markers; emit per-instance RPS and 1s/1m percentiles.
• Track active sessions and concurrent users (heartbeat or websocket count) to compare calculated concurrency.
• Create dashboards: global RPS, regional RPS, P95/P99 latency, error rates.
• Capture request origin, session-id, user-id to validate assumptions (requests per session, session duration, burst patterns).
• Run load tests modelling observed session duration/burstiness, then compare telemetry to predict scaling rules (autoscaling thresholds, rate limiting).
• Regularly recompute estimates from live metrics and adjust capacity and SLOs.