Estimation and Capacity Sizing Questions

Situation / assumptions: Annual gross volume (GMV) = $50,000,000. Average transaction = $2 → total transactions/year = 50,000,000 / 2 = 25,000,000 tx.

1. Fees & fraud calculation (annual)

• Processor fee = 2.5% + $0.10 per tx
  • Percentage portion = 2.5% of $50,000,000 = $1,250,000
  • Per-transaction portion = $0.10 * 25,000,000 = $2,500,000
  • Total processor fees = $3,750,000
• Fraud losses = 0.5% of volume = 0.005 * $50,000,000 = $250,000
• Net revenue after fees & fraud = GMV - fees - fraud = $50,000,000 - $3,750,000 - $250,000 = $46,000,000

So annual net = $46M (this is gross of any taxes, refunds, or other costs).

2. Peak transactions per second (TPS)

• Daily volume = $50,000,000 / 365 ≈ $136,986/day → transactions/day ≈ 25,000,000 / 365 ≈ 68,493 tx/day.
• 20% of daily volume in one hour → tx in peak hour = 0.20 * 68,493 ≈ 13,699 tx in 1 hour.
• TPS during that hour = 13,699 / 3600 ≈ 3.81 ≈ ~4 TPS.
• For safety/engineering headroom, plan for higher concurrency (e.g., 10x): design for ~40 TPS and burst capacity up to 100 TPS.

3. Fraud mitigation recommendations to preserve unit economics

• Reduce fraud rate (0.5%) which is $250k/year; even halving it saves $125k.
  • Implement device & behavioral fingerprinting + ML risk scoring at authorization to decline high-risk payments.
  • Use rules engine + adaptive thresholds (amount, velocity, geo, BIN) to block suspicious flows.
  • Require step-up authentication (3DS2) for medium-risk transactions to shift liability and reduce chargebacks.
  • Push tokenization and card-on-file to reduce friction and fraud on repeat customers.
  • Monitor and rapidly iterate using labeled chargeback data; A/B test rules to balance false positives vs losses.
  • Work with processor for optimized pricing (volume discounts) and liability shifts; negotiate lower per-transaction fees or interchange-plus model.
  • Set up real-time dashboards and alerts, and a small manual review team for high-risk / high-value segments.

Takeaway: With current assumptions net ≈ $46M. Peak TPS ~4, but design with 10–25× headroom. Small reductions in fraud or fee negotiation materially improve unit economics.

I’d break the problem into a small set of multiplicative components, identify data to populate each, then prioritize uncertainties to resolve.

Decomposition:

• Monthly Active Users (MAU) for the product / cohort exposed to the feature
• Sessions per user per month (or DAU→MAU conversion)
• Fraction of sessions where the feature is available/visible (exposure rate)
• Impressions per exposed session (how many ad slots shown when feature is used)
• Fill rate (ads actually served) and viewability adjustments (visible impressions)
  Estimate = MAU × sessions/user × exposure rate × impressions/session × fill rate × viewability %

Data sources I’d seek:

• Analytics (segment MAU, sessions) — Mixpanel/Amplitude/GA
• Feature-flag rollout plan and UI telemetry (exposure)
• A/B test or pilot telemetry for impressions/session and engagement
• Ad platform logs for fill rate and viewability benchmarks
• Competitor benchmarks and industry CPM reports for sanity checks
• Sales/monetization inputs for reserved inventory

Uncertainty resolution order:

1. MAU / target audience size (largest multiplier; start here)
2. Exposure rate (depends on rollout + UI placement)
3. Impressions per exposed session (empirical from pilot/A/B)
4. Fill rate & viewability (lower impact but refines final number)
   Use ranges and scenario modeling (conservative/likely/optimistic) and surface assumptions to stakeholders. Run a small pilot if key drivers are uncertain — it’s the fastest way to collapse variance and produce a defensible estimate.

A p-value is a number that helps you judge how surprising your A/B test result would be if there were actually no difference between variants. In plain terms: it tells you the probability of seeing data at least as extreme as yours assuming the null hypothesis (no effect) is true. Statistical significance means the p-value is below a pre-set threshold (commonly 0.05), so the observed difference is unlikely to be due to random chance alone.

Why this matters for A/B tests:

• It gives a systematic way to decide whether an observed lift is likely real or just noise.
• It helps avoid acting on random fluctuations, so you make more reliable product decisions.

One important limitation to watch: statistical significance ≠ business importance. A very small effect can be statistically significant with large samples but irrelevant to revenue or UX. Always check the effect size and its practical impact (confidence intervals, ROI) before shipping changes.

Back-of-the-envelope (BOE) calculations are quick, high-level estimates using simplified assumptions and arithmetic to get an order-of-magnitude sense (e.g., addressable users × conversion rate × ARPU). Detailed statistical forecasting uses historical data, validated models (time series, causal regressions), confidence intervals, seasonality, and scenario analysis to produce probabilistic predictions and drill into drivers and uncertainty.

When to choose:

• BOE: early-stage decisions, rapid prioritization, scoping, or when data is scarce and speed matters.
• Full forecast: investment decisions, budget requests, launch sizing, supply chain planning, or when regulatory/financial accuracy is required.

Example — BOE appropriate:

• Deciding whether to prototype a new onboarding flow. I’d estimate incremental conversions: 100k MAUs × 10% reachable × 5% lift × $2 ARPU ≈ $10k/month — enough to justify an experiment. Fast, low-cost, informs prioritization.

Example — Detailed forecast necessary:

• Predicting revenue for a new paid tier to set annual targets and hiring. Requires cohort analysis, churn modeling, pricing elasticity tests, churn/upgrade funnels, and scenario ranges.

Consequences of the wrong approach:

• Using BOE when a detailed forecast was needed risks under/over-hiring, cash shortfalls, or wrong pricing decisions. Using an over-precise forecast too early wastes time and can ossify decisions on shaky assumptions. Good PMs match the method to decision risk, document assumptions, and iterate (BOE → refined forecast as data arrives).

Situation & goal: detect a 10% relative uplift in 30‑day retention from 30% → 33% (absolute +3pp) while accounting for right‑censoring (users who haven’t reached day‑30 by analysis) and early churn.

Power/sample-size approach (survival-aware):

1. Translate retention target to a hazard ratio under an exponential survival assumption:
   • S0(30)=0.30, S1(30)=0.33 → HR ≈ ln(1/S1)/ln(1/S0) ≈ 0.92 (treatment reduces hazard by ~8%).
2. Use Schoenfeld formula for events required for a log‑rank/Cox test:
   • D = (Zα/2 + Zβ)^2 / (ln HR)^2.
   • For 80% power, α=0.05: (1.96+0.84)^2 ≈ 7.84. ln(0.92)≈−0.0834 → D ≈ 7.84 / 0.00696 ≈ 1,126 events.
3. Convert events → total N using expected event rate (churn by day 30). Baseline event rate = 1−S0 = 0.70. If average event rate across arms ≈0.685, N ≈ D / 0.685 ≈ 1,644 users (total).
4. Adjust for censoring/attrition: if fraction c of users are censored before 30d (e.g., due to test cut‑off), divide by (1−c). For c=20% → N ≈ 1,644 / 0.8 ≈ 2,055 total users. Also inflate slightly for noncompliance or instrumentation loss (e.g., +10%).

Practical experiment steps:

• Pre-specify primary analysis as log‑rank test / Cox PH on time-to-churn up to 30d; report KM estimates and median times. That uses partial information from censored users.
• Simulate expected accrual and censoring (bootstrap or parametric) to validate sample and duration estimates under realistic arrival and censoring patterns.
• Use leading indicators to speed learning:
  • 7‑day retention or activation metrics (shorter horizons) as secondary outcomes—validate they correlate with 30d in historical data before using as proxies.
  • Intermediate survival curves (e.g., KM at 7/14 days) to check early signals; be cautious with peeking—use sequential testing (alpha spending / O’Brien‑Fleming) or pre‑registered stopping rules.
• Randomize and stratify by major covariates (platform, country, power users) to reduce variance; include those covariates in Cox model to increase power.
• Instrumentation: ensure events (first churn, last activity) and enrollment timestamps are reliable; log reasons for censoring.
• Monitoring & rollout: run the experiment until required events achieved (event-driven stop). If early leading indicators are strong and robust to multiple testing corrections, consider phased rollout.

Key tradeoffs:

• Assumptions (exponential, proportional hazards) simplify computation; verify PH assumption post‑hoc and report KM differences if violated.
• Large required events for small relative improvements—if impractical, consider targeting larger MDE or using stronger leading indicators validated historically.

This plan balances correct handling of censoring with pragmatic steps (simulations, leading indicators, sequential tests) so you reach a robust decision without unnecessary duration.