Business Metrics and Unit Economics Questions

Situation: We ingest driver app logs, payment records, and ride events to compute rides/day and revenue/day. As PM I’d define and prioritize robust DQ checks, owned SLAs, and runbooks so metrics stay trustworthy.

Six checks and remediation:

1. Ingestion completeness — daily row count vs expected (e.g., previous 7-day median ± threshold).
   Action: Alert pipeline team; switch to fallback dataset window (last known good) and tag metric as “partial” until fixed.

2. Schema/format validation — required fields (ride_id, driver_id, timestamp, amount) present and types correct.
   Action: Reject bad rows to quarantine table; notify owners with sample bad rows; backfill after fix.

3. Referential integrity — ride events joinable to payment or driver records (foreign key match rate).
   Action: If match rate drops, pause new downstream reports, trigger investigation (late arrivals vs data loss); apply reconciliation logic (e.g., mark unmatched rides pending).

4. Timestamp consistency — event timestamps within plausible bounds and timezone-normalized; monotonicity for lifecycle events.
   Action: If timestamps out of range, mark affected window, rerun normalization, or exclude corrupted events with annotation.

5. Aggregate reconciliation — compare derived daily metrics to independent sources (payment gateway totals, financial ledger) within tolerance.
   Action: If divergence > threshold, initiate cross-team financial reconciliation and block stakeholder-facing dashboards until resolved.

6. Duplication and idempotency — detect duplicate ride_ids or payments (high duplicate ratio).
   Action: Deduplicate at query layer; root-cause pipeline idempotency bug; prioritize fix and reprocess.

For failures: enforce severity-based SLA (auto-alert → on-call → incident), communicate impact to stakeholders (dashboard health/status), provide temporary mitigations (annotations, partial data, estimates), and track postmortem + permanent fixes prioritized on roadmap.

Situation: Investor asked for a 3‑year ARR forecast. I build simple ARR roll-forward each year:
ARR_next = ARR_start * GrossRetention + Expansion% * ARR_start + NewBookings

Base assumptions:

• Gross retention = 85%
• Expansion = 5% of starting ARR each year
• New bookings Y1 = $6.0M, then +20% YoY

Numbers (millions):

Year0 (current): ARR0 = 12.00

Base:
Year1: retained = 12.000.85 = 10.20
expansion = 12.000.05 = 0.60
new = 6.00
ARR1 = 10.20+0.60+6.00 = 16.80
Year2: retained = 16.800.85 = 14.28
expansion = 16.800.05 = 0.84
new = 6.001.20 = 7.20
ARR2 = 14.28+0.84+7.20 = 22.32
Year3: retained = 22.320.85 = 18.972
expansion = 22.320.05 = 1.116
new = 7.201.20 = 8.64
ARR3 = 18.972+1.116+8.64 = 28.728 (~28.73)

Upside (more optimistic): retention 90%, expansion 7%, new bookings growth +30% YoY
ARR1 ≈ 17.64; ARR2 ≈ 24.91; ARR3 ≈ 34.30

Downside (pessimistic): retention 80%, expansion 3%, new bookings grow only +10% YoY
ARR1 ≈ 15.96; ARR2 ≈ 19.85; ARR3 ≈ 23.73

Key sensitivities to highlight to investor:

• Gross retention (churn): small % changes materially shift multi‑year ARR.
• New bookings pace and ramp timing (front‑loaded vs. back‑loaded).
• Expansion % (upsell/expansion motions effectiveness).
• Average contract value / pricing changes and churn concentration in large accounts.
• Sales efficiency (CAC, time-to-close) affecting sustainable new bookings.

Recommendation: present Base as central case, show sensitivity table (±5pp retention, ±5pp expansion, ±10pp new bookings growth) and run waterfall charts to explain drivers.

Customer Acquisition Cost (CAC): the average fully loaded cost to acquire one paying customer over a defined period. Include marketing spend, sales salaries & commissions, creative/agency costs, and attribution-adjusted channel expenses, divided by new customers acquired in that period.

Customer Lifetime Value (LTV): the present-value expected gross profit from a customer over their life with you. Typically computed as (Average Revenue Per Account × Gross Margin) ÷ Churn Rate, or sum of discounted future contribution margins.

Why the LTV:CAC ratio matters: it shows unit-economics sustainability. If LTV >> CAC, you can scale profitably; if LTV ≈ CAC or < CAC, growth burns cash or loses money. A common rule of thumb: target LTV:CAC ≥ 3:1 (2:1 signals marginal efficiency; >5:1 may indicate under-investment in growth). Also monitor payback period — ideally customer acquisition costs pay back within 12 months for SaaS.

Common pitfalls:

• CAC: omitting sales overhead, misattributing multi-touch channel costs, using bookings instead of new customers.
• LTV: ignoring churn heterogeneity, using revenue instead of gross margin, failing to discount future cash flows, and conflating cohort LTV with company-wide averages.
  State period, cohort, margin and discount assumptions clearly in any brief.

Requirements & constraints:

• Executive metrics must be accurate, reproducible, auditable, and updatable with minimal disruption. Support real-time/near‑real-time dashboards and monthly/quarterly official reports. Everyone must use the same canonical definition.

High-level design:

• Single Source of Truth (SSOT) metrics layer: a versioned metrics catalog + implementation (SQL/transform code) deployed from CI to prod datasets. Lineage metadata stored in a catalog (e.g., dbt + data catalog).
• Automated validation pipelines that run on every build and on schedule: unit tests for logic, integration checks vs. raw source, anomaly detection, and reconciliation jobs writing validation reports.

Key components & responsibilities:

• Metrics Owner (Product Manager): defines business logic, approves changes, prioritizes metrics backlog.
• Data Steward (per domain): maintains source mappings, SLA for fixes.
• Data Engineer: implements transforms, tests, deployments.
• BI Analyst: builds dashboards against SSOT, runs ad-hoc reconciliations.
• Exec Sponsor / Governance Council: approves policy changes, resolves disputes.
• SRE/Monitoring: maintain pipeline availability.

Change-management & reproducibility:

• All metric definitions are versioned (semantic versions) and stored in the catalog. Changes require:
  1. Proposal doc: reason, business impact, sample data.
  2. Automated test suite updated (unit + integration + golden-file reconciliation).
  3. Review: Data Steward + Metrics Owner sign off.
  4. Staging deploy and smoke tests (including historical backfill validation).
  5. Freeze window for official reports; rollouts annotated with release notes.
• Backwards-compatibility: support old and new metric versions concurrently for at least one reporting cycle; tag “official” for reporting cutoffs.

SLAs & monitoring:

• Data freshness SLA: e.g., nightly ETL completes by 4:00am UTC; latency < 2 hours for near‑real‑time metrics.
• Accuracy SLA: reconciliation discrepancy < 0.5% for revenue; larger discrepancies trigger P1 incident.
• Availability SLA: dashboards 99.9% during business hours.
• Incident SLAs: detect within 1 hour, owner ACK within 2 hours, remediation plan within 8 hours.

Validation & automated checks:

• Unit tests covering edge cases (refunds, currency).
• Reconciliation vs. source of truth (finance ledger) with tolerances and historical diffs.
• Anomaly detection (statistical and rule-based) with auto-alerts.
• Data lineage checks to ensure dependencies intact.

Governance rituals:

• Weekly metrics triage for incidents and change requests.
• Monthly metric review with stakeholders to retire/rename metrics.
• Quarterly audit: sample-based verification against finance and legal.

Example: Revenue metric policy

• Definition: Recognized revenue = sum(invoice_line.amount) where invoice.status = 'paid' and recognition_date within period; exclude refunds captured by refund_events.
• Implementation: dbt model revenue_v1 (v1.0), tests for duplicates, negative amounts, currency normalization.
• Change workflow: product requests change → update doc & tests → staging run shows historical diffs → Data Steward + PM approve → deploy v2.0 and annotate dashboards; old v1 tagged archived.

Outcome:

• Clear ownership, automated validation, versioned definitions, and SLAs reduce ambiguity and ensure executive reports are trustworthy and reproducible.

Objective: test whether a 10% increase in base fares in City Y reduces driver cancellations without materially reducing ride volume or hurting revenue/customer satisfaction.

Primary metric:

• Cancellation rate per completed trip request (percent). (Goal: statistically significant reduction)

Secondary metrics (impact / guardrails):

• Rides per active rider per week (elasticity signal)
• Gross bookings / revenue per ride and per user
• Conversion rate: accepted requests / requests
• Trip completion rate (including cancellations + no-shows)
• Customer NPS / complaints and driver earnings metrics
• Retention: 7- and 30-day active users

Experiment design:

• Randomized controlled A/B at user or request level (prefer user-level to avoid cross-treatment leakage). Treatment: +10% base fare for trips in City Y. Control: status quo.
• Exclude new users in first 7 days, extremely infrequent users, and surge periods for initial sanity checks.

Sample size considerations (qualitative):

• Power to detect a small absolute change in cancellation rate (e.g., 1–2 percentage points) requires larger N. Use historical baseline cancellation rate, variability, and desired MDE to compute N with standard power formulas. If cancellations are rare, need more users/longer time.
• Ensure balance across geography, time of day, rider segments (business vs casual).

Length of experiment:

• Minimum 2–4 weeks to cover weekly cycles; preferable 4–8 weeks to capture behavior and retention effects and weekday/weekend patterns. Monitor early signals but avoid stopping early for small p-values.

Biases and controls:

• Time-of-day and day-of-week effects: stratified randomization or include as covariates.
• Seasonality or promotions overlapping the test: pause overlapping marketing or exclude promo-triggered trips.
• Spillover: avoid city-level driver pricing changes that let drivers learn and change behavior across cohorts.
• Selection bias: ensure assignment is consistent across devices/accounts to avoid re-randomization.
• Confounders: monitor supply-side changes (driver incentives, fleet size), competitor pricing changes, and external events.

Analysis plan:

• Pre-specify primary metric, MDE, significance (alpha 0.05) and use ITT analysis. Report confidence intervals, uplift, and heterogeneous effects by segment (high-frequency riders, time of day).
• Safety checks: revenue per trip, retention, complaints. If any guardrail deteriorates beyond predefined thresholds, abort or roll back.