Company Research and Business Understanding Questions

1. Clarify goal & KPI

• Define retention precisely (e.g., 30-day active user retention rate = users with >1 session in 0–7d who return in day 30).
• Define business threshold for “impact” (e.g., +2 ppt lift in 30d retention within 95% CI).

2. Set baseline

• Pull 6–12 weeks historical retention to estimate mean, variance, seasonality, churn cohorts.
• Compute cohort retention curves, weekly active users, standard error, and minimum detectable effect (MDE) given current sample size.
• Example metrics: cohort size, baseline retention pct, SEM, MDE at α=0.05, power=0.8.

3. Choose causal approach (map to initiative)

• A/B test: preferred for product changes with randomizable treatment (e.g., new onboarding). Randomize at user or cluster level, pre-specify primary/secondary metrics, run for at least 1.5–2x max retention window.
• Difference-in-differences (DiD): use when rollout is phased (regional or by account) and parallel trends assumption holds. Validate pre-trend similarity.
• Propensity score matching (PSM): observational where treatment not random; match on pre-treatment engagement, demographics; use as robustness check, not primary causal claim.
• Use two-stage approach: run RCTs when feasible; otherwise DiD + PSM + regression adjustments with controls and time fixed effects.

4. Instrumentation & data requirements

• Event schema: consistent user_id, timestamp, event_type (session_start, purchase, churn_flag), cohort_tag, treatment_flag, assignment_timestamp.
• Logging: exposure logs for experiments, randomization buckets, rollouts, and feature flags; ensure deduplication and idempotency.
• Data quality: automated checks (missing IDs, time-travel, retention calc tests).
• Tracking upstream (ad spend, campaign IDs) to control confounders.

5. Analysis plan & stats

• Pre-register hypotheses, primary metric, sample size, analysis window, and stopping rules.
• Use ITT (intent-to-treat) estimates, compute lift, confidence intervals, p-values, and Bayesian credible intervals as complementary.
• Pre-period balance checks, covariate adjustments to increase power.
• Sensitivity analyses: heterogeneous effects by cohort, churn modeling, survival analysis (Kaplan–Meier, Cox proportional hazards).

6. Reporting cadence & visuals for executives

• Cadence: Weekly operational dashboard during experiment; Executive summary at experiment end + monthly impact review.
• Visuals:
  • Executive one-pager: baseline vs. post-intervention retention lift (bar with CI), MDE vs. observed lift, NPV/revenue impact estimate.
  • Cohort retention curves (line chart) comparing treatment/control.
  • Waterfall: pathway from feature -> engagement lift -> retention -> revenue.
  • Funnel/segmentation heatmap: where retention gains concentrated.
  • Statistical appendix: p-values, SEM, sample sizes, assumptions, robustness checks.
• Delivery: interactive dashboard (Looker/Tableau) with filters by cohort/segment plus PDF executive memo summarizing conclusions and recommended next steps.

7. Operationalize learning

• If significant lift: estimate ARR impact, propose rollout plan with monitoring.
• If null: surface diagnostics (power, exposure, uptake), run post-hoc subgroup analyses, iterate.

This plan ensures rigorous causal claims, reproducible instrumentation, and executive-ready visuals linking BI work to concrete retention improvements.

TAM, SAM, SOM in plain terms:

• TAM (Total Addressable Market): the total revenue opportunity if every potential customer worldwide bought your product — the big ceiling.
• SAM (Serviceable Addressable Market): the portion of TAM you can serve given your product features, geography, and channels today.
• SOM (Serviceable Obtainable Market): the realistic share of SAM you can capture in the near term (12–36 months) given competition, capacity, and go-to-market constraints.

Recommended BI dashboard: "Market Penetration Funnel"

• Top row: TAM, SAM, SOM tiles with numeric values and % of TAM (large font).
• Middle: Actuals — current users/customers and revenue mapped against SOM and SAM (bar gauges showing penetration: customers captured / SOM).
• Time series: cumulative customers and revenue vs. target curves for SOM and SAM.
• Cohort/segment filters: region, industry, product tier to see penetration by segment.
• Map visualization: geographic heatmap showing penetration percent of SOM by region.
• Action panel: alert flags where penetration is below target and suggested levers (pricing, sales coverage).

External datasets to use:

• Industry market reports (Gartner, IDC, Forrester) for TAM/SAM estimates
• Public financials and market share data of competitors (SEC filings, Crunchbase)
• Census / demographic data (country/region population, business counts)
• App store or web analytics benchmarks (Sensor Tower, SimilarWeb) for category adoption
• Third-party firmographic datasets (D&B Hoovers, Clearbit) to size addressable customers by segment
• Macroeconomic datasets (World Bank, OECD) for normalization and growth trends

Why this helps: it ties high-level market sizing to real-time product performance, highlights gaps between potential and actual, and gives stakeholders clear levers and regions to prioritize.

Step-by-step research plan:

1. Clarify scope: which product lines / regions the BI role will support.
2. Quick company overview: website, About, mission, market positioning.
3. Product & customers: read product pages, pricing, case studies to map product lines and target segments.
4. Financial & strategy: review investor deck, annual/quarterly reports, press releases for growth targets and KPIs.
5. Product details & roadmap signals: product docs, developer/API docs, engineering blog, GitHub for features and technical constraints.
6. Market & competitors: analyst reports, Crunchbase, news articles to understand TAM, competitors, recent funding/acquisitions.
7. Culture & org: LinkedIn org chart to identify stakeholders; Glassdoor for team structure and pain points.
8. Internal sources (if available): OKRs, prior dashboards, data dictionary, BI tickets, SQL queries, stakeholder interviews.
9. Synthesize: map top metrics (ARR, MRR, activation, churn, LTV:CAC, NPS), data gaps, and opportunities for dashboards.

Public and internal sources to consult:

• Public: investor deck, SEC filings, product pages, case studies, engineering blog, press releases, Crunchbase, LinkedIn, Glassdoor, industry reports.
• Internal: data dictionary, BI backlog, analytics events tracking, GA/segment/heap, Redshift/BigQuery schemas, existing dashboards, stakeholder roadmaps, OKRs.

Signals I look for:

• Growth levers: pricing changes, new channels, partnerships, enterprise vs SMB mix.
• Churn reasons: product fit, onboarding drop-offs, support tickets, cancellation cohorts.
• Top customers & revenue concentration: account list, logo churn, ARR per cohort.
• Product usage patterns: DAU/MAU, feature adoption, time-to-first-value, funnel conversion.
• Data quality risks: missing events, late-arriving data, inconsistent IDs.

Three targeted interviewer questions:

1. Which top-level metrics does leadership prioritize this quarter, and where do you see the biggest blind spots in reporting today?
2. How mature is your event tracking/data model (e.g., product events, user IDs) — any known gaps the BI team must work around?
3. Who are the primary stakeholders for BI deliverables (product, sales, customer success), and how are requests prioritized against strategic OKRs?

Requirements & constraints:

• Allow analytics across events and cohorts without sharing direct personal identifiers (PII) between third-party analytics tools.
• Preserve ability to track user journeys, funnels, retention, and segment-level KPIs while minimizing re-id risk.
• Comply with law (no reversible identifier sharing), scalable, auditable.

High-level approach:

1. Data minimization & classification

• Only ingest attributes needed for a report; tag data as PII/sensitive in the ETL catalog.
• Apply retention limits and purpose-based access.

2. Technical controls (combined)

• Tokenization (pseudonymous IDs): Replace raw PII with provider-specific stable tokens generated in a central, secure service. Tokens are non-reversible and scoped per tool (per-tool salt/pepper) to prevent cross-tool linkage.
• Hashing with salts: Use keyed hash (HMAC) with per-target key stored in a KMS. Rotate keys regularly. Avoid raw unsalted hashing.
• Differential privacy: For aggregated reports (counts, sums, histograms), enforce DP mechanisms (Laplace/Gaussian noise) at query layer with an enforced privacy budget per dataset and per analyst.
• Synthetic data: For model training or exploratory analysis where individual-level structure is needed but PII cannot be used, provide vetted synthetic datasets generated to preserve statistical properties while preventing re-id.
• Query-time access controls: Anonymization library in BI connector that applies transformations (aggregation thresholds, k-anonymity checks) before results leave warehouse.

3. Policy & governance

• Purpose & role-based access: RBAC and just-in-time approvals for sensitive segments.
• Data use agreements and attestation for third parties.
• Auditing & logging of all tokenization, key-usage, and DP budget consumption.
• Incident response and re-identification testing program (regular privacy risk assessments, red-team).

Trade-offs to BI fidelity and mitigations:

• Loss of cross-tool user-level stitching if tokens are tool-scoped -> mitigated by creating an internal analytics-only stable token under strict governance for aggregate longitudinal analysis (never shared externally).
• DP noise reduces small-cohort accuracy -> mitigate by: increasing aggregation windows, raising k thresholds, using precomputed private aggregates with tuned epsilon per business need, and giving analysts access to "noisy but explainable" error bounds.
• Synthetic data may not capture rare event correlations -> use hybrid workflows: synth for modeling baseline, then run limited, audited queries on pseudonymized real data for validation.
• Key rotation and per-tool salts complicate joins -> central token service exposing controlled mapping endpoints for authorized analytics jobs, not for raw export.

Operational checklist:

• Implement central tokenization service + KMS for keys.
• Add DP layer in query engine (or use privacy-preserving DB features).
• Integrate transformation library into BI connectors (Looker/Power BI).
• Define privacy budget, audit cadence, and stakeholder SLAs.

This combination preserves most high-level BI needs (funnels, retention, cohorts) with provable privacy controls; accept trade-offs at micro-levels (rare cohorts, exact counts) and manage via aggregation, explainability, and governed exceptions.

Objective: Validate whether a targeted GTM (pricing + messaging + channel) increases trial-to-paid conversion in a new country vs baseline before full market entry.

Hypothesis:

• H0: Targeted GTM conversion = baseline conversion.
• H1: Targeted GTM conversion > baseline conversion (one-sided).

Segmentation:

• Priority segments by ARR potential: SMB (0–$250k), Mid-market ($250k–$2M), Enterprise (>$2M). Also segment by acquisition channel (organic, paid search, partners) and language preference. Run experiment in top two cities/regions per country to control for geography.

Sample size estimation (example):

• Baseline conversion p0 = 2% (0.02). Minimum detectable uplift = 50% relative → p1 = 3% (0.03).
• Alpha = 0.05, Power = 0.8, one-sided zα=1.645, zβ=0.84.
• Pooled p = (p0+p1)/2 = 0.025.
• n per arm ≈ [ (zαsqrt(2p*(1-p)) + zβsqrt(p0(1-p0)+p1*(1-p1)))^2 ] / (p1-p0)^2
• Plugging numbers → n ≈ 12,000 users per arm. If traffic smaller, increase test duration or raise detectable effect.

Success metrics & thresholds:

• Primary: trial → paid conversion rate (absolute and relative uplift).
• Secondary: CAC, LTV (projected at 90 days), churn at 30 days, activation rate (key event), ARR per account.
• Statistical threshold: require p-value < 0.05 (one-sided) and 80% power; Bayes factor > 3 as secondary check.
• Practical threshold: ≥25% relative uplift in conversion OR payback period <6 months.

Dashboards to monitor (BI deliverables):

• Executive overview: cohort funnel (visitors → signups → activation → paid), daily/weekly conversion, cumulative lift, CI bands.
• Segmented view: by segment/channel/region with sample sizes, conversion, CTR, CAC, LTV.
• Health & validity: randomization balance checks, sample pacing (reaching n target), anomaly detection (sudden spikes).
• Cost model: CAC vs projected LTV, break-even timeline.
• Statistical summary tile: p-value, confidence intervals, power achieved, decision flag.

Decision criteria:

• Scale: if primary metric meets statistical threshold AND practical threshold across target high-value segments, and CAC/LTV economics positive → roll out phased country-wide (start with channels/segments that drove effect).
• Stop or iterate: if no statistical significance but directionally positive with insufficient sample power → extend test or run stratified experiments. If negative or increases churn/CAC beyond limits → stop and run root-cause analysis (qualitative user interviews, funnel leakage).
• Guardrails: require consistent lift for at least 2 weeks post-activation and no regressions in NPS/support volume before full launch.

Implementation notes for BI:

• Instrument events with consistent schema, expose experiment flags in user table, automate ETL to update dashboards daily, and schedule a decision review when sample targets and time windows are met.