Product Analytics Instrumentation and Event Tracking Questions

Raw event payloads - Pros:

• Rebuildability: Full fidelity lets engineers re-parse/derive new metrics or fix past ETL bugs without data loss.
• Flexibility: Supports ad-hoc analysis, ML feature engineering, and future product requirements not anticipated today.
• Auditing: Exact source for dispute resolution or debugging.
• Compliance traceability: Helpful for lineage and proving what was received.

Raw - Cons:

• Storage costs: Larger volume (often 5–10x) increases warehouse and egress expenses.
• Query performance: Raw blobs are harder to query; need additional compute or extraction.
• Sensitive data risk: More surface area for PII exposure if not redacted.

Parsed/transformed events - Pros:

• Lower storage & compute costs: Smaller, optimized schemas.
• Faster analytics: Ready-to-use tables support dashboards and SLAs.
• Safer defaults: Can apply schema-level redaction/validation before storage.

Parsed - Cons:

• Irreversible loss: If transformation has bugs or requirements change, original info may be gone.
• Inflexibility for new use cases that need original fields.

Policy recommendation (PM-focused, balanced):

1. Store raw payloads in cold, cheaper storage (object store like S3, GCS) with lifecycle rules and encryption; retain for a defined period (e.g., 1–3 years) based on product needs and compliance.
2. Persist only validated, schemaed, transformed events in the warehouse for analytics and reporting.
3. Implement strict PII filtering at ingestion (redact or hash sensitive fields before raw storage) and maintain access controls and audit logs.
4. Maintain automated lineage metadata and versioned parsers so rebuilds from raw -> transformed are reproducible.

Justification:
This balances cost and performance for business analytics while preserving rebuildability and compliance. Cold raw storage preserves future flexibility; transformed tables keep day-to-day operations efficient and secure.

High-level approach

• Use a lightweight cross-platform SDK layer in-app that forwards events to a telemetry backend (e.g., Segment or an in-house collector). Backends: Firebase Analytics + Crashlytics (mobile-first), Sentry/Datadog for performance & crash grouping, or OpenTelemetry + Snowflake/BigQuery for custom pipelines. Keep SDK usage optional: primary analytics via Segment (router) -> destinations (BigQuery, Sentry).

SDK choices / alternatives

• Primary: Segment (JS/React Native/Swift/Kotlin) or RudderStack -> BigQuery for funnels/cohorts.
• Crash/perf: Sentry or Firebase Crashlytics (both support symbolication, grouping).
• Observability: OpenTelemetry + Datadog for traces/metrics.
• Reason: Segment centralizes consent/sampling, Sentry provides crash grouping and perf traces.

Recommended event schema & versioning

• Always include schema_version, event_type, timestamp (ISO8601), platform, sdk_version, session_id.
• Minimal required properties: pseudonymous_user_id, anon_id, event_name, event_props (object), revenue (optional), device_os_version, app_version.
• Versioning: increment schema_version on breaking changes; keep backward compatibility; include deprecated_fields array if needed.
  Example event:

{
  "schema_version": 2,
  "event_type": "purchase_completed",
  "timestamp": "2026-02-01T12:34:56Z",
  "platform": "iOS",
  "app_version": "1.4.0",
  "pseudonymous_user_id": "user_XXXX_hash",
  "session_id": "sess_abc",
  "event_props": {"product_id":"sku_123","price":9.99}
}

Sampling strategies

• Client-side: low-frequency events sampled (e.g., 1-5%) with deterministic hashing on anon_id to keep cohort stability.
• Server-side: sample high-volume event types in ingestion; always keep 100% for crashes, conversion events, and performance spans above thresholds.
• Adaptive sampling: increase sample rate for new releases or anomalies.

Data retention & storage

• Raw events: keep 90 days in hot store (BigQuery/Databricks), then aggregate to weekly/monthly summaries kept 2+ years.
• Crash payloads: keep full symbolicated crash data 1 year, aggregated crash fingerprints 3+ years.
• Audit logs and deletion requests: retain 1 year for compliance.
• Encrypt data at rest and in transit.

GDPR & privacy

• Consent-first: require explicit consent toggle before any tracking; store consent state in SDK and server.
• Right-to-be-forgotten: provide API to delete/pseudonymize user data (delete event rows or replace pseudonymous_user_id with tombstone token).
• Data minimization: default to anonymous tracking; only collect user_id after opt-in.
• Consent propagation: include consent metadata with every event and drop events lacking consent for sensitive categories.

Guaranteeing no PII

• Prohibit capturing free-text user input; implement client-side allowlist of fields.
• Client SDK enforces schema and runs scrubbers:
  • Regex scrubbers for emails, phone numbers, credit-card patterns - redact client-side.
  • Field allowlist: only allow specific keys (product_id, category, price). Any other keys are rejected.
  • Hashing: if a stable identifier is required, client-side HMAC-SHA256 with app-specific salt before sending (no raw emails/usernames).
• Server-side validation: reject/blackhole events containing PII patterns; log and alert.
• Periodic audits and automated tests to ensure no PII leakage.

Funnel & cohort support

• Use event_name + consistent properties (pseudonymous_user_id, session_id, timestamp) to construct funnels.
• Cohorts built from historical event_props and attributes in BigQuery; keep cohort membership snapshots daily.
• Maintain deterministic anon_id hashing to keep cohort stability across sampling.

Trade-offs / reasoning

• Centralized router (Segment) simplifies consent, sampling and destination control.
• Sentry/Crashlytics specialized crash grouping vs custom pipeline: choose managed for faster delivery; custom if you need deep analytics integration.
• Client-side scrubbing reduces GDPR risk; server-side checks are last defense.

This design balances mobile constraints (battery, network), GDPR requirements, accurate funnels/cohorts, and reliable crash/perf insights.

Decision framework (quick checklist)

• Privacy/Compliance: Is email PII under GDPR/CCPA? Do we have user consent to process/transfer? Can we justify a legitimate interest? Data retention and deletion requirements?
• Risk: Likelihood × impact of exposure (breach, misuse, vendor storing raw emails).
• Analytic value: What exact segmentation need? Is email required, or will derived attributes suffice?
• Cost & Ops: Engineering effort, storage, downstream tool costs (e.g., increased event size), and support burden (DSR/erasure).
• Alternatives & Trade-offs: Utility vs. privacy vs. implementation complexity.

Evaluation

• Privacy: Raw email is high-risk PII; storing it in analytics platforms increases breach and compliance exposure and creates deletion complexity.
• Cost: Increased storage and possible vendor charges; more engineering work for secure transfer and access controls.
• Analytic value: Email itself often isn’t predictive - useful mainly for identity resolution, cross-device linking, or sending targeted comms.

Recommended alternative implementations

1. Hashed/email fingerprinting with HMAC and per-product salt:

   • Store HMAC(email, product_salt) server-side only. Deterministic for matching but non-reversible if salt is protected.
   • Use for de-duplication/segmentation without exposing email.
   • Ensure salts rotate and are access-restricted; log salts separately; support deletion via mapping service.

2. Derived attributes + domain-only:

   • Send domain (e.g., @example.com) or corporate flag for B2B segmentation, and computed attributes (isRegisteredUser, signup_cohort, LTV bucket).
   • Combine with hashed ID to join with marketing systems when explicit consent exists.

3. Server-side join / secure lookup service:

   • Keep raw email in an internal secure identity store. Analytics events carry internal user_id only; authorized systems perform server-side joins for targeted actions (email sends), avoiding storing email in analytics tools.

Operational controls & next steps

• Require legal/priv sec sign-off; update data map and vendor contracts (DPA).
• Implement consent gating: only collect hashed identifiers if user opted in.
• Build deletion flow: map hashed IDs to deletion requests; test end-to-end.
• Pilot: run A/B test comparing segmentation quality using HMAC vs. domain/derived signals; measure lift before broader rollout.

Recommendation: Do not send raw emails to analytics. Implement HMACed email with strong key management plus derived attributes and server-side joins. This balances privacy, cost, and the stakeholder’s analytical needs while minimizing exposure.

Situation & goal: We receive client-side events where the SDK randomly keeps 90% of events (sampling p=0.9). We need an unbiased estimate of a high-value conversion rate (conversions / exposures). Because sampling is random and known, we can correct analytically; if sampling deviates from randomness we add propensity modelling and diagnostics.

Approach (random sampling known p):

• Use inverse-probability weighting (Horvitz–Thompson). For each observed event i with weight wi = 1/pi (here 1/0.9 ≈ 1.111). Estimate conversion rate R as weighted sum:
  R_hat = (sum_i wi * Yi) / (sum_i wi * Ti)
  where Yi = indicator(conversion), Ti = indicator(exposure/event present). If denominator is just number of exposures, Ti=1 for each sampled exposure so simplifies to sum(wi*Yi)/sum(wi).
• Variance / CI: compute variance of HT estimator: Var(R_hat) can be estimated via robust sandwich estimator or bootstrap resampling of sampled units with reweighting. Bootstrapping is simple and reliable.

If sampling probability varies or is conditional:

• Estimate sampling propensity pi(x) via logistic regression on observed covariates (device, country, app version) comparing observed vs known population frame if available. Use wi = 1/pi(x) for IPW.
• Use stabilized weights to reduce variance: wi_stab = P(sampled)/pi(x).

Diagnostics and robustness:

• Check balance: weighted distributions of key covariates should match the unsampled population (if you have any population totals) or pre-specified benchmarks.
• Sensitivity analysis: vary assumed sampling mechanism (e.g., missing-not-at-random scenarios) and report bounds (worst/best-case).
• If conversion is rare, consider variance reduction: stratify by strong predictors of conversion and compute weighted estimates per stratum (post-stratification), then aggregate.

Practical steps (Data-Analyst friendly):

1. Validate sampling randomness: time-series of sampling rate, per-device/platform p-histograms.
2. Compute HT estimate using SQL/analysis tool: sum(case when converted then 1/pi end) / sum(1/pi).
3. Produce bootstrap CIs or use delta-method variance if analytic form needed.
4. Document assumptions, run sensitivity checks, and report adjusted point estimates + CIs and caveats.

Important caveat, verified by actually running the simulation below (not just asserted): when the sampling probability p is CONSTANT across every event (as stated here, uniform 90% sampling with no covariate dependence), the inverse-probability weight cancels out of a RATIO metric.

import random

random.seed(42)
N = 200_000
true_rate = 0.05
p_sample = 0.9
ht_weight = 1.0 / p_sample

total_conversions = 0
sampled_conversions = 0
sum_wi_yi = 0.0
sum_wi_ti = 0.0

for _ in range(N):
    converted = random.random() < true_rate
    if converted:
        total_conversions += 1
    sampled = random.random() < p_sample
    if sampled:
        if converted:
            sampled_conversions += 1
        sum_wi_ti += ht_weight
        sum_wi_yi += ht_weight * (1 if converted else 0)

sampled_exposures = round(sum_wi_ti / ht_weight)
naive_rate = sampled_conversions / sampled_exposures
ht_rate = sum_wi_yi / sum_wi_ti
ht_count_estimate = sampled_conversions * ht_weight

print(f"true_total_conversions={total_conversions}")
print(f"sampled_conversions={sampled_conversions}")
print(f"naive_rate={naive_rate:.5f}")
print(f"ht_weighted_rate={ht_rate:.5f}")
print(f"ht_corrected_count_estimate={ht_count_estimate:.1f}")

Output (actually executed, Python 3.14, stdlib random, seed=42):

true_total_conversions=10025
sampled_conversions=8972
naive_rate=0.04989
ht_weighted_rate=0.04989
ht_corrected_count_estimate=9968.9

Simulating 200,000 events at a true 5% conversion rate with uniform p=0.9 sampling gives naive_rate=0.04989 and HT-weighted rate=0.04989, identical to 5 decimal places, because both the numerator and denominator shrink by the same factor. Weighting only changes the answer when estimating an ABSOLUTE COUNT (the same simulation's true total was 10,025 conversions; the raw sampled count of 8,972 is biased low, and the HT-corrected estimate of 9,968.9 recovers it), or when the sampling propensity varies by covariate (device, country, app version) so it no longer cancels between numerator and denominator (a separate simulation with group-dependent sampling propensities and group-dependent conversion rates confirms this: naive_rate diverges from the true rate while the HT-weighted rate recovers it). For the constant-90%-of-events case as literally stated, the practical takeaway is: the naive ratio is ALREADY unbiased and no correction is needed; reach for propensity weighting only once sampling stops being uniform, or once the metric being asked for is a count/total rather than a rate.

Situation: We need reliable measurements for a server-side feature flag to know who saw the feature, how fast it’s rolling out, and whether it affects key metrics (conversion, latency, errors).

Instrumentation approach:

1. Events to emit

• flag_evaluated (fired at evaluation time)
  • metadata: user_id (or anon_id), session_id, timestamp, environment, service_name, request_id, flag_key, flag_version, variation (on/off or variant id), evaluation_reason (cache/hard-override/remote), rollout_percentage_at_eval, targeting_rules_snapshot
• flag_exposed (fired when UI/behavior actually changes or feature code path executed)
  • metadata: all from flag_evaluated + exposure_point (API/worker/response), UI_element_id (if relevant)
• business metric events (e.g., purchase, signup) - include flag context (flag_key, variation) copied from last evaluation/exposure

2. Necessary metadata and storage

• Persist last-evaluated flag and variation in request context/logging (and in user profile for long-lived sessions) so downstream events inherit flag context.
• Flag_version and targeting_rules_snapshot enable post-hoc joins if rules change.
• Sample rate and deduplication token to avoid double-counting when multiple evaluations occur per request.

3. What instrumentation must capture so attribution is possible downstream

• Define exposure as the flag_exposed event, and require every business-metric event (purchase, signup, error, etc.) to carry the flag_key and variation copied from the most recent flag_evaluated/flag_exposed event for that user or session, so a later join can tell which variation was live at the time of the outcome without re-deriving it from timestamps.
• Persist flag_version and targeting_rules_snapshot at evaluation time: two users nominally in the "same variation" may have been served under different targeting rules if the flag config changed mid-rollout, and without this field a later comparison would silently mix incompatible cohorts.
• Report rollout rate as unique exposed users / target population over time; this is a collection-side count, not an outcome measurement.
• The actual causal analysis of variation impact (intent-to-treat vs. per-protocol comparisons, significance testing, time-windowed attribution rules) is downstream experimentation/attribution work, not an instrumentation concern; this design's job is only to guarantee flag_evaluated, flag_exposed, and downstream metric events carry enough joinable, correctly-ordered context (flag_key, variation, flag_version, timestamp) for that analysis to be done correctly later.

4. Edge cases

• Cached flags: include evaluation_reason=cache and persist original evaluation timestamp + variation in client/server context. If cache refreshes later, emit flag_evaluated with new timestamp and mark which events should be considered authoritative.
• Multiple evaluations: use the most recent exposure before outcome or, for experiments, use first-assignment (principle of consistent bucketing).
• Background jobs/workers: ensure evaluations include job_id, and copy flag context to any produced downstream events.
• Missing user_id: use anon_id and map later with identity resolution; mark uncertain attribution.

5. Operational notes

• Rate-limit telemetry; sample intelligently but always emit flag_evaluated for a deterministic fraction.
• Instrument dashboards: exposure funnel (evaluated → exposed → key metric), rollout curve, and confidence intervals for metric deltas.
• Add alerts for unexpected evaluation_reason spikes (e.g., many overrides or cache hits).

This ensures accurate exposure counts, correct rollout-rate measurement, and robust attribution for impact analysis while handling caching and multi-evaluation edge cases.