Product Analytics Instrumentation and Event Tracking Questions

Requirements / goals:

• Measure adoption (first use, active use) and retention (D1/D7/D30, rolling cohorts) for a single cross-platform feature across web, Android, iOS, including users who never log in.
• Enable attribution to user (or device) across sessions and platforms, support merging when a user logs in, and maintain privacy/compliance.

High-level approach:

1. Required events (canonical schema across platforms)

• feature_viewed
  • props: feature_id, variant, platform, screen/URL, timestamp
• feature_started (user enters flow)
  • props: same + session_id, entry_point
• feature_completed (successful conversion)
  • props: outcome, value (monetary/score)
• feature_failed / feature_abandoned
  • props: step, error_code
• identity_bind (when user logs in/registers)
  • props: anonymous_id, user_id, timestamp, platform

Common event properties (all events): event_id, timestamp (UTC), platform, sdk_version, app_version, locale, device_id, anonymous_id, user_id (nullable), ip_hash (optional), user_agent_hash (optional), experiment_id.

2. Identity resolution

• Primary deterministic keys:
  • user_id when available (source of truth).
  • anonymous_id: persistent client-generated ID (web cookie/localStorage; mobile: installed UUID persisted across app launches).
• Stitching strategy:
  • Track anonymous_id for all anonymous activity.
  • On login/registration emit identity_bind event linking anonymous_id -> user_id.
  • In downstream analytics, merge anonymous events into user_id timeline (prefer server-side merging to prevent double-counting).
• Probabilistic/heuristic fallback:
  • When deterministic link not available, use hashed device fingerprint (ip_hash + ua_hash + device_props) with low confidence - use only for aggregate metrics, not user-level.

3. Measuring retention cohorts

• Cohort definition options:
  • Adoption cohort: users whose first feature_started occurred in a window (e.g., week starting 2025-11-01).
  • Activation cohort: users who completed key step within N days of first start.
• Retention metric:
  • For each cohort, compute percentage returning/using feature on D1/D7/D30 (based on first_event_date), where "use" = feature_started or feature_completed.
• Cohort types:
  • Time cohorts (weekly/monthly)
  • Behavioral cohorts (users who completed step X)
  • Platform cohorts (web-only, mobile-only, cross-platform)
• Implementation:
  • ETL computes first_event_date per entity (user_id if exists else anonymous_id).
  • For merged users, reassign anonymous cohort to user cohort when identity_bind occurs - use first_event_date as earliest of anonymous or user events.
• Reporting considerations:
  • Use Kaplan-Meier or retention curves to handle censored data for long windows.
  • Always show cohort size (N) and confidence intervals.

4. Handling users who never log in

• Persist anonymous_id for lifetime or until clear:
  • Web: cookie + localStorage with long TTL; plus server-set cookie on optional first-server-interaction.
  • Mobile: stable app instance ID (GUID), tied to installation.
• Cross-device anonymous users:
  • Cannot deterministically stitch; treat as separate anonymous entities.
  • Use probabilistic stitching only for aggregate-level insights; mark low-confidence.
• Prevent double-counting when user later logs in:
  • On identity_bind, backfill & merge anonymous events into user profile; mark events as merged to avoid duplicate attribution.
• Privacy & opt-out:
  • Honor Do Not Track / platform privacy settings; allow opt-out flag in events; exclude those from user-level linking when required.
  • Store only hashed device identifiers; rotate salts; keep retention of identifiers minimal per policy.

5. Implementation & operational notes

• Enforce strict event contract (schema validation in SDK/server).
• Server-side ingestion pipeline that:
  • Validates events, normalizes properties, writes raw events and aggregated materialized tables (first_seen, daily_active, cohort membership).
  • Runs daily batch to compute cohorts and retention metrics; provide near-real-time metrics for adoption (first 24-48h).
• Instrumentation QA:
  • Test flows across platforms, simulate anonymous->login transitions, verify merge behavior.
• KPIs to track:
  • Adoption rate = new users who start feature / active users
  • Activation rate = completes / starts
  • D1/D7/D30 retention by cohort and platform
  • Cross-platform lift: percent of users using feature on multiple platforms

Trade-offs:

• Deterministic stitching provides accuracy but needs login; probabilistic gives broader coverage but lower reliability.
• Client-side persistent IDs ease tracking but may be reset (cookies cleared, app reinstall) - treat as expected churn.

This design balances cross-platform traceability, privacy, and clean cohort measurement while enabling product decisions (where adoption drops, which platform retains best, effect of experiments).

Overview / Goal
Ensure experiment metrics reflect real user behavior by making events complete, uniquely attributed, and continuously validated so design conclusions are trustworthy.

Prioritized QA checklist (high → low)

1. Event schema & contract verification
   • Confirm required fields (event_name, timestamp, user_id, device_id, session_id, experiment_id).
   • Validate types/format and TTL (timestamps not future/past-bounds).
2. Data completeness tests
   • Compare client-side sent vs server-received counts per hour/day by platform.
   • Spike/drop detection relative to baseline.
3. Identity resolution & attribution
   • Verify presence and precedence rules (logged-in user_id > anonymous_id).
   • Check cross-device joins rate; flag high anonymous-to-user conversion gaps.
4. Deduplication checks
   • Detect duplicate event_ids within ingestion window; verify idempotency logic.
5. Experiment assignment fidelity
   • Ensure each user has consistent experiment_id across sessions; check randomization balance.
6. End-to-end UX validation
   • Instrument manual sessions and automated scripts to generate known events and assert pipeline arrival.

Remediation plan

• Fix schema violations at source; add client-side validation and guarded sends.
• Implement server-side idempotency using event_id + dedupe window.
• Implement identity graph: deterministic joins (user_id ↔ device_id), unify anonymous→login mapping, and store resolution confidence flags.
• Backfill/replay: replay missing events from client logs where available; label replayed events and recompute affected metrics; if logs missing, impute only with clear annotation and sensitivity analysis.
• Recompute experiment metrics after dedupe and identity resolution; publish comparison report with impact.

Monitoring & long-term controls

• Real-time alerts: ingestion drop, duplicate rate > threshold, identity mismatch rate.
• Dashboards: raw vs cleaned counts, experiment assignment stability, conversion by resolved identity type.
• Regression tests in CI for analytics schema and A/B randomization.
• Documentation & SLAs: event contracts, ownership, and runbook for incidents.

This ensures experiments used by research are reliable, traceable, and actionable for design decisions.

Requirements & constraints:

• Functional: produce cross-device user-level joins for measurement while maximizing accuracy and enabling cohort/metric reporting.
• Non-functional: GDPR compliance (data minimization, purpose limitation, lawful basis, user rights), low-latency joins for analytics, explainability for bias checks.

High-level approach:

• Use a hybrid graph: deterministic links (auth IDs such as hashed login_email, user_id) form the core; probabilistic edges (device fingerprint signals) fill gaps with scored match confidence. Store only minimal, purpose-limited artifacts and aggregate outputs for reporting.

Identifiers to store (minimal & purpose-limited):

• Deterministic keys: salted+hashed user_id or email (only if user consented to analytics/auth linking). Use per-purpose salt.
• Device pseudonyms: one-way hashed device_id (e.g., mobile_ad_id hashed client-side) with per-environment salt.
• Signals for probabilistic matching: coarse IP ranges (not full IP), browser family/version, OS, timezone, anonymized and binned timestamps, and hashed user-agent tokens. Avoid raw PII.
• Metadata: match_score, match_method (deterministic/probabilistic), timestamp, consent_flags.

Hashing & anonymization practice:

• Hash all identifiers with HMAC-SHA256 using a secret per-legal-entity and per-purpose salt. Rotate keys; store key-rotation logs. Do not store raw PII server-side.
• Apply k-anonymity thresholds for any attribute buckets before exporting (e.g., only report cohorts >= N users).
• Differential privacy or noise addition for sensitive aggregate metrics where re-identification risk exists.

Consent handling & lawful basis:

• Treat BOTH auth-linked deterministic joins AND probabilistic cross-device matching as requiring explicit opt-in consent for cross-device analytics: the ePrivacy Directive's consent requirement for storing/reading identifiers on a user's device (cookies, device IDs, fingerprinting signals) applies regardless of GDPR's Article 6(1)(f) legitimate-interest basis, and cross-device tracking is not a strictly-necessary purpose, so legitimate interest cannot substitute for consent here. Run a DPIA to document risk, and record consent timestamp and scope per purpose (deterministic linking vs probabilistic fingerprinting). Enforce consent checks in join pipeline to exclude records without required consent from either deterministic or probabilistic joins.
• Implement consent flags at ingestion and propagate to derived datasets. Support right-to-be-forgotten: delete or tombstone hashed identifiers and remove from joins; track deletions in downstream aggregates via re-computation or retention windows.

Matching model & measuring quality:

• Maintain ground truth samples from deterministic-linked users (consented) to train and evaluate probabilistic models.
• Measure precision, recall, F1 at multiple score thresholds; build ROC and precision-recall curves. Track calibration (predicted score vs. empirical correctness).
• Compute uplift vs. deterministic-only baseline on key metrics (e.g., conversion rate per user). Monitor false-merge impact by estimating change in unique user counts and per-user metric distributions.
• Store match provenance: which signals contributed to a match to support audits.

Impact on reported metrics & biases:

• Deterministic links are high precision; probabilistic matches increase coverage but can inflate or deflate per-user metrics due to false merges/splits.
• Report both "deterministic-only" and "hybrid" versions for key metrics and show sensitivity bands (e.g., +/- based on match-score thresholds).
• Monitor demographic and device-type biases by stratifying match-accuracy by cohorts (region, OS, browser). Probabilistic methods often underperform for privacy modes or minority device types - flag and correct (reweight or exclude) where necessary.
• Keep lineage: allow analysts to filter reports by match_method and confidence.

Operational considerations for a Data Analyst:

• Build standardized SQL views: deterministic_users, probabilistic_matches_with_score, resolved_user_graph, and aggregated metrics with match provenance.
• Automate validation jobs: daily precision/recall against ground truth, drift detection for signal distributions, and consent-compliance audits.
• Document assumptions and provide dashboards showing metric sensitivity to match thresholds so stakeholders understand uncertainty.

Trade-offs:

• Higher coverage via probabilistic matching vs. increased privacy risk and metric distortion. Favor conservative thresholds for business-critical metrics and use probabilistic matches more in exploratory analyses, surfacing uncertainty to decision-makers.