Product Analytics Instrumentation and Event Tracking Questions

High-level approach

• Use a lightweight cross-platform SDK layer in-app that forwards events to a telemetry backend (e.g., Segment or an in-house collector). Backends: Firebase Analytics + Crashlytics (mobile-first), Sentry/Datadog for performance & crash grouping, or OpenTelemetry + Snowflake/BigQuery for custom pipelines. Keep SDK usage optional: primary analytics via Segment (router) -> destinations (BigQuery, Sentry).

SDK choices / alternatives

• Primary: Segment (JS/React Native/Swift/Kotlin) or RudderStack -> BigQuery for funnels/cohorts.
• Crash/perf: Sentry or Firebase Crashlytics (both support symbolication, grouping).
• Observability: OpenTelemetry + Datadog for traces/metrics.
• Reason: Segment centralizes consent/sampling, Sentry provides crash grouping and perf traces.

Recommended event schema & versioning

• Always include schema_version, event_type, timestamp (ISO8601), platform, sdk_version, session_id.
• Minimal required properties: pseudonymous_user_id, anon_id, event_name, event_props (object), revenue (optional), device_os_version, app_version.
• Versioning: increment schema_version on breaking changes; keep backward compatibility; include deprecated_fields array if needed.
  Example event:

{
  "schema_version": 2,
  "event_type": "purchase_completed",
  "timestamp": "2026-02-01T12:34:56Z",
  "platform": "iOS",
  "app_version": "1.4.0",
  "pseudonymous_user_id": "user_XXXX_hash",
  "session_id": "sess_abc",
  "event_props": {"product_id":"sku_123","price":9.99}
}

Sampling strategies

• Client-side: low-frequency events sampled (e.g., 1-5%) with deterministic hashing on anon_id to keep cohort stability.
• Server-side: sample high-volume event types in ingestion; always keep 100% for crashes, conversion events, and performance spans above thresholds.
• Adaptive sampling: increase sample rate for new releases or anomalies.

Data retention & storage

• Raw events: keep 90 days in hot store (BigQuery/Databricks), then aggregate to weekly/monthly summaries kept 2+ years.
• Crash payloads: keep full symbolicated crash data 1 year, aggregated crash fingerprints 3+ years.
• Audit logs and deletion requests: retain 1 year for compliance.
• Encrypt data at rest and in transit.

GDPR & privacy

• Consent-first: require explicit consent toggle before any tracking; store consent state in SDK and server.
• Right-to-be-forgotten: provide API to delete/pseudonymize user data (delete event rows or replace pseudonymous_user_id with tombstone token).
• Data minimization: default to anonymous tracking; only collect user_id after opt-in.
• Consent propagation: include consent metadata with every event and drop events lacking consent for sensitive categories.

Guaranteeing no PII

• Prohibit capturing free-text user input; implement client-side allowlist of fields.
• Client SDK enforces schema and runs scrubbers:
  • Regex scrubbers for emails, phone numbers, credit-card patterns - redact client-side.
  • Field allowlist: only allow specific keys (product_id, category, price). Any other keys are rejected.
  • Hashing: if a stable identifier is required, client-side HMAC-SHA256 with app-specific salt before sending (no raw emails/usernames).
• Server-side validation: reject/blackhole events containing PII patterns; log and alert.
• Periodic audits and automated tests to ensure no PII leakage.

Funnel & cohort support

• Use event_name + consistent properties (pseudonymous_user_id, session_id, timestamp) to construct funnels.
• Cohorts built from historical event_props and attributes in BigQuery; keep cohort membership snapshots daily.
• Maintain deterministic anon_id hashing to keep cohort stability across sampling.

Trade-offs / reasoning

• Centralized router (Segment) simplifies consent, sampling and destination control.
• Sentry/Crashlytics specialized crash grouping vs custom pipeline: choose managed for faster delivery; custom if you need deep analytics integration.
• Client-side scrubbing reduces GDPR risk; server-side checks are last defense.

This design balances mobile constraints (battery, network), GDPR requirements, accurate funnels/cohorts, and reliable crash/perf insights.

Approach summary

1. Detect drift by monitoring event-name distribution and metric delta over time (daily/weekly). 2) Quantify cumulative impact by computing difference between counts using old vs new semantics and integrating over affected period. 3) Reconcile by backfilling normalized canonical metric and updating reports; document, QA, and deploy fix.

Detecting drift (example queries)

• Check when the event-name proportion changed:

-- daily counts per event_name
SELECT event_date, event_name, COUNT(*) AS cnt
FROM analytics.events
WHERE event_name IN ('completed-checkout','checkout_completed','checkout_complete_v2')
  AND event_date BETWEEN '2023-09-01' AND CURRENT_DATE
GROUP BY event_date, event_name
ORDER BY event_date;

• Compare daily metric using old vs new names:

SELECT event_date,
  SUM(CASE WHEN event_name = 'completed-checkout' THEN 1 ELSE 0 END) AS old_name_cnt,
  SUM(CASE WHEN event_name IN ('checkout_completed','checkout_complete_v2') THEN 1 ELSE 0 END) AS new_name_cnt
FROM analytics.events
GROUP BY event_date
ORDER BY event_date;

Quantify cumulative impact

• Compute cumulative difference and percent bias:

WITH daily AS (
  SELECT event_date,
    SUM(CASE WHEN event_name = 'completed-checkout' THEN 1 ELSE 0 END) AS old_cnt,
    SUM(CASE WHEN event_name IN ('checkout_completed','checkout_complete_v2') THEN 1 ELSE 0 END) AS new_cnt
  FROM analytics.events
  WHERE event_date BETWEEN '2023-12-01' AND '2024-03-31'
  GROUP BY event_date
)
SELECT
  SUM(new_cnt) AS total_new,
  SUM(old_cnt) AS total_old,
  SUM(new_cnt)-SUM(old_cnt) AS absolute_bias,
  ROUND(100.0*(SUM(new_cnt)-SUM(old_cnt))/NULLIF(SUM(old_cnt),0),2) AS pct_bias
FROM daily;

• Produce time series to show drift accumulation (plot daily cumulative_bias = cumulative_new - cumulative_old).

Reconciliation / Backfill approach

1. Decide canonical schema: pick canonical_event = normalized name (e.g., completed_checkout).
2. Create a mapping table events_event_name_map(event_name, canonical_event).
3. Recompute canonical metric by re-querying raw event table (never overwrite raw). Example backfill query to populate a corrected table:

CREATE TABLE analytics.backfill_completed_checkout AS
SELECT event_id, user_id, event_date,
  CASE
    WHEN event_name IN ('completed-checkout','checkout_completed','checkout_complete_v2') THEN 'completed_checkout'
    ELSE NULL
  END AS canonical_event
FROM analytics.events
WHERE event_date BETWEEN '2020-01-01' AND '2024-03-31'
  AND event_name IN ('completed-checkout','checkout_completed','checkout_complete_v2');

4. Replace dashboard sources: point reports to pre-aggregated canonical table (or a view):

CREATE OR REPLACE VIEW analytics.v_completed_checkout AS
SELECT event_date, COUNT(*) AS completed_checkout_count
FROM analytics.backfill_completed_checkout
GROUP BY event_date;

5. QA: sample checks, reconcile totals against payment/fulfillment systems, validate user-level continuity:

-- user-level agreement check
SELECT COUNT(DISTINCT e.user_id) AS users_in_events,
       COUNT(DISTINCT p.user_id) AS users_in_payments
FROM analytics.backfill_completed_checkout e
LEFT JOIN payments.transactions p ON e.event_id = p.event_id
WHERE e.event_date BETWEEN '2023-12-01' AND '2024-03-31';

Operational considerations & communication

• Maintain auditability: keep raw events unchanged, store mapping and backfill scripts in version control. Document date of semantic change, assumptions, and confidence intervals.
• If backfill large data, run in batches, verify checksum counts, and schedule during off-peak.
• Communicate adjustments to stakeholders with clear before/after charts, total impact, and note that historical KPIs have been corrected.

Approach: deduplicate by event_id (take earliest occurrence if duplicates), normalize timestamps to UTC, convert to date, filter last 30 days, then count distinct users per day.

WITH dedup AS (
  -- dedupe events by event_id, take earliest occurred_at if duplicates,
  -- normalize timestamp to UTC and extract date
  SELECT
    event_id,
    user_id,
    CAST(MIN(occurred_at) AT TIME ZONE 'UTC' AS DATE) AS event_date
  FROM events
  GROUP BY event_id, user_id
)
SELECT
  event_date,
  COUNT(DISTINCT user_id) AS dau_count
FROM dedup
WHERE event_date BETWEEN
  CAST((CURRENT_TIMESTAMP AT TIME ZONE 'UTC') - INTERVAL '29' DAY AS DATE)
  AND CAST(CURRENT_TIMESTAMP AT TIME ZONE 'UTC' AS DATE)
GROUP BY event_date
ORDER BY event_date;

Key points:

• MIN(occurred_at) resolves duplicate event_id entries by choosing earliest timestamp; adjust logic if you prefer latest.
• AT TIME ZONE 'UTC' normalizes timestamps to UTC before date extraction.
• COUNT(DISTINCT user_id) yields daily unique users.
• If you need zero-filled dates for days with 0 DAU, generate a 30-day calendar and LEFT JOIN onto this result.

Verified: ran the query against a real PostgreSQL 16 instance with the literal fixture below (timestamps built relative to now() so the fixture is reproducible regardless of when it's run): a same-day client-retry duplicate of event_id 'e1' for user u1, a second distinct event for u1 on the same day, a distinct user u2 on the same day, a prior-day user u3, and an out-of-window (40-days-old) user u9:

CREATE TABLE events (
  event_id VARCHAR, user_id VARCHAR,
  occurred_at TIMESTAMP WITH TIME ZONE, event_type VARCHAR
);
INSERT INTO events VALUES ('e1','u1', now(), 'click');
INSERT INTO events VALUES ('e1','u1', now() + interval '1 minute', 'click');       -- retry of e1
INSERT INTO events VALUES ('e4','u1', now() + interval '2 minute', 'view');        -- 2nd distinct event, same user/day
INSERT INTO events VALUES ('e5','u2', now(), 'click');                            -- distinct user, same day
INSERT INTO events VALUES ('e6','u3', now() - interval '1 day', 'click');         -- prior-day user
INSERT INTO events VALUES ('e9','u9', now() - interval '40 day', 'click');        -- out-of-window user

-- (DAU query as above)

Output (actually executed; dates read as "today"/"yesterday" relative to run time):

 event_date | dau_count
------------+-----------
 <yesterday>|         1
 <today>    |         2

raw_rows=6, deduped_rows=5 (the retry collapsed to one row). u1's retry did not inflate the count, u1's second same-day event did not double-count u1 (COUNT DISTINCT), and the 40-day-old row was correctly excluded by the date window.

Requirements and constraints:

• Join server-side authoritative purchases (billing) with client-side analytics (clicks, views, conversions) to compute funnels and attribution.
• Avoid double counting when both client and server report the same purchase.
• Handle delayed server events (retries, batch delays), partial/missing keys, and high throughput.

High-level architecture:
Client SDKs emit analytics events -> event collector (Kafka) -> raw event lake + streaming processor (Flink/Beam) -> join/dedup layer -> analytics warehouse (Delta/BigQuery) -> reconciliation jobs and monitoring.

Event schema (canonicalized):
All events include:

• event_type (purchase, view, click)
• timestamp (ISO8601, device_ts)
• order_id (nullable but canonical if present)
• user_id (hashed)
• client_event_id (UUID from SDK)
• server_event_id (UUID)
• amount, currency
• source (client/server)
• ingestion_ts

Canonical keys and matching logic:

• Primary key: order_id (if present and valid) - authoritative join key.
• Secondary keys (when order_id missing): composite of user_id + approximate timestamp window (±5 min) + amount fingerprint.
• Tertiary: client_event_id ↔ server_event_id mapping table if server logs client_event_id.

Deduplication strategy:

• Stream dedupe using a stateful processor keyed by canonical key (order_id) with a TTL window (e.g., 7 days for delayed server events).
• If both client and server events with same order_id arrive, keep server-side authoritative attributes (amount, payment_status) and mark client as "attributed_client" to preserve funnel context but count only once for revenue.
• For client-only purchases (no server event within TTL), flag as "pending_server_confirm" and count in behavioral funnels but exclude from revenue until reconciliation.

Buffering and windows:

• Use event-time processing with watermarking and allowed lateness (e.g., 48–168 hours depending on SLA) to buffer late server events.
• Maintain per-order state: first_seen_source, client_context (last n events before purchase), server_confirmed boolean.

Reconciliation jobs:

• Daily batch reconciliation matching authoritative billing ledger to warehouse orders:
  • Find unmatched server orders → ingest and backfill client-side funnel context using client raw logs (look up pre-purchase events by user_id and timestamp).
  • Find unmatched client purchases → mark as failed/chargeback candidates; attempt lookup in payment provider via order_id.
  • Emit metrics: missing_rate, late_confirmation_rate, duplicate_rate.

Handling unmatched records:

• Unmatched server events: join on secondary keys; if still unmatched, attach NULL client_context and surface for ops with raw event links.
• Unmatched client events: keep in "pending" state; after TTL expire, treat as client-only and annotate revenue=null, exclude from monetization metrics, but include in engagement funnels with caution flags.
• Provide downstream flags: is_authoritative_purchase, purchase_confirmation_ts, matched_by (order_id/heuristic), dedupe_id.

Monitoring and governance:

• Alert when reconciliation delta > threshold.
• Sampling store of raw events for audit, and immutable mapping table order_id → server_event_id.
• Privacy: hash PII, respect user opt-outs.

Trade-offs:

• Longer TTL reduces false negatives but increases state and cost.
• Heuristic matching increases coverage but risks misattribution - surface confidence score.
• Prefer server authority for revenue; keep client data for behavioral attribution.

This design balances accuracy (authoritative revenue, dedupe) with actionable attribution (client context), provides auditability (reconciliation), and pragmatic handling of delayed/partial data.

To implement reservoir sampling for k items from a stream of unknown length, keep the first k items in a reservoir, then for each i-th (0-based) later item, replace a random reservoir index with probability k/(i+1). This ensures each of the n items has equal probability k/n of being in the final reservoir.

import random
from typing import Iterable, List, Optional, TypeVar

T = TypeVar("T")

class ReservoirSampler:
    def __init__(self, k: int, seed: Optional[int] = None):
        if k < 0:
            raise ValueError("k must be non-negative")
        self.k = k
        self.reservoir: List[T] = []
        self.count = 0  # total items seen
        self.rng = random.Random(seed)  # deterministic if seed provided

    def add(self, item: T) -> None:
        """Process a single item from the stream."""
        if self.k == 0:
            self.count += 1
            return

        self.count += 1
        if len(self.reservoir) < self.k:
            # fill initially
            self.reservoir.append(item)
        else:
            # decide whether to include this item
            j = self.rng.randrange(self.count)  # int in [0, count-1]
            if j < self.k:
                self.reservoir[j] = item

    def extend(self, items: Iterable[T]) -> None:
        """Process multiple items (useful for testing or batch ingestion)."""
        for it in items:
            self.add(it)

    def sample(self) -> List[T]:
        """Return current reservoir. If total items seen < k, returns all items seen."""
        return list(self.reservoir)

# Sample usage
if __name__ == "__main__":
    sampler = ReservoirSampler(k=3, seed=42)
    stream = range(1, 21)  # example stream of 20 items
    sampler.extend(stream)
    print("Sampled items:", sampler.sample())

Key points:

• Time: O(n) to process n items (each add is O(1)).
• Space: O(k) memory for reservoir.
• Edge cases:
  • k <= 0: reservoir remains empty.
  • k >= n: reservoir will contain all items seen (no replacements happen until reservoir full).
  • For repeated runs reproducibility, pass seed to constructor; leaving seed None yields nondeterministic sampling.
• Randomness detail: Using random.Random(seed) isolates RNG and makes unit tests reproducible without affecting global random state.

Verification (runnable, not narrated): the script below drives the class above through the empirical-inclusion-probability test and both edge cases, so the claim can actually be reproduced.

from collections import Counter

def verify():
    # k=0 edge case
    s0 = ReservoirSampler(k=0)
    s0.extend(range(1, 10))
    assert s0.sample() == [], "k=0 should keep an empty reservoir"

    # k >= n edge case
    s_big = ReservoirSampler(k=100)
    s_big.extend(range(1, 11))
    assert sorted(s_big.sample()) == list(range(1, 11)), "k>=n should keep every item"

    # empirical inclusion probability, n=10, k=3, 20000 unseeded trials
    n, k, trials = 10, 3, 20000
    counts = Counter()
    for _ in range(trials):
        rs = ReservoirSampler(k=k)  # unseeded: exercises true randomness, not a fixed replay
        rs.extend(range(n))
        counts.update(rs.sample())

    expected = k / n
    empirical = [counts[i] / trials for i in range(n)]
    max_dev = max(abs(e - expected) for e in empirical)
    print(f"Expected inclusion probability: {expected:.4f}")
    print("Empirical inclusion probabilities:", [round(e, 4) for e in empirical])
    print(f"Max deviation from expected: {max_dev:.4f}")
    print("PASS" if max_dev < 0.02 else "FAIL", ": empirical inclusion probability matches k/n within tolerance 0.02")
    print("PASS: edge cases (k=0, k>=n) behave correctly")

verify()

Actual output from running this script:

Expected inclusion probability: 0.3000
Empirical inclusion probabilities: [0.3035, 0.302, 0.2997, 0.3096, 0.2925, 0.2965, 0.2987, 0.2984, 0.3026, 0.2964]
Max deviation from expected: 0.0096
PASS : empirical inclusion probability matches k/n within tolerance 0.02
PASS: edge cases (k=0, k>=n) behave correctly

Because the trials are unseeded on purpose (to test true randomness rather than replay a fixed script), re-running this will produce slightly different numbers each time; the property being verified is that max deviation stays well inside tolerance, not any single exact value. This confirms the algorithm is actually uniform, not just non-crashing.