Product Analytics Instrumentation and Event Tracking Questions

Situation: Our funnel metrics are unreliable because mobile and web events are inconsistently named, have different schemas, and are missing key attributes.

Goal: Deliver consistent, trusted funnel metrics within 3 months and a sustainable governance model.

Priority plan (practical, time-boxed):

1. Quick wins (0–2 weeks)

• Inventory: run a fast audit using existing analytics UI (Mixpanel/GA/Amplitude) + pull current event list from mobile and web SDKs.
• Stop-gap mapping: create a crosswalk table mapping equivalent events and critical attributes (user_id, session_id, timestamp, product_id).
• Short-term dashboard: publish a “trusted funnel” built from mapped events with clear caveats for stakeholders.

2. Instrumentation conventions & required platform changes (2–8 weeks)

• Define canonical event taxonomy: canonical names, required/optional attributes, data types, and versioning. Publish as a single-source-of-truth doc (Confluence).
• SDK/Platform work: implement a shared tracking library (or wrappers) for web and mobile that enforces schema, auto-populates common fields, and validates payloads client-side.
• Backfill & migration: implement server-side transformation layer (ETL) to normalize historical events into canonical schema for analytics.

3. Verification tests & CI (2–6 weeks concurrent)

• Unit tests in SDK to validate schema and required fields.
• End-to-end smoke tests: automated flows that trigger events and assert they arrive with correct schema and values (use test user IDs).
• Data quality checks in pipeline: anomaly detectors (missing event rates, schema drift, attribute cardinality) that alert to Slack.

4. Governance & long-term (ongoing)

• Analytics guild: establish a cross-functional council (PM, Eng, Data, QA) owning taxonomy and changes.
• Change process: require PRs to update taxonomy, a staging environment for tracking changes, and sign-off SLA.
• Monitoring & SLA: SLAs for instrumentation fixes, weekly dashboards of event health, and quarterly audits.
• Documentation & onboarding: playbooks, examples, and a “how to instrument” checklist for engineers.

Metrics of success:

• 95% of funnel events conform to canonical schema

• Median time-to-fix instrumentation incidents <48 hours
• Stakeholder confidence (survey) >8/10

This approach balances quick impact (trusted dashboard) with durable platform changes (shared SDKs, CI tests) and governance to prevent drift.

Collaboration approach (how we work together)

• Kickoff: PM defines product goals, key experiments/metrics (MAU, conversion, retention) and success criteria. Data Scientist lists required signals and derived metrics. Data Engineer proposes feasible capture points, schema standards, and retention/throughput constraints.
• Joint design session: map user flows to events, agree ownership, enforce naming and field contracts, and add observability hooks (sampling, throttles).
• Sign-off & QA: PM signs behavioral intent, DS signs analytical sufficiency, DE signs implementation and operational constraints.
• Iteration: instrument in staging, run validation tests, roll out progressively (feature flags, percentage rollout).

Pre-release checklist (must be agreed and documented)

1. Event names (convention: snake_case, prefix by domain)
   • e.g., product_view, signup_started, purchase_completed
2. Event payload required fields
   • event_type (string)
   • user_id (stable, anonymized if needed)
   • session_id (UUID)
   • product_id / feature_id (canonical IDs)
   • event_version (int)
   • event_props (object/map with documented typed fields)
3. Identifiers & identity resolution
   • user_id (primary), anon_id (cookie/device), account_id (if B2B)
   • specify format, hashing/encryption rules, PII handling
4. Timestamp format
   • event_timestamp ISO8601 UTC with millisecond precision (e.g., 2025-12-06T14:23:12.123Z)
   • server_received_at for ingestion latency tracking
5. Failure modes & validation
   • missing required field -> drop or route to error topic with reason
   • malformed payload -> reject and store raw for debugging
   • duplicate events -> include event_id (UUID) and idempotency handling
   • high-volume spikes -> sampling policy and backpressure behavior
6. Schema versioning & governance
   • backward-compatible additions allowed; breaking changes require migration plan and version bump
   • central schema registry (JSON Schema/Avro/Protobuf)
7. Observability & SLAs
   • metric dashboards: event volume, schema validation failure rate, ingestion lag
   • alert thresholds (e.g., >1% schema errors)
8. Test plan (signed by DE & DS)
   • unit tests for event serializers/deserializers
   • staging end-to-end tests: simulate user flows producing events; validate presence, schema, timestamps
   • contract tests against schema registry
   • load test for expected peak QPS + 2x
   • backfill test for late-arriving events handling
9. Release & rollback plan
   • feature-flagged rollout (10% → 50% → 100%)
   • monitoring checks at each stage and rollback criteria
10. Documentation & handover

• canonical event spec with examples, required fields, types, allowed values, and sample payloads stored in central doc repo

As Data Engineer I drive schema enforcement, implement ingestion pipelines, provide test harnesses and dashboards, and own rollback/operational playbooks. Data Scientists validate analytical sufficiency; PM ensures product intent and acceptance criteria. Agreement on the checklist before release prevents costly rework and ensures reliable analytics from day one.

High-level approach: define a small consistent event envelope (common fields + event-specific payload). Include versioning, unique event_id for idempotency, minimal PII, and identifiers to support cross-device stitching.

Common envelope (required for all events)

• event_id: string (UUIDv4) - unique per emission
• version: string (e.g., "1.0")
• event_type: string (search, listing_view, add_to_cart, checkout, booking_confirm, cancel)
• occurred_at: string (ISO 8601 UTC)
• user_id: string | null (internal stable user id when authenticated)
• anonymous_id: string (UUID per-install/per-browser)
• session_id: string
• device_id: string | null
• platform: string (web, ios, android)
• context: object (geo: {country, region}, locale, app_version)
• client_ts: integer (ms since epoch)
• revenue: object | null {amount: decimal, currency: string}
• metadata: object (freeform for A/B or experiment tags)

Event-specific required fields

• search: {query: string|null, checkin: date|null, checkout: date|null, guests: int, filters: object}
• listing_view: {listing_id: string, host_id: string|null, price: decimal, currency: string, availability_snapshot: object|null}
• add_to_cart / checkout: {listing_id: string, nights: int, price: decimal, currency: string, fees: decimal}
• booking_confirm: {booking_id: string, listing_id: string, price_total: decimal, currency: string, payment_method: string (token_id), checkin, checkout}
• cancel: {booking_id: string, listing_id: string, cancel_reason: string|null, refund_amount: decimal|null, currency: string|null}

Idempotency & deduplication

• Use event_id (UUID) as primary dedupe key on ingestion. Keep short TTL dedupe cache (e.g., 7–30 days) in ingestion layer (Kafka dedupe, or DB/Redis).
• For critical downstream operations (e.g., transactional booking_confirm), also include business_id (booking_id) + event_type to enforce idempotency in transactional systems.
• Producers should persist last-sent event_id to retry safely.

Cross-device user linking

• Prefer server-side stable user_id when authenticated.
• For anonymous linking, use deterministic identity graph: capture hashed identifiers (email_hash, phone_hash) only after consent. Use salted HMAC with service key (never send raw PII).
• Stitch by user_id primarily; fallback to deterministic hashed identifiers + device fingerprinting + gradual merge rules (merge after successful login).
• Record link events (identity.link) that map anonymous_id -> user_id (with timestamp) to enable historical stitching.

PII minimization & security

• Never store raw email/phone in event payload. Use one-way HMAC(email, salt_key) as email_hash if needed for de-dup/linking.
• Limit free-text fields; redact any user-entered text on client-side before send.
• Store sensitive tokens (payment_method) as token_id only; actual payment details live in PCI-compliant vault.
• Encrypt event payloads in transit (TLS) and at rest; enforce RBAC in analytics DBs and mask PII columns in BI tools.

Versioning

• Use top-level version field. When schema changes (add/remove fields, semantics) increment major/minor. Maintain backward-compatible parsers; emit migration metadata.

Example event JSON (listing_view)

{
  "event_id": "5f2a4b8e-9d3c-4a1f-b2c1-0a1b2c3d4e5f",
  "version": "1.0",
  "event_type": "listing_view",
  "occurred_at": "2025-12-05T14:22:31Z",
  "client_ts": 1733451751000,
  "user_id": "user_12345",
  "anonymous_id": "anon_98765",
  "session_id": "sess_abc123",
  "device_id": "device_xyz",
  "platform": "web",
  "context": {"country":"US","locale":"en-US","app_version":"web-2025.12"},
  "payload": {
    "listing_id": "lst_54321",
    "host_id": "host_999",
    "price": 135.00,
    "currency": "USD",
    "availability_snapshot": {"2026-01-01": true, "2026-01-02": false}
  },
  "metadata": {"experiment":"homepage_redesign"}
}

Why this fits BI needs

• Consistent envelope simplifies ingestion, schema registry, and transforms for dashboards.
• event_id + booking_id enable accurate funnel metrics and deduplication.
• Minimal PII and hashed identifiers allow cross-device stitching while complying with privacy rules.
• Versioning lets BI pipelines evolve without breaking historical reports.

An "event" in product analytics is a discrete, instrumented record of a user action or system occurrence (e.g., "Clicked Sign Up" at timestamp T by userID X). It captures raw, contextual data (event name, properties like page, variant, device) and is the primary input for analysis. An aggregated metric is a computed summary (e.g., daily sign-ups = 1,234) derived from many events - useful for reporting but lossy for detailed analysis.

Three concrete event examples for onboarding:

• sign_up_submitted {user_id, method:email|google, timestamp, utm_campaign}
• onboarding_step_completed {user_id, step_id, duration_seconds, timestamp}
• welcome_email_opened {user_id, email_id, timestamp, client}

Two example metrics derived from those events for an onboarding funnel:

• Conversion rate by step = count(onboarding_step_completed step_n) / count(onboarding_step_completed step_n-1)
• Median time-to-complete onboarding = median(sum(duration_seconds) per user from first step to final step)

Why choice matters:

• Events preserve granularity and enable cohort, funnel, and root-cause analyses (filter by channel, device, or property). Aggregated metrics are efficient for dashboards and SLA tracking but hide heterogeneity (e.g., slow sub-cohort). Instrument the right events so analysts can recompute metrics, slice by dimensions, and avoid re-instrumentation later.

Framework: balance speed for analysis, reproducibility, cost, and compliance. Proposal below includes justification, deletion/archival process, and operational controls.

Retention policy:

• Raw events (clicks, impressions, assignment logs): keep hot for 90 days (fast queries for debug, attribution), then move to cold archive for 12 months total; after 12 months either delete or move to long-term archive (Glacier) for 3–5 years if legally required. Reason: raw events are heavy and mainly needed short-term for debugging and reprocessing; 90 days covers most experiment windows and rollouts.
• Aggregated experiment results (per-experiment metrics, treatment summary tables): keep for 3 years. Reason: supports business reporting, cohort comparisons, regulatory audits and product decisions across yearly cycles; small footprint so retention is inexpensive.
• Derived SQL tables / feature flag materialized views (operational lookups used by dashboards): keep rolling 12–24 months depending on rebuildability. If fully reproducible from raw events, 12 months; if costly to regenerate, 24 months.

Deletion & archive process:

• Automated lifecycle policies: implement data lifecycle in object store (S3 lifecycle to Intelligent-Tiering → Glacier → expire) and in warehouse (partition-level TTL jobs).
• Graceful transition: on archiving, register snapshots in data catalog with metadata (schema, time range, checksum) and flag as read-only.
• Reproducibility guardrails: ensure pipeline DAGs and SQL scripts are versioned; before raw deletion verify aggregated tables can be recomputed from archived raw or mark experiments pinned.
• Compliance & privacy: implement per-user deletion hooks to remove PII; log deletions for audits.
• Monitoring & rollback: alerts for failed expirations, monthly audits, and ability to restore from Glacier within SLA for legal requests.

Stakeholders: engineering owns implementation, analytics owns aggregation, legal/infosec for retention windows, product for policy. Metrics: storage cost, average time to restore archived data, number of times archived raw needed for reprocess (should be rare).