Product Analytics Instrumentation and Event Tracking Questions

1. Connectivity & batching

• Difference: Mobile clients often have intermittent connectivity and send batched events; web is usually online and sends events in near real-time.
• Schema & sampling impact: Schema must include batch metadata (batch_id, client_time, send_time, network_type) and allow deduplication. For sampling, prefer client-side adaptive sampling on mobile to avoid spikes when reconnecting (e.g., sample less during bulk uploads) while applying server-side rate limiting for web.

2. Session model & lifetime

• Difference: Mobile apps run longer, can generate background/periodic events; web sessions are shorter and page-focused.
• Schema & sampling impact: Include persistent device_id, app_version, foreground/background flag, and session_start/end. For mobile, sample lower-frequency background telemetry more aggressively (e.g., 1% for heartbeat) but keep full fidelity for foreground user actions. On web, prioritize page-view and click events with higher fidelity.

3. Rich device/context signals & privacy constraints

• Difference: Mobile provides richer device/context (OS, battery, sensors) but tighter privacy and permission constraints; web has browser/user-agent and cookie limitations.
• Schema & sampling impact: Design nullable context blocks and explicit consent flags; store sensitive fields separately or hashed. Apply differential sampling: keep full context for a stratified subset of mobile users to analyze device-specific bugs, sample or drop extra context for others to reduce storage and comply with privacy.

Overall: tag events with provenance and sampling_reason to make downstream analysis aware of bias; prioritize high-signal business metrics for full retention and apply targeted sampling elsewhere.

Summary:
Sampling at ingestion drops or reduces data when it first arrives; sampling at query/analytics time retains raw/near-raw data and samples only when answering a query. Each approach trades cost, flexibility, and analytic accuracy differently.

Sampling at ingestion

• When to choose: very high-volume telemetry with tight storage/ingest budget, stable/known KPIs, and mostly pre-defined aggregate dashboards (e.g., simple counts, rates).
• Pros: big savings in storage, indexing, and long-term compute; simpler pipeline; lower downstream costs.
• Cons: irreversible loss of fidelity - you can’t run new analyses later that require discarded data; introduces sampling bias if not carefully designed.
• OLAP implications: lower cost but reduced accuracy and higher variance for low-frequency events; need to use appropriate weighting/estimation and document the sampling scheme for correct interpretation.

Sampling at query/analytics time

• When to choose: exploratory analytics, anomaly detection, compliance/audit needs, or when product teams expect evolving questions about the data.
• Pros: maximum analytical flexibility; unbiased results if sampling is done correctly at query time; supports ad-hoc, retrospective analyses.
• Cons: higher storage and ingestion costs; larger query compute (may require more aggressive query sampling/approximation techniques).
• OLAP implications: better accuracy/fidelity and ability to compute confidence intervals per query, but cost scales with retention and query complexity.

Practical patterns / hybrid options

• Store full data for a small fraction (100% for 1% of users or errors) + ingestion-sample the rest - preserves rare-event analysis while cutting cost.
• Use stratified/deterministic sampling keyed by user/session/error severity to reduce bias.
• Retain raw for a short hot window (e.g., 7 days) then sample older data.
• Attach sampling metadata (sample rate, key strata) at ingestion so analysts can weight results correctly.

Recommendation (product POV)

• If customers require deep, evolving analytics or regulatory auditability, favor query-time sampling (invest in cost controls: tiered storage, cold paths).
• If cost is a dominant constraint and analytics needs are fixed and well-understood, use ingestion sampling with careful design, clear documentation, and hybrid safeguards to protect rare-event visibility.

Collaboration approach (how we work together)

• Kickoff: PM defines product goals, key experiments/metrics (MAU, conversion, retention) and success criteria. Data Scientist lists required signals and derived metrics. Data Engineer proposes feasible capture points, schema standards, and retention/throughput constraints.
• Joint design session: map user flows to events, agree ownership, enforce naming and field contracts, and add observability hooks (sampling, throttles).
• Sign-off & QA: PM signs behavioral intent, DS signs analytical sufficiency, DE signs implementation and operational constraints.
• Iteration: instrument in staging, run validation tests, roll out progressively (feature flags, percentage rollout).

Pre-release checklist (must be agreed and documented)

1. Event names (convention: snake_case, prefix by domain)
   • e.g., product_view, signup_started, purchase_completed
2. Event payload required fields
   • event_type (string)
   • user_id (stable, anonymized if needed)
   • session_id (UUID)
   • product_id / feature_id (canonical IDs)
   • event_version (int)
   • event_props (object/map with documented typed fields)
3. Identifiers & identity resolution
   • user_id (primary), anon_id (cookie/device), account_id (if B2B)
   • specify format, hashing/encryption rules, PII handling
4. Timestamp format
   • event_timestamp ISO8601 UTC with millisecond precision (e.g., 2025-12-06T14:23:12.123Z)
   • server_received_at for ingestion latency tracking
5. Failure modes & validation
   • missing required field -> drop or route to error topic with reason
   • malformed payload -> reject and store raw for debugging
   • duplicate events -> include event_id (UUID) and idempotency handling
   • high-volume spikes -> sampling policy and backpressure behavior
6. Schema versioning & governance
   • backward-compatible additions allowed; breaking changes require migration plan and version bump
   • central schema registry (JSON Schema/Avro/Protobuf)
7. Observability & SLAs
   • metric dashboards: event volume, schema validation failure rate, ingestion lag
   • alert thresholds (e.g., >1% schema errors)
8. Test plan (signed by DE & DS)
   • unit tests for event serializers/deserializers
   • staging end-to-end tests: simulate user flows producing events; validate presence, schema, timestamps
   • contract tests against schema registry
   • load test for expected peak QPS + 2x
   • backfill test for late-arriving events handling
9. Release & rollback plan
   • feature-flagged rollout (10% → 50% → 100%)
   • monitoring checks at each stage and rollback criteria
10. Documentation & handover

• canonical event spec with examples, required fields, types, allowed values, and sample payloads stored in central doc repo

As Data Engineer I drive schema enforcement, implement ingestion pipelines, provide test harnesses and dashboards, and own rollback/operational playbooks. Data Scientists validate analytical sufficiency; PM ensures product intent and acceptance criteria. Agreement on the checklist before release prevents costly rework and ensures reliable analytics from day one.

High-level approach: define a small consistent event envelope (common fields + event-specific payload). Include versioning, unique event_id for idempotency, minimal PII, and identifiers to support cross-device stitching.

Common envelope (required for all events)

• event_id: string (UUIDv4) - unique per emission
• version: string (e.g., "1.0")
• event_type: string (search, listing_view, add_to_cart, checkout, booking_confirm, cancel)
• occurred_at: string (ISO 8601 UTC)
• user_id: string | null (internal stable user id when authenticated)
• anonymous_id: string (UUID per-install/per-browser)
• session_id: string
• device_id: string | null
• platform: string (web, ios, android)
• context: object (geo: {country, region}, locale, app_version)
• client_ts: integer (ms since epoch)
• revenue: object | null {amount: decimal, currency: string}
• metadata: object (freeform for A/B or experiment tags)

Event-specific required fields

• search: {query: string|null, checkin: date|null, checkout: date|null, guests: int, filters: object}
• listing_view: {listing_id: string, host_id: string|null, price: decimal, currency: string, availability_snapshot: object|null}
• add_to_cart / checkout: {listing_id: string, nights: int, price: decimal, currency: string, fees: decimal}
• booking_confirm: {booking_id: string, listing_id: string, price_total: decimal, currency: string, payment_method: string (token_id), checkin, checkout}
• cancel: {booking_id: string, listing_id: string, cancel_reason: string|null, refund_amount: decimal|null, currency: string|null}

Idempotency & deduplication

• Use event_id (UUID) as primary dedupe key on ingestion. Keep short TTL dedupe cache (e.g., 7–30 days) in ingestion layer (Kafka dedupe, or DB/Redis).
• For critical downstream operations (e.g., transactional booking_confirm), also include business_id (booking_id) + event_type to enforce idempotency in transactional systems.
• Producers should persist last-sent event_id to retry safely.

Cross-device user linking

• Prefer server-side stable user_id when authenticated.
• For anonymous linking, use deterministic identity graph: capture hashed identifiers (email_hash, phone_hash) only after consent. Use salted HMAC with service key (never send raw PII).
• Stitch by user_id primarily; fallback to deterministic hashed identifiers + device fingerprinting + gradual merge rules (merge after successful login).
• Record link events (identity.link) that map anonymous_id -> user_id (with timestamp) to enable historical stitching.

PII minimization & security

• Never store raw email/phone in event payload. Use one-way HMAC(email, salt_key) as email_hash if needed for de-dup/linking.
• Limit free-text fields; redact any user-entered text on client-side before send.
• Store sensitive tokens (payment_method) as token_id only; actual payment details live in PCI-compliant vault.
• Encrypt event payloads in transit (TLS) and at rest; enforce RBAC in analytics DBs and mask PII columns in BI tools.

Versioning

• Use top-level version field. When schema changes (add/remove fields, semantics) increment major/minor. Maintain backward-compatible parsers; emit migration metadata.

Example event JSON (listing_view)

{
  "event_id": "5f2a4b8e-9d3c-4a1f-b2c1-0a1b2c3d4e5f",
  "version": "1.0",
  "event_type": "listing_view",
  "occurred_at": "2025-12-05T14:22:31Z",
  "client_ts": 1733451751000,
  "user_id": "user_12345",
  "anonymous_id": "anon_98765",
  "session_id": "sess_abc123",
  "device_id": "device_xyz",
  "platform": "web",
  "context": {"country":"US","locale":"en-US","app_version":"web-2025.12"},
  "payload": {
    "listing_id": "lst_54321",
    "host_id": "host_999",
    "price": 135.00,
    "currency": "USD",
    "availability_snapshot": {"2026-01-01": true, "2026-01-02": false}
  },
  "metadata": {"experiment":"homepage_redesign"}
}

Why this fits BI needs

• Consistent envelope simplifies ingestion, schema registry, and transforms for dashboards.
• event_id + booking_id enable accurate funnel metrics and deduplication.
• Minimal PII and hashed identifiers allow cross-device stitching while complying with privacy rules.
• Versioning lets BI pipelines evolve without breaking historical reports.

Requirements:

• Capture user events reliably on mobile even when offline
• Preserve event order per user/session, avoid duplicates, allow retries
• Low bandwidth (batch/compress), configurable latency vs accuracy
• Reconcile client/server events into analytics warehouse with eventual consistency

High-level architecture:
Mobile SDK (local buffer + metadata) → Ingestion API / Gateway → Event processing (idempotent writer, dedupe store) → Stream (Kafka) → ETL → Analytics warehouse

Mobile-side design:

• Local buffering: append-only persistent queue (SQLite or file) with per-event metadata: event_id (UUID), client_seq (monotonic per-session), timestamp, device_id, user_id, retry_count.
• Batching & compression: send batches (size/time thresholds) and gzip/Protobuf to reduce payloads; include batch_checksum.
• Retry & backoff: exponential backoff with jitter; immediate retry only for transient network; persist batches until ACK.
• Deduplication tokens: server must ACK batch IDs; client deletes events only after ACK; include event_id for server-side dedupe.
• Sequence numbers & ordering: client_seq ensures per-session ordering; server stores highest-seen seq per device+session to drop/flag out-of-order if necessary.
• Conflict resolution: client events vs server-generated events use timestamps + source priority; attach vector (client_seq, server_seq) to detect duplicates and causal order; for critical idempotent actions include an operation_id.

Server-side processing:

• Idempotent ingestion: dedupe by event_id using TTL-backed dedupe store (Redis with sufficient retention).
• Ordering: use client_seq for ordering in per-key streams; if out-of-order, buffer short window then accept or mark as late.
• Persist canonical event stream to durable log (Kafka) with partition key user_id or device_id.

Reconciliation & analytics warehouse:

• ETL consumes Kafka, applies dedupe, keeps last-wins or timestamp-based conflict resolution, materializes event_time and ingestion_time.
• Maintain staging + fact tables: raw_events (ingest_time, source, event_id), normalized_events (deduped), and late_events table.
• Periodic reconciliation job: reprocess late-arriving events (upsert into facts), recalc affected aggregates or maintain incremental corrections (delta tables, event-time windowing).
• Track data quality: counters for duplicate drops, late events, missing sequences; generate alerting.

Implications for metrics:

• Accuracy: eventual consistency means metrics can change after late arrivals/corrections - provide confidence windows (e.g., 24–48 hours) and surface “finalized” vs “preliminary” metric states.
• Latency: batching and backoff increase delivery latency; tuning batch thresholds trades freshness for bandwidth/efficiency. Use real-time critical flags for low-latency events (smaller batches).
• Bias: Offline users create right-censored timestamps (client_time vs server_time) - normalize using event_time with TTL for late arrivals; be cautious of user clock skew.
• Determinism: deduplication and idempotency improve accuracy but may drop events if keys collide or retention is insufficient - monitor dedupe rates.

Recommendations for analysts:

• Use event_time for Historical analysis; filter on ingestion_time when measuring freshness.
• Expose late-arrival metrics and correction logs to stakeholders.
• Maintain SLA for metric stabilization and document 'finalization' window.