Product Analytics Instrumentation and Event Tracking Questions

Step 1 - Triage / assess impact

• Quantify scope: percent of events with NULL timestamps, future timestamps, by source/system, event type, user, date.
• Prioritize dashboards/reports affected (business-critical first).

Step 2 - Short-term fixes (triage + quick remediation)

• Exclude bad rows in downstream reports until fixed: add WHERE event_timestamp IS NOT NULL AND event_timestamp <= CURRENT_TIMESTAMP.
• Backfill obvious missing timestamps when possible (e.g., use ingestion_time or event_id-based mapping) and mark rows as imputed.
• Add a temporary flag column (timestamp_quality = 'null'|'future'|'ok') so stakeholders see data quality.
  Example SQL to filter and flag:

SELECT *,
  CASE
    WHEN event_timestamp IS NULL THEN 'null'
    WHEN event_timestamp > CURRENT_TIMESTAMP THEN 'future'
    ELSE 'ok'
  END AS timestamp_quality
FROM events
WHERE event_timestamp IS NOT NULL AND event_timestamp <= CURRENT_TIMESTAMP;

Step 3 - Fix root causes (medium-term)

• Trace by source (API, SDK, ETL job) to identify where NULLs/future values are created (client-side bugs, timezones, incorrect default values).
• Implement backfill scripts with validation and logging; keep original raw data immutable and write cleaned table.

Step 4 - Preventive measures (long-term)

• Data contracts: require timestamp field, format, timezone, and validation at producer level.
• Validation at ingestion: reject or quarantine records with NULL/future timestamps; send alerts.
• Schema and constraints: enforce NOT NULL and CHECK (event_timestamp <= CURRENT_TIMESTAMP + allowed_skew) where appropriate.
• Instrument SDKs/clients to set timestamps reliably (use server-side timestamp when client unreliable).
• Monitoring & alerting: daily data-quality jobs that track volume and anomalies, with automated alerts to owners.
• Documentation & runbooks: define acceptable imputation rules, SLA for fixes, owner contacts.

Step 5 - Communicate

• Share impact, temporary mitigations, and timeline with stakeholders; surface the timestamp_quality flag in dashboards until fully resolved.

This approach minimizes reporting disruption, fixes data at source, and prevents recurrence through contracts, validation, and monitoring.

High-level principle: split responsibilities by intent - Product owns the “what” (metrics & use-cases), Data owns the “definition/contract” and validation, Engineering owns the “implementation, delivery and runtime reliability.” Tie with one accountable owner per decision to avoid paralysis.

RACI (Activities → Product / Data / Engineering):

• Instrumentation design (events, metrics, tags)

  • Responsible: Product (defines metrics/use-cases)
  • Accountable: Data (schema/design approval)
  • Consulted: Engineering (feasibility, SDKs)
  • Informed: Stakeholders (PMs, Eng leads)

• Schema changes (new fields, renames)

  • Responsible: Data (maintains schema registry & contracts)
  • Accountable: Data
  • Consulted: Product (use-case impact), Engineering (implementation)
  • Informed: Consumers (downstream teams)

• Pipeline maintenance (ETL, streaming jobs)

  • Responsible: Engineering (platform/SRE team)
  • Accountable: Engineering
  • Consulted: Data (transform correctness), Product (priority)
  • Informed: Incident channels, Data consumers

• Data quality alerts (validation, anomaly detection)

  • Responsible: Data (write checks, maintain rules)
  • Accountable: Data
  • Consulted: Engineering (signal routing, remediation playbooks)
  • Informed: Product (impact on KPIs), On-call engineers

• SLA monitoring (latency, freshness, availability)

  • Responsible: Engineering (instrumentation + monitoring)
  • Accountable: Engineering
  • Consulted: Data (SLO definitions), Product (business SLAs)
  • Informed: Execs, Product ops

Operationalizing handoffs:

1. Schema Registry + Change Proposal: all schema changes require a Schema Change Proposal (simple PR) in a central registry with impact matrix. Data reviews and approves within SLA (48–72h).
2. Instrumentation Playbook: product authors an Instrumentation Spec template (event name, owner, business meaning, examples, retention), engineering attaches implementation PR referencing spec; Data signs off before merge.
3. CI/CD & Tests: instrumentation SDKs include unit tests that surface missing required fields; pipeline accepts only validated schema versions.
4. Change Windows & Deprecation Policy: versioned fields with clear deprecation timelines (e.g., announce 90d, deprecate 30d later); automated consumer breakage reports.
5. On-call & Runbooks: Data owns detector rules; Engineering owns alerting channels and runbooks. Alerts route to on-call pager with escalation and playbook links.
6. Governance Rhythm: weekly triage for urgent fixes, monthly data governance sync (Product + Data + Eng) and quarterly review of contracts, KPIs, and ownership shifts.
7. Dashboard & Audit Trail: public dashboard with pipeline SLAs, Q/A metrics, recent schema changes and consumer impact logs. All handoffs logged in ticket/PR for audit.

This model balances clear accountability while enabling collaborative consultation; it enforces guardrails (registry, tests, SLAs) so handoffs are explicit, automated where possible, and auditable.

Event-level

• event_name: string, required - "checkout" (fixed value to identify event)
• event_timestamp: iso8601 string, required - UTC time of checkout for ordering and sessionization
• platform: enum (web/iOS/android), required - source platform for segmentation
• event_id: string (UUID), required - unique event identifier for deduplication

User / Identity (PII considerations: avoid raw PII; prefer hashed IDs and consent flags)

• user_id: string, optional - stable hashed user identifier (PII: do not send raw email/phone; hash+salt)
• anonymous_id: string, required - client-generated anonymous id for unauthenticated users
• user_email_hashed: string, optional - SHA256 hashed email only if consented; flag consent required
• is_logged_in: boolean, required - whether user was authenticated

Revenue & Order

• order_id: string, required - unique order identifier
• currency: string (ISO 4217), required - e.g., USD
• order_total: number (decimal), required - total charged to customer (includes taxes & shipping)
• subtotal: number (decimal), optional - sum of item prices before tax/discounts
• tax_amount: number (decimal), optional
• shipping_amount: number (decimal), optional
• discount_amount: number (decimal), optional - total discounts applied
• payment_method: string, optional - tokenized or high-level (e.g., card, apple_pay); avoid card details

Item-level (array of objects)

• items: array<object>, required - list of purchased items
  • item_id: string, required - product SKU or id
  • name: string, optional - product name
  • category: string, optional - product category/line for aggregation
  • price: number (decimal), required - unit price charged
  • quantity: integer, required
  • item_subtotal: number (decimal), optional - price * quantity after item-level discounts
  • variant: string, optional - size/color/etc.
  • is_gift: boolean, optional

Campaign / Attribution

• campaign_source: string, optional - e.g., google, facebook
• campaign_medium: string, optional - e.g., cpc, email
• campaign_name: string, optional - campaign identifier
• campaign_term: string, optional - search keyword
• campaign_content: string, optional - creative id or variant
• attribution_model: string, optional - last_click/first_click/custom

Context & Device

• locale: string, optional - user locale (en-US)
• ip_anonymized: string, optional - first 3 octets zeroed if collected (PII)
• user_agent: string, optional - or parsed device/os fields
• app_version: string, optional - app version for mobile
• referrer: string, optional - sanitized URL for external source (strip query PII)

Metadata

• integrations: object, optional - flags for downstream tool routing
• schema_version: string, required - versioning for future changes

Notes on PII & privacy:

• Never send raw emails, phone numbers, or payment card data.
• If using hashed PII (email), store hashing method and salt policy; only send if user consented.
• Prefer stable hashed user_id and anonymous_id for identity stitching.
• Ensure currency and numeric fields use consistent precision and timezone normalization.

Policy overview: a concise, consistent naming convention for events and properties to make instrumentation discoverable, queryable, and reliable for analysts and engineers.

Event naming (pattern)

• Use: domain.action.object.context
• Lower_snake_case, no spaces, max 4 segments.
• Examples: checkout.start_cart, product.view_detail, auth.success_web
• Action verbs: start/complete/click/view/submit/error

Property naming (types & units)

• Lower_snake_case. Use a type-signaling convention: identifier fields end with an id suffix (user_id, session_id, order_id), not an id prefix; boolean fields start with an is_ prefix (is_logged_in); count fields start with a num_ prefix (num_items).
• Types: string, integer, float, boolean, timestamp, enum.
• Units: append the actual stored unit to the name, and never let the name claim a different unit than what is stored: price_cents (integer, USD cents), not price_usd, since price_usd would imply a dollar-denominated value; duration_ms (integer, ms); distance_km (float, km).
• Example properties: user_id (string), session_id (string), price_cents (integer), duration_ms (integer), is_logged_in (boolean), plan_tier (enum: free/premium)

Versioning

• Add event_version integer property (event_version: 1). Increment when schema changes incompatibly.
• For minor additive changes, keep same version and document optional fields.

Deprecation

• Mark deprecated in catalog with deprecation_date and replacement_event.
• Keep emitting deprecated events for a grace period (e.g., 90 days) while consumers migrate.
• After grace period, stop emitting and archive schema.

Governance & docs

• Maintain a searchable catalog (with examples, owners, versions).
• Require instrumentation review before release.

How this helps

• Predictable names speed SQL discovery and joins.
• Typed properties reduce interpretation errors and ensure correct aggregations.
• Versioning + deprecation avoids silent schema breaks, gives analysts time to adapt.

Goal: ensure product instrumentation (events, schemas, attributes) stays complete and correct, detect regressions quickly after releases, and prevent bad releases from reaching users.

1. Define coverage metrics

• Per page/feature catalog: list mandatory events + required attributes (e.g., "checkout.page_view", "checkout.submit" with user_id, order_total). Store as source-of-truth YAML/JSON in repo.
• Coverage KPIs: event presence rate (fraction of sessions/pages with the event), attribute completeness (non-null %), schema conformance rate (percentage matching expected schema), and latency (time from event emission to pipeline ingestion).
• Set SLO targets (example): event presence ≥ 98% for primary flows, attribute completeness ≥ 99%, schema conformance ≥ 99.9%.

2. Automated checks in CI

• Schema linting: pre-commit/CI job validates instrumentation spec files (JSON schemas) and runs a schema linter against any changed event definitions. Fail the PR on violations.
• Unit tests: mock components that emit events include tests asserting the correct event name, attributes, and types.
• Contract tests: producer-side tests that serialize one sample event per mandatory event and validate against central schema registry.
• Gate: block merge if CI fails; require approvals for changes to mandatory-event lists.

3. Synthetic tests (end-to-end)

• Create lightweight synthetic users that execute critical flows in staging and production (e.g., sign-up, search, purchase) and emit expected events.
• Tests validate: events received in event bus/analytics pipeline, fields present, and values reasonable. Run on deploy and nightly.
• Use canary runs: small percent of production traffic mirrored/duplicated to test pipelines without affecting customers.

4. Baseline comparisons & regression detection

• Maintain rolling baselines (7/30-day) per metric. After each release, compute:
  • Relative drop: (baseline - post-release)/baseline
  • Statistical test for significance (e.g., binomial test for presence rates, proportion z-test) to avoid false positives from noise.
• Automated anomaly detection: time-series models (Seasonal-ARIMA, Prophet) or simple moving-median with MAD to detect deviations beyond expected variance.
• Example rule: if event presence falls >5 percentage points AND p-value < 0.01 OR outside 3σ of expected, mark as regression.

5. Alerting rules & triage

• Multi-tier alerts:
  • Critical (blocking): mandatory event missing entirely or schema conformance < 95% for core flows → Page/Slack + pager to on-call SRE/owner.
  • High: event presence drop >5pp with statistical significance → Slack + ticket.
  • Medium: attribute completeness 2–5pp drop → dashboard + email.
• Include context in alert: affected event, baseline, current value, roll-forward diff, recent deploys, PRs touching instrumentation.
• Auto-create JIRA incident with runbook links.

6. Release gating

• Pre-release: require CI contract/schema checks passing, synthetic tests green in staging.
• Pre-promote: run synthetic tests in canary; if critical alerts fire, block promotion automatically via CD (job fails).
• Post-release monitoring window: for X hours after deploy, enforce stricter thresholds (e.g., require event presence ≥ 99%). If critical regression detected, automated rollback or feature flag disable and notify stakeholders.
• Policy: instrumentation regressions on mandatory events are treated like functional regressions.

7. Operational practices & ownership

• Versioned schema registry and change-review process; owners per feature for on-call.
• Dashboards showing per-feature coverage, trends, and recent deploy metadata.
• Quarterly audits: identify drift, add missing mandatory events, and update baselines.
• Runbooks: steps to validate issue, hotfix, rollback, and communication templates.

Why this works

• Combines preventative (CI, linting, contract tests), detective (synthetic tests, baselines, statistical alerts), and corrective (gating, rollback) controls.
• Statistical tests reduce noise; strict gates on mandatory events protect downstream ML/data quality and business metrics.
• Clear ownership and automated triage shorten MTTR and keep analytics reliable for modeling and decisions.