Goal: create a single, reliable identity layer that supports accurate analytics while respecting privacy and minimizing engineering risk.
High-level approach
- Build a deterministic identity graph first (high precision), then augment with probabilistic linking (coverage) where acceptable. Treat identity as a product: define SLAs (match accuracy, latency), owner (analytics PM), and KPIs (merge precision/recall, percent unified sessions).
Deterministic joins (primary backbone)
- Normalize and canonicalize identifiers: hashed_email (web), device_id (mobile), anonymous_widget_id.
- Capture and persist explicit, consented mappings via login/verify events (e.g., user logs in on mobile → map device_id ↔ hashed_email). Use a canonical “person_id” (internal stable UUID).
- Store mappings in an append-only identity store (time-versioned) and expose via a low-latency read API for analytics and attribution.
Probabilistic linking (coverage for gaps)
- Use probabilistic models only when deterministic links absent: device fingerprinting, behavioral similarity, IP/time heuristics. Score links with confidence and keep them separate from deterministic matches.
- Mark downstream analytics with confidence bands; avoid deterministic attribution decisions (billing, legal) based on probabilistic links.
Privacy & compliance trade-offs
- Default to minimal linking without explicit consent. Keep hashed emails as one-way hashes; avoid storing raw PII in analytics tier.
- Support consent flags and GDPR/CCPA erase flows: identity store must honor data deletion and unmapping.
- Document retention policies, encryption at rest/in transit, RBAC for identity access, and an audit trail for merges/unmerges.
Stepwise implementation roadmap
- Discovery (2–4 wks): inventory sources, fields, consent status; define person_id schema and success metrics.
- Deterministic MVP (6–8 wks): implement canonical person_id, capture login/consent events, build identity store and simple API; route analytics to use person_id.
- Reconciliation & QA (4 wks): backfill historical deterministic joins, measure precision/coverage, provide dashboards.
- Probabilistic pilot (6–10 wks): build scoring models, label evaluation set, surface confidence scores, gate by product teams.
- Governance & Privacy hardening (ongoing): consent management, deletion workflows, access controls, legal signoff.
- Rollout & monitoring (ongoing): expose unified metrics, monitor merge rates, false-merge alerts, iterate.
Risks & mitigations
- False merges: require manual review queue + conservative thresholds.
- Regulatory pushback: default opt-out for linking; legal review before enabling probabilistic joins.
- Engineering complexity: start with deterministic, reuse existing event pipeline, incremental releases.
Success metrics
- % of active users unified deterministically
- Reduction in duplicate user counts
- Attribution accuracy improvement (A/B or holdout test)
- Number of privacy incidents = 0
This balances accuracy, coverage, and compliance while giving product teams usable person-level analytics without exposing raw PII.
