Requirements clarification (assumptions): Meta needs usable event-level data for short-term product diagnostics and longer-term trend/cohort analysis, must honor deletion/erasure requests, minimize re‑identification risk, and provide auditable compliance.
Proposal summary:
- Retention windows
- Raw event-level (user-id or pseudonymized): 30 days. Purpose: debugging, funnel/quality analysis requiring sequence.
- Short-term aggregated & enriched event store (sessionized, device-level): 180 days.
- Long-term aggregated/retention-safe analytics (daily/week/month metrics, cohorts, RFM): 3 years (or business-justified TTL), stored without identifiers.
- Immutable metadata/audit logs: 7 years encrypted (access controlled) for legal/analysis of pipeline health.
- Anonymization & privacy techniques
- Ingest: deterministic hashing + per-customer salt to create pseudonyms; separate mapping table stored in a secured key vault and accessible only to identity services.
- After 30 days, remove mapping and replace pseudonyms with irreversible one-way hashes + noise via differential privacy for long-term datasets.
- Apply k-anonymity thresholds and suppress low-count cells (< k, e.g., k=10) in exported reports.
- Use differential privacy for ad-hoc queries and cohort exports (epsilon budgeting).
- Aggregation tiers
- Tier 0 (raw): full fidelity, 30-day TTL, highly restricted.
- Tier 1 (sessionized): limited PII, 180 days, used by product analytics teams.
- Tier 2 (aggregates/cohorts): no PII, 3 years, read-only for business intelligence.
- Access controls & governance
- RBAC + attribute-based access control: only authorized roles can query Tier 0/1; require justifications and approvals logged.
- Time-limited credentials for sensitive queries; require manager sign-off and automatic expiration.
- Query-level enforcement: query engine enforces DP/k-anonymity/suppression policy before returning results.
- Automated erasure flow: GDRP/CCPA requests routed to a deletion orchestration service that:
- Removes mapping entries
- Triggers propagation to all downstream stores (CDC invalidation)
- Recomputes affected aggregates where possible
- Records audit trail with timestamps and job IDs
- Tooling & automation
- Ingest & orchestration: Apache Kafka + Kafka Streams for real-time masking; Airflow for scheduled TTL drops and recomputations.
- Storage: Partitioned data lakes (Delta Lake/BigQuery) with row-level TTL enforcement and soft-delete flags.
- Privacy enforcement: integrate OpenDP or Google DP libraries into query layer; use Privacera/Ranger for centralized policy enforcement.
- Access + secrets: HashiCorp Vault for salts/keys; IAM integration for short-lived credentials.
- Compliance automation: policy-as-code (OPA) + CI checks for data schema changes; nightly jobs that run automated compliance tests:
- Validate no PII in Tier 2 tables
- Verify DP budgets not exceeded
- Simulate erasure propagation on random samples
- Monitoring & audit: immutable audit logs (SIEM), alerts for policy violations, monthly privacy risk reports.
- Trade-offs & rationale
- 30-day raw TTL preserves product troubleshooting fidelity while limiting exposure.
- Longer aggregated retention supports product insights and legal requirements.
- Differential privacy + suppression balance utility vs. re-identification risk; budgeted DP prevents unlimited noisy queries.
- Automation reduces human error and provides defensible audits for regulators.
Key metrics to track: percentage of requests honored within legal SLA, number of queries blocked by privacy enforcement, DP budget consumption, time-to-propagate erasure. This approach balances analytics needs with robust privacy, scalable automation, and clear governance.
