Data Subject Rights and Request Handling Questions

Immediate blockers

• Stop new ingestion of data for affected market until compliance rules are enforceable: add geo/market flags and a kill-switch in ingestion pipelines.
• Implement a fast lookup for deletion requests: build/enable an API that maps identifiers (user ID, email hash) to storage locations (raw events, derived tables, S3 paths, Kafka offsets).
• Audit current retention: run a discovery job to report where personal data exists (databases, parquet, logs, backups) and surface high-risk stores to Legal/PM within 48–72 hours.

Medium-term engineering work (4–12 weeks)

• Build a delete-or-erase pipeline:
  • Tier 1: logical deletion—mark records as erased and filter queries (recommended short-term).
  • Tier 2: physical deletion—expunge PII from raw and derived datasets using lineage-aware jobs (Spark jobs that rewrite partitions, delete objects, re-ingest cleaned aggregates).
  • Maintain immutable audit logs for requests (no PII stored in logs).
• Add metadata & lineage: tag datasets with sensitivity labels and market scope; integrate with data catalog.
• Automate request processing: queueing, authentication, idempotency, SLA tracking, retry/backfill for historical joins.
• Update backups & retention policies and implement safe deletion for backups.

Policy decisions to propose

• Definition of "personal data" scope and acceptable pseudonymization vs full deletion.
• SLA for fulfilling requests (e.g., 30 days) and whether to prioritize logical vs physical deletion to meet deadlines.
• Tradeoffs on analytics fidelity vs compliance: agree which aggregated datasets can remain.

Communicating tradeoffs

• For Product: present options with impact metrics—time-to-complete, % of analytics lost, engineering effort, cost—so PMs can prioritize features vs compliance speed.
• For Legal: show technical constraints and propose a phased plan with mitigations (logical deletion + audit) to meet legal intent quickly while scheduling full physical purge.
• Deliverables: compliance runbook, risk matrix, milestones, and a demo of request workflow.

Why this approach

• Balances urgent compliance (kill-switch, logical deletion) with sustainable engineering (lineage, automated deletion), provides measurable risks and timelines, and keeps stakeholders aligned on tradeoffs.

Requirements & constraints:

• Functional: on explicit user request, delete all personal data within 24 hours across ingestion, streaming state, historical stores, and backups.
• Non-functional: minimal analytic disruption, acceptable cost, auditable proof, scalable to millions of users, eventual consistency allowed within 24h window.

High-level approach:

• Treat deletion as a first-class operation propagated through pipelines using a Delete Command (immutable event): {user_id, request_id, timestamp, scope}.
• Build dataflow where Delete Commands flow through the same messaging bus (Kafka/ PubSub) and drive selective deletion/obfuscation in each store.

Architecture & component changes:

1. Ingestion

• Add request validation and dedupe layer; any incoming event with PII tagged is annotated with user_id.
• On delete request, emit Delete Command into Kafka topic with high priority/compact retention.
• Maintain schema registry to ensure PII fields are marked for removal.

2. Streaming state stores (e.g., Kafka Streams / Flink state)

• Materialize keyed state by user_id. Subscribe to Delete Commands; on receipt, either:
  • Expire keys immediately (state.remove(user_id)), or
  • Overwrite with tombstone/NULLed PII while keeping non-PII metrics.
• Use compacted topics for state snapshots so tombstones propagate and compact removes keys.

3. Historical data stores (data lake / warehouse)

• Store raw events partitioned and optionally encrypted; maintain an index mapping user_id → list of partitions/rowIDs (secondary index table).
• Two-tier deletion:
  • Logical deletion: add Delete Command to a control table and rewrite metadata so queries exclude deleted user_id (views/row-level security) for immediate effect.
  • Physical deletion: schedule targeted rewrites (ALTER/REWRITE/OPTIMIZE) of affected files within 24 hours using the index to locate files; use predicate pushdown to avoid full-scan rewriting. For columnar files (Parquet/ORC), rewrite only files containing PII rows.
• For analytical completeness, keep aggregated metrics that are non-identifying; if aggregates include small cohorts, apply differential privacy or suppress to avoid re-identification.

4. Backups / Snapshots

• Versioned backups must support deletion: maintain a deletion journal that records which backups contain the user’s PII.
• Two options:
  • Fast: mark backups as containing PII and block restore of ranges including that user until rewrites complete. Maintain backup-level metadata to filter during restores.
  • Strong compliance: implement backup rewrites: use incremental restore-and-rewrite for backup objects that contain the user, then re-ingest. This is costly; consider retention policy minimization (shorter retention) to reduce exposure.

Propagation & orchestration

• Orchestrator service (Delete Orchestrator) receives request, emits Delete Command, tracks progress across stores, retries failed steps, enforces SLA (24h), and records audit trail.
• Idempotency and strong dedupe for commands.
• Use priority queues / separate delete processors to ensure deletes outrank heavy ingestion.

Verification & Auditing

• Audit log with immutable proof of deletion events and per-store confirmations.
• Automated verification jobs:
  • Query-based: attempt to select any PII rows for user_id across stores; expect zero.
  • File index verification: check that physical files rewritten do not contain user_id.
  • Spot checks and cryptographic proofs: store hashes of PII per row to verify removal (store salted hashes; deletion removes hash or stores deletion marker).
• Expose deletion status API to compliance and user-facing UI.

Trade-offs

• Performance: immediate state removal is cheap; physical rewrites of large historical partitions are expensive and can impact cluster IO. Mitigate with targeted rewrites and background throttling.
• Cost: maintaining indexes and running rewrite jobs + backup rewrites increases compute and storage cost. Shorter backup retention reduces cost/risk.
• Analytic completeness: logical deletes (views) give fast compliance but leave PII in raw files until physical rewrite—risk if backups are restored. Physical deletes are strongest but costly and may cause temporary analytic gaps during rewrites. Use aggregate-only views and DP for small cohorts to preserve analytics without PII.
• Complexity: orchestration and robust testing increase engineering effort.

Edge cases & mitigations

• Cross-references (joins, derived tables): propagate delete markers to derived datasets and schedule rebuilds.
• Late-arriving events for deleted users: use a blacklist filter in ingestion for requests after deletion; log and drop or route to quarantine for review.
• Multi-tenant/third-party: ensure contracts require deletion propagation.

Summary
Emit a Delete Command, handle immediate logical exclusion in queries and streaming state removals, perform targeted physical rewrites for historical files within 24h, and manage backups via metadata-driven filtering or expensive rewrites. Orchestrate, verify, and audit every step. Balance cost vs. strictness by choosing faster logical approaches plus prioritized physical clean-up when necessary.