Situation: As a data engineering intern at a mid-size healthcare analytics firm, I joined a six-week project to prepare our data platform for a HIPAA readiness audit after we began ingesting patient referral data.
Task: My responsibility was to discover and classify PHI across our ingestion pipelines, implement pseudonymization/anonymization in ETL, enforce access controls and logging, and produce auditor-ready evidence.
Action:
- Scoping: Focused on inbound ETL from EHRs and partner CSVs into our AWS data lake (S3), downstream Spark jobs, and Redshift reporting schemas.
- Discovery & classification: Ran automated scans using AWS Macie + custom Spark jobs to detect name, SSN, DOB, and medical codes. Tagged datasets in Glue Data Catalog with sensitivity labels (PHI, PII, non-sensitive).
- Pseudonymization/anonymization: Implemented a two-tier approach in Spark ETL:
- Pseudonymization for analytic needs: deterministic HMAC-SHA256 with an environment-specific secret stored in AWS KMS to map patient_id → token_id (reversible only via KMS-wrapped keys held by Security).
- Anonymization for de-identified exports: removed direct identifiers, generalized DOB to age buckets, and applied k-anonymity checks (k≥10) using a Spark aggregator; for small cohorts we suppressed or aggregated.
- Access controls & encryption: Enforced least privilege via AWS IAM roles for Glue/Athena/Redshift. S3 buckets had SSE-KMS; Redshift used encryption and VPC-only access. Row- and column-level access in Redshift via views and IAM-controlled query roles limited who could see tokenized vs. raw fields.
- Logging & monitoring: Enabled CloudTrail, Audit logs in Redshift, and Glue job logs forwarded to ELK. Data lineage captured in Glue tags and Airflow DAG metadata. Alerts for unauthorized schema changes or access patterns were configured in CloudWatch -> PagerDuty.
Result (audit demonstration):
- Prepared an evidence package: data classification reports, Glue catalog screenshots, KMS key policy, IAM role lists, CloudTrail access logs showing role-based queries, examples of ETL code snippets (Spark pseudonymization functions), and k-anonymity reports for de-identified datasets.
- During the audit, we showed reproducible pipelines that produced tokenized outputs and the strict key management process; auditors accepted our controls as meeting HIPAA’s de-identification and access requirements. Post-audit, we automated the classification scans and added quarterly rekeying and key-rotation procedures.
Learning: Balancing analyst needs and privacy requires layered controls: strong pseudonymization, strict key management, automated discovery, and clear, reproducible evidence for auditors.
