Company Privacy Landscape Questions
Demonstrate company specific understanding of privacy and data protection considerations. This covers the organization public privacy commitments, data handling scale and types, major privacy initiatives, known privacy risks or incidents, applicable privacy regulations for their markets and products, data governance practices, and how privacy requirements influence product design, analytics, and third party integrations. Interviewers look for evidence you researched the company privacy context and can discuss implications for compliance, user trust, and practical privacy engineering or policy tradeoffs.
MediumSystem Design
75 practiced
Design a reliable deletion pipeline that fulfills right-to-be-forgotten requests across multiple persistent stores: S3 data lake (Parquet partitions), a BigQuery analytics dataset, and an operational MySQL database. Requirements: process a deletion request for a user id, ensure idempotency, produce auditable logs, and attempt removal from backups where possible. Sketch the components and the workflow.
Sample Answer
Requirements clarification:- Input: deletion request for a user_id (GDPR right-to-be-forgotten).- Guarantees: best-effort removal from S3 (Parquet partitions), BigQuery dataset, and MySQL; idempotent processing; auditable, tamper-evident logs; attempt to remove from backups where feasible; report success/failure per store.High-level architecture (components):- API/Portal: accepts deletion requests, assigns request_id (UUID), validates.- Orchestrator: workflow engine (Airflow / Temporal) to coordinate steps and retries.- Message bus: Pub/Sub or Kafka for async tasks.- State store: idempotency & progress tracker (DynamoDB / CloudSpanner / MySQL control table) keyed by request_id: statuses per target store.- Workers: - MySQL worker: transactional deletes / pseudonymization. - BigQuery worker: delete jobs (DML or overwrite partitions / write-time filter). - S3 worker: delete from raw Parquet by rewriting affected partitions or marking records and compacting via Spark.- Audit store: append-only audit log (Cloud Audit Logs / write-once S3 + checksum) capturing request_id, user_id, timestamps, actions, result codes.- Backup scrub coordinator: triggers backup retention/expunge where supported (snapshot index + manual operator flow).- Monitoring & Reporting UI.Workflow:1. Request received (request_id created). Persist initial record in state store with status=RECEIVED and write audit entry.2. Orchestrator enqueues a deletion job (request_id, user_id) to message bus and starts workflow.3. Idempotency check: worker reads state store; if step already COMPLETED, skip; else set step=IN_PROGRESS (compare-and-set).4. MySQL step: - Prefer soft-delete + physical delete in same transaction if possible. - If referential constraints exist, cascade or anonymize. - On success, mark state COMPLETED and audit.5. BigQuery step: - If table partitioned by user ingestion time or has user_id column: submit a DELETE DML job or run a TABLE SNAPSHOT → create new table excluding user rows and atomically swap; for large tables prefer export-transform-load with partition pruning. - Record job ID, poll until done, mark COMPLETED in state store and audit result.6. S3 (Parquet) step: - Identify partitions/objects containing user_id via partition metadata/index or precomputed inverted index. - If small: rewrite partition files removing rows for user (Spark job writing new objects and then atomically swapping manifests). - If impossible to rewrite (very large / immutable lake): mark-as-deleted in a sidecar deletion manifest and exclude in downstream reads; schedule compaction for physical removal. - Update state and audit.7. Backup handling: - Check backup systems metadata to see if deletion is feasible (some cold backups are immutable). - If possible, queue backup-expunge requests; otherwise record evidence that backups could not be erased and set flag for retention policy shortening on next retention cycle.8. Finalization: - When all target statuses are COMPLETED (or COMPLETED/UNSCRUBBABLE with justification), set request state DONE and emit final audit record and notification.Key design points:- Idempotency: use request_id + CAS updates in state store; workers must be safe to re-run (DELETE is idempotent).- Atomicity per-store: use store-native atomic operations (DB transactions, BigQuery job swaps, S3 manifest swaps).- Auditing: append-only log with request_id, step, actor, timestamp, hashes of modified objects; store copies in two locations for tamper-evidence.- Scalability: workers scale horizontally; use partition pruning to limit scan cost; maintain auxiliary indexes mapping user_id → object/row locations.- Consistency model: eventual consistency across stores; provide visibility of partial completions and reasons.- Backups: document legal/operational constraints; if backups immutable, record evidence and shorten future retention if allowed.Edge cases & trade-offs:- Large historical data: physical deletion expensive; prefer marked-deleted + compaction windows.- Referential integrity: anonymization may be safer where deletion breaks analytics.- Audit vs privacy tension: audit records must not reintroduce personal data — store only metadata (request_id, hashes, counts) or encrypt logs and restrict access.- Cost vs timeliness: immediate full-table rewrites are costly; balance with SLA for compliance.This pipeline provides idempotent, auditable, and extensible deletion handling across MySQL, BigQuery, and S3 while making explicit trade-offs for backups and very large datasets.
HardTechnical
53 practiced
During an investigation you find that a scheduled job exported multiple Parquet files containing PII to a third-party S3 bucket due to misconfiguration. Describe the forensic steps you would take to preserve evidence, identify scope and timeline, contain and remediate, and prepare findings for legal and regulatory notification. Mention specific logs and metadata you would collect.
Sample Answer
Situation / overview: I discover a scheduled data-export job (e.g., Spark/Glue/Airflow) has written multiple Parquet files containing PII to an external third‑party S3 bucket because of a configuration error. My response covers: preserving evidence, establishing scope & timeline, containment/remediation, and preparing an evidence package for legal/regulatory notification.Preserve evidence (first 30–60 minutes)- Take immutable snapshots where possible (enable S3 Object Lock on internal buckets if available; snapshot EC2/EBS volumes used by the pipeline).- Collect and preserve copies (read‑only) of all affected Parquet files and directory listings; compute SHA256 hashes for each object and store hashes separately to prove integrity.- Record chain-of-custody: who collected what, when, and where stored.- Freeze relevant configuration (Airflow DAGs, Glue job configs, EMR job configs, cron entries) and export them.Logs & metadata to collect (critical)- AWS CloudTrail event history (S3 PutObject, PutObjectAcl, CreateBucket, PutBucketPolicy), including eventTime, userIdentity, sourceIPAddress, userAgent, requestParameters, awsRegion, and requestID.- S3 Server Access Logs (if enabled) and S3 Object metadata: LastModified, ETag, Content-Length, x-amz-version-id, x-amz-meta-*, multipart upload IDs.- S3 Inventory reports if available.- VPC Flow Logs for egress IPs and networking path.- CloudWatch Logs for Glue/Spark/EMR, Lambda, Airflow scheduler & worker logs.- IAM/STS logs: temporary credentials issuance, AssumeRole calls.- KMS CloudTrail logs if objects were encrypted.- AWS Config snapshot for resource state and changes (bucket policies, ACLs).- Third‑party bucket owner logs or access records; if the third party supports access logging or CloudTrail for cross-account access, request those records.- System metadata from runner machines: process lists, cron history, environment variables, credentials files, and instance metadata service queries.- Audit the schedule trigger: Airflow task run history, Glue job run IDs, EMR step logs, scheduler timestamps.Identify scope & timeline- Correlate CloudTrail PutObject events to the job start and end times; build an event timeline with hashes, object keys, uploader identity, source IPs, and job run IDs.- Identify all objects exported, affected internal datasets, downstream consumers, and whether exports were complete vs partial (multipart uploads).- Determine PII types and count records (sample analysis on preserved copies with strict access control).Containment & remediation- Immediately revoke or rotate compromised credentials (IAM keys, service roles) used by the job; disable relevant IAM roles or apply least-privilege policies temporarily.- Remove external write permissions: tighten bucket policy/ACLs on target and internal buckets; if third-party bucket still accessible, request that the third-party quarantine or delete objects pending investigation.- Stop scheduled job runs (pause DAGs, disable Glue triggers, stop cron).- Reconfigure job to write to a secure internal bucket with proper encryption, access controls, and VPC endpoints. Re-run in a sandbox to validate.- If data was encrypted with KMS, audit KMS key usage and consider key rotation.Prepare findings & legal/regulatory notification- Produce an evidence package: timeline, hashes, preserved object copies, exported configs, relevant logs (CloudTrail, S3 access logs, CloudWatch logs), IAM/Config snapshots, and analysis of PII types/counts.- Summarize root cause, scope (number of records, data categories), exposure risk, remediation steps taken, and recommended controls.- Coordinate with Legal/Compliance to determine breach notification obligations (GDPR, CCPA, sector rules), include notification timelines, required content, and offer remediation steps for affected data subjects if needed.- Retain original artifacts in secure, access‑controlled storage for legal hold.Lessons & mitigations (post-incident)- Enforce least privilege for export roles, require cross-account approvals, require pre-deploy checks for external destinations.- Add automated detection: S3 Block Public Access, Macie for sensitive data discovery, GuardDuty alerts for unusual S3 writes, CloudWatch alarms on PutObject to non-whitelisted buckets.- Add CI checks to validate job destinations, secrets management and key rotation, and runbooks for rapid containment.This approach preserves evidence integrity, produces an auditable timeline, contains further exposure, remediates root cause, and provides a defensible package for legal and regulatory action.
MediumTechnical
65 practiced
For big data systems like Hive, BigQuery, and Snowflake, describe what audit logging you would capture for data access and modifications to support investigation and compliance. Include retention strategy, tamper-evidence, indexing for search, and integration with SIEM or compliance dashboards.
Sample Answer
Situation: As a data engineer responsible for compliance-ready data platforms, I’d design audit logging that makes investigations fast, defensible, and tamper-evident across Hive, BigQuery, and Snowflake.What to capture (all events):- Who: user id, service account, IAM role, client_cert, and proxy (if present)- What: operation type (SELECT/query, CREATE/ALTER/DROP, INSERT/UPDATE/DELETE, GRANT/REVOKE, EXPORT, COPY, JOB_CANCEL)- Where: dataset/table/view/schema names, object IDs, cluster/warehouse- When: UTC timestamp with ms precision, job id, query id- How/context: full SQL text, bind parameters (or query fingerprint), bytes scanned, rows returned/affected, runtime, result status, client IP, user agent, originating app, session id- Change metadata: before/after schema snapshots for DDL; for sensitive DML, row-level audit (if feasible) or change-data-capture (CDC) references- Admin actions: policy changes, role/permission changes, key rotations, export/ingest to external systems- Correlation: trace IDs, pipeline/job lineage, DAG idSystem-specific sources (examples):- BigQuery: Cloud Audit Logs + INFORMATION_SCHEMA.JOBS_BY_PROJECT and JOBS_BY_USER, EXPORT of audit logs via Pub/Sub to SIEM or partitioned log dataset- Snowflake: ACCESS_HISTORY, QUERY_HISTORY, OBJECT_DEPENDENCIES and ACCOUNT_USAGE views; enable Snowflake event tables and stream to S3 or an internal secure database- Hive/Hadoop: HDFS audit logs, Ranger/Atlas audit records, YARN and HiveServer2 logs; collect from Namenode and Ranger policy audit topicsRetention strategy:- Tiered retention: hot (3–90 days) in fast-search index for immediate investigation; warm (90 days–2 years) in compressed columnar store (e.g., BigQuery/Snowflake logging dataset) for analytics; cold/archival (2–7+ years) in immutable object storage (S3/GS/Azure Blob) for legal/regulatory retention and eDiscovery.- Policy-driven retention by sensitivity: PII-access events may be retained longer; ability to place legal hold per case.- Data minimization: redact or hash sensitive payloads (e.g., query parameters) where policy requires.Tamper-evidence and integrity:- Centralize logs immediately (agentless or lightweight agents) to immutable storage with WORM/immutability (S3 Object Lock, GCP Bucket Lock) and retention policies.- Sign and checksum log files (SHA256) and store checksums in a separate metadata DB or append-only log (blockchain-style hash chain) to detect alteration.- Use cloud provider’s native audit logs (Cloud Audit Logs, Snowflake cloud logs) which are tamper-resistant and access-controlled.- Enforce strict IAM for log write/erase; separate admin roles for retention policy changes and for reading logs; enable MFA and key rotation via KMS.- Periodic automated integrity checks and alerts on checksum mismatches.Indexing and search:- Ingest logs to an indexed search layer (Elastic/Opensearch) for fast, text-searchable queries across SQL text, user, IP, and resource with Kibana dashboards.- Also store structured logs in a columnar analytics store (BigQuery / Snowflake logging dataset) partitioned by date and clustered on user, object, and query_id for performant ad-hoc and historical queries.- Create derived tables: query fingerprints, top-access patterns, abnormal access baselines.- Maintain pre-built indexes: partition by ingestion_date, cluster on user and object, and maintain inverted index for SQL text/fields to support "find all queries that referenced table X" quickly.Integration with SIEM and dashboards:- Ship logs in near real-time to SIEM (Splunk/QRadar/Elastic SIEM) via native connectors: Pub/Sub→SIEM, Cloud Logging sink→SIEM, Snowflake external stages or Snowpipe, Kafka/Ranger audit topics for Hive.- Enrich logs with identity context (HR/IDP attributes), data sensitivity/classification tags (from Data Catalog/Ranger/Atlas), and pipeline lineage to prioritize incidents.- Build real-time detection rules: unusual table scan volumes, queries outside of business hours, privilege escalations, large exports, service-account anomalies, geo/IP anomalies.- Create compliance dashboards: access heatmaps per dataset, top consumers, recent DDL changes, retention compliance, legal holds, and unresolved alerts; expose to auditors with read-only views and scheduled exports.Practical considerations:- Balance full SQL capture vs privacy: mask literal values where necessary and keep query fingerprints to link back to raw logs under stricter controls.- Sampling is acceptable for heavy-read workloads for anomaly detection, but full capture for admin and DDL events.- Automate retention enforcement and demonstrate chain-of-custody for auditors: show every step from ingestion to archiving with timestamps, checksums, and access logs.This approach ensures investigators can reconstruct “who did what, when, where, and how,” supports regulatory retention requirements, provides tamper-evidence, and integrates with SIEM for alerting and compliance reporting.
EasyTechnical
52 practiced
What is a Data Protection Impact Assessment (DPIA)? As a data engineer, list the artifacts and inputs you would provide to a DPIA for a new analytics product that processes large-scale user behavioral data.
Sample Answer
A Data Protection Impact Assessment (DPIA) is a structured process to identify, assess and mitigate privacy risks of a processing activity—especially where processing is large-scale or likely to result in high risk to individuals’ rights. It documents lawful basis, risks to data subjects, mitigation measures, and residual risk for reviewers (privacy, legal, security).As a data engineer building a new analytics product that processes large-scale user behavioral data, I would provide the following artifacts and inputs:- Processing overview: concise description of data sources, purpose, categories of personal data (identifiers, behavioral events, device IDs), and processing activities (ingest, join, enrichment, aggregation).- Data flow diagrams: end-to-end diagrams showing ingestion, staging, transformations, storage, downstream consumers, third-party integrations, and export points.- Data inventory & schema docs: tables/streams with field-level schema, data types, sample values, PII/PIA tags, and data lineage.- Volume and retention estimates: expected throughput (events/sec, TB/day), retention periods per dataset, and archival strategy.- Access & governance controls: IAM roles, RBAC, network segmentation, audit logging, and least-privilege policies for pipelines, warehouses, and notebooks.- Technical safeguards: encryption at rest/in transit, key management approach, tokenization/UID mapping, and pseudonymization/anonymization techniques (hashing salts, k-anonymity, differential privacy plan).- Minimization & aggregation approach: how raw identifiers are dropped, which fields are aggregated, sampling or downsampling strategies.- Consent & legal inputs: flags/fields capturing consent state, signals for opted-out users, and how those are enforced in pipelines.- Risk assessment notes: identified privacy risks (re-identification, unauthorized access, profiling), likelihood/impact, proposed mitigations, and residual risk.- Monitoring & incident response: pipeline monitoring metrics, alerting for unexpected schema/volume, breach detection, and remediation playbook.- Tests and validation artifacts: unit/integration tests that enforce PII handling rules, anonymization verification results, and privacy-preserving statistical checks.- Third-party vendor information: contracts, data processing agreements, where data is sent, and their security posture.- Deployment & change control: CI/CD process for pipelines, roll-back plans, and change review steps affecting personal data.For each artifact I’d include concrete examples (sample schema, small data flow diagram), and be prepared to walk through how pipelines enforce consent and pseudonymization in code (e.g., hashing user IDs with HMAC-SHA256 + per-environment secret) and how retention enforcement jobs delete or mask historical data. These inputs let privacy/legal/security make an informed DPIA decision and help implement effective mitigations.
MediumTechnical
61 practiced
Describe how you would implement field-level encryption for sensitive columns stored in a data warehouse (e.g., SSN or credit card suffix). Discuss choices for encryption (client-side vs server-side, envelope encryption), key management, queryability (searchable encryption trade-offs), and effect on analytical performance.
Sample Answer
Clarify requirements first: which columns need protection (full SSN vs last-4), who must be able to decrypt (analysts vs only apps), what queries must run (equality lookups, joins, range filters, aggregations), compliance constraints (PCI/PII rules), and performance/scale targets.Recommended approach (practical, scalable)- Envelope encryption with a KMS-backed root key: generate per-row or per-partition data keys (DEKs) to encrypt column values, then encrypt DEKs with a master key in a cloud KMS (AWS KMS / Azure Key Vault / Google KMS) or HSM. This minimizes KMS calls and centralizes key rotation/audit.- Use server-side encryption for bulk pipeline stages (e.g., S3/Blob + KMS SSE) but perform field-level encryption at ingestion (client-side in ETL) so the warehouse never stores plaintext. For streaming pipelines, encrypt in the producer or ETL job before writing to the data lake/warehouse.Key management- Master key in KMS/HSM, strict IAM roles, audit logs, and rotation policies. Store encrypted DEKs alongside records (header metadata) and rotate DEKs by re-encrypting DEKs with a new master key; optionally re-encrypt data with new DEKs during maintenance windows.- Consider per-tenant or per-class keys to limit blast radius; use key derivation for many keys.- Maintain an access control service that maps roles to allowed decrypt operations; require decryption through a service or tokenized proxy to avoid exposing raw keys to analysts.Queryability & trade-offs- Deterministic encryption allows equality joins/filters (same plaintext -> same ciphertext) but leaks frequency and is vulnerable to dictionary attacks. Use only for low-entropy fields if necessary and combine with access controls.- Probabilistic (randomized) encryption provides strong confidentiality but prevents any meaningful searching.- Hashing or searchable indexes: store a keyed HMAC or Bloom filter for searchable lookups. HMACs allow secure equality search without revealing full value but still leak collisions; use salts/per-record nonces when appropriate.- Tokenization or format-preserving encryption (FPE) for credit card suffixes lets you preserve formats and enable some analytics while protecting most of the value.- For analytics that require aggregation or range queries, decrypt in a secure enclave / trusted compute (e.g., AWS Nitro Enclaves) or use differential privacy/secure multi-party techniques — otherwise pre-compute anonymized aggregates.Impact on analytical performance- Encrypted columns cannot be scanned/processed efficiently for many operations: predicates on encrypted bytes prevent index usage and pushdown, increasing CPU and I/O. Deterministic encryption can preserve some indexability.- Additional storage overhead for IVs/DEKs/HMACs and for re-encryption during rotation.- Mitigations: encrypt only truly sensitive fields; keep derived, analytics-friendly columns (hashes, flags, cohorts) to support common queries; perform decrypt-and-join in ETL to produce sanitized analytic tables; cache decrypted results in a secured analytics zone; use hardware acceleration (AES-NI) on worker nodes.- Plan for key-access latency: envelope pattern reduces KMS calls. Benchmark ETL and query paths.Operational controls- Strong logging/monitoring of KMS operations, regular key rotation, periodic re-encryption, least-privilege IAM, and automated testing of decryption flows. Provide documented decryption APIs rather than giving direct key access to analysts.Example pattern- At ingest: ETL job requests a DEK from a key service, encrypts SSN with AES-GCM (unique IV), stores ciphertext + encrypted DEK + IV. Store an HMAC of last-4 for fast lookups. Analysts query pre-computed, masked/hashed analytics tables; only a small, audited service can request KMS to decrypt raw SSNs for approved workflows.This balances strong protection, scalable key management, controlled queryability, and acceptable analytic performance by encrypting minimally, using envelope encryption, and providing alternate search primitives or decryption services for approved needs.
Unlock Full Question Bank
Get access to hundreds of Company Privacy Landscape interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.