Privacy-Preserving Analytics and Experimentation Questions
Doing measurement and data science without over-collecting or exposing individuals: privacy-preserving experiment design, aggregate and on-device measurement, and privacy-respecting attribution. Covers techniques for analytics and A/B testing that limit personal-data use and honor consent. Includes reconciling measurement quality with privacy constraints.
MediumTechnical
149 practiced
Privacy regulations require excluding certain users from aggregated reports. Describe how you would implement privacy-aware reporting for product metrics across Meta's apps. Address consent flags, differential privacy or aggregation thresholds, and how to communicate limitations to stakeholders.
Sample Answer
Situation: As a BI analyst responsible for product metrics across Meta apps, I must ensure reports exclude users per privacy regulations (GDPR/CCPA) while keeping metrics useful.Approach / Implementation:- Data model & consent flags: persist explicit consent/status flags at ingestion (consent_timestamp, consent_version, granular flags like analytics_opt_in). Join reports only to users with valid consent; mark historical data retention windows per regulation.- Privacy enforcement layer: implement a centralized privacy middleware (SQL views / materialized tables) that applies exclusion logic before any BI tool hits data. All downstream ETL and Looker/Tableau models reference these sanitized tables.- Aggregation protections: - Thresholding / suppression: require minimum cohort size (e.g., k=10); suppress or return “<10” when below threshold. - Differential privacy for high-sensitivity metrics: apply DP noise (Laplace/Gaussian) with defined epsilon per metric and a global privacy budget; add noise post-aggregation (not per-user) and track cumulative budget per report. - Rounding and bucketing: round small counts, group rare categories to avoid re-identification.- Access controls & auditing: RBAC for raw tables, immutable audit logs of queries and privacy decisions, automated tests to verify exclusions.Communicating limitations:- Dashboard metadata: show visible banners/tooltip explaining exclusions, thresholds, and whether DP was applied (include epsilon and expected noise magnitude).- Provide data quality panel: cohort sizes, suppressed counts, and guidance on interpreting noisy values (confidence intervals).- Stakeholder playbook: short doc and templates explaining when to use sanitized vs. restricted datasets, procedures to request access to raw data (legal review), and escalation for product decisions needing precise counts.Result: This preserves regulatory compliance and user trust while keeping stakeholders informed about accuracy trade-offs and how to interpret metrics.
HardTechnical
78 practiced
A compliance request requires that any aggregated result shown to end users must not expose groups smaller than k=5 individuals. As a BI analyst, propose SQL-level strategies to produce aggregates that comply: suppression of small groups, coarsening/binning to ensure minimum group size, adding noise or differential privacy approaches, and describe performance trade-offs and dashboard UX impacts.
Sample Answer
Approach: provide SQL patterns for (1) suppression, (2) coarsening/binning, (3) noise / DP-style aggregation plus practical notes on performance and dashboard UX trade-offs.Suppression (hide groups < k):- Simple, fast (single GROUP BY). UX: shows blanks/“<5” labels; users lose visibility into small segments.Coarsening / binning (increase group sizes):- Pre-aggregate or create materialized views for repeated use. UX: less granular but preserves insights; requires stakeholder alignment on bucket definitions.Adding noise / DP-like approach:- Use DB or external library to add calibrated noise (or use a DP library). Performance: similar to normal aggregates but requires care to compute sensitivity and calibration. UX: provides plausible deniability; can confuse users — show confidence intervals and explainability.Hybrid pattern (suppress small then add small noise to remaining):- Suppress n<k, add noise to remaining to prevent re-identification via differencing.Performance trade-offs:- GROUP BY on high-cardinality columns is expensive; add indexes, pre-aggregate tables, or materialized views.- DP libraries may require multiple passes or cryptographic operations; consider offline batch processing.- Coarsening reduces cardinality and speeds queries.Dashboard UX impacts & recommendations:- Always display a clear legend/footnote explaining suppression, binning, or noise and what k=5 means.- For suppressed cells show “<5” (non-numeric) instead of NULL to avoid misinterpretation.- Provide drill-paths into approved aggregated views rather than raw data.- Offer explainability panels: sample sizes, confidence intervals, and the method applied.- Align with stakeholders to balance utility vs. privacy; default to stricter methods for executive/public reports.Operational tips:- Implement these in ETL or materialized aggregates to avoid per-dashboard computation.- Log anonymization parameters and versions for audits.- Test with synthetic data to measure analytic distortion before rollout.
sql
WITH grp AS (
SELECT country, COUNT(*) AS n, SUM(amount) AS total
FROM sales
GROUP BY country
)
SELECT country,
CASE WHEN n >= 5 THEN total ELSE NULL END AS total_masked,
CASE WHEN n >= 5 THEN n ELSE NULL END AS n_masked
FROM grp;sql
WITH b AS (
SELECT
CASE
WHEN age BETWEEN 18 AND 24 THEN '18-24'
WHEN age BETWEEN 25 AND 34 THEN '25-34'
ELSE '35+' END AS age_bucket,
COUNT(*) AS n, SUM(amount) AS total
FROM users JOIN sales USING(user_id)
GROUP BY 1
)
SELECT age_bucket, total, n
FROM b
WHERE n >= 5;sql
-- Laplace noise example (conceptual; DB must support random functions)
WITH agg AS (
SELECT country, COUNT(*) AS n, SUM(amount) AS total FROM sales GROUP BY country
)
SELECT country,
n + CAST( ( -LN(RANDOM()) * sign - ) AS INT) AS n_noisy, -- pseudocode
total + (RANDOM() - 0.5) * sensitivity * scale AS total_noisy
FROM agg;MediumTechnical
75 practiced
Medium: A stakeholder requests a bespoke metric that requires joining customer support tickets, billing events, and product events. Outline the privacy, PII, and compliance considerations you would review before building this metric and how you would design access controls around it.
Sample Answer
First, clarify the metric requirements (fields, granularity, audience, business need) so you can map required data elements to risk.Privacy, PII & compliance review- Identify PII: names, emails, phone, billing identifiers, payment card data, device IDs, IPs. Classify each field per company data classification.- Legal constraints: GDPR (personal data, lawful basis, data subject rights), CCPA/CPRA, PCI-DSS for card data, HIPAA if health info, contractual/SLA clauses.- Purpose & minimization: ensure data collected/used is proportionate to the business purpose; avoid pulling full PII if not required.- Consent & retention: check user consent status, opt-outs, and retention policies; enforce retention/deletion rules.- DPIA & approvals: run a Data Protection Impact Assessment if profiling/combining datasets increases risk; get Privacy/Legal sign-off.Designing access controls- Principle of least privilege: limit who can view the metric. Create roles (e.g., exec, product, support) with explicit permissions.- Data-level controls: use column-level masking/obfuscation in BI tool or views for PII (hash IDs, truncate emails, redact names). Expose joins using surrogate keys instead of raw identifiers.- Row-level security: enforce filters so users see only customers within their domain/region.- Environment separation: build and preview in a secured dev workspace; only promote to production after approvals.- Auditability & monitoring: log queries, dashboard views, data exports; alert on anomalous access.- Export controls: disable CSV/PDF export or require approval for exports containing sensitive fields.- Technical enforcement: implement masked SQL views/stored procedures, use IAM (RBAC) in BI tool, and encryption-at-rest/in-transit.- Review cadence: periodic access reviews and attestation by data owners.Example implementation- Create a secure reporting view that returns event counts and an opaque customer_id (HMAC with rotation). Mask emails and billing tokens. Apply RLS by team and region. Require Privacy sign-off and log all accesses; allow exports only for users with an approved business justification.This approach protects subjects, meets compliance, and still delivers actionable insights.
HardTechnical
89 practiced
Design an analytics architecture that allows BI reporting while complying with GDPR/CCPA when using pseudonymized user identifiers. Address: separation of identity and analytics layers, data minimization, access controls, auditability for deletion or subject access requests, and how to handle historical aggregates needing deletion.
Sample Answer
Requirements & constraints:- BI must use pseudonymized IDs (no direct identifiers in analytics layer).- Support GDPR/CCPA: right-to-access, right-to-delete, data minimization, audit trail.- Low-latency dashboards (Looker/Tableau) and scheduled reports.High-level architecture:- Identity Layer (secure service): stores PII and mapping (user_id -> pseudonym_id) in an encrypted, access-controlled store (vault/DB). Exposes APIs for lawful queries (auth, logging).- Ingestion & ETL: Raw events include pseudonym_id only. PII never flows into analytics pipelines. ETL performs minimal enrichment (no PII), tags events with pseudonym_id.- Analytics Warehouse: columnar store (BigQuery/Snowflake) holding pseudonymized event data and dimensional tables without PII. BI tools connect only to this layer.- Access & Governance: role-based access (RBAC) and attribute-based controls (ABAC) on warehouse; BI extracts limited to aggregated views (no row-level personal profiling unless approved).- Audit & Compliance: centralized audit logs for identity queries, deletion requests, and ETL jobs; immutable write-once logs (WORM) for auditability.Data minimization & controls:- Store only fields required for analytics; drop or hash unnecessary fields at ingestion.- Use one-way hashes with per-environment salt for pseudonym_id to prevent cross-system re-identification.- Implement data retention policies: partition-by-date and automated TTL to purge raw partitions.Handling deletion / subject requests:- Deletion request flow: verify identity → identity layer removes/flags mapping and records deletion event → emits deletion token/event to ETL/CDC pipeline.- Analytics deletion strategies: 1. Soft-delete + filter: mark pseudonym_id as deleted; any future queries or dashboards exclude flagged IDs via a global filter. Good for quick compliance but leaves historical aggregates. 2. Recompute aggregates: maintain event-level data partitioned; on deletion, trigger recompute of affected aggregate materialized views for windows that included the user. Use incremental recompute jobs for affected partitions for scalability. 3. Privacy-preserving aggregates: where recompute is infeasible, apply techniques: - Differential privacy noise to aggregated metrics so single-user removal has negligible effect. - Minimum group-size thresholds (suppress cells with < k users).- Recommendation: Combine (1) + (2) for critical KPIs (sales, billing) and (3) for large, derived cohorts where full recompute is cost-prohibitive.Auditability:- Log every identity lookup, deletion, and recompute job with who/why/timestamp and checksum of resulting aggregates.- Provide subject access: identity layer can use mapping to fetch pseudonymized records and produce an export from analytics data filtered to that pseudonym_id, with access logged.- Maintain an immutable trail linking deletion request → ETL events → aggregate recompute jobs → final state (hashes/checksums) for regulators.Operational considerations for a BI Analyst:- Build BI views as parameterized, governed datasets that automatically respect global "exclude_deleted" filters.- Document which dashboards require recompute on deletion vs. privacy-preserving approaches.- Work with engineering to schedule nightly incremental recomputes for business-critical aggregates; surface delays and uncertainty in dashboards metadata.- Add data quality/monitoring alerts for failed deletion propagation and aggregate drift.Trade-offs:- Full recompute ensures strongest compliance but costs time/compute for high-cardinality aggregates.- Privacy-preserving methods reduce cost but may reduce accuracy and require stakeholder buy-in.This design separates identity from analytics, minimizes PII in BI systems, enforces strict access controls, provides auditable deletion and access flows, and offers practical strategies for handling historical aggregates.
MediumTechnical
99 practiced
Describe how Airbnb's BI and data engineering teams should balance personalization (which relies on user-level signals) with privacy and regulatory compliance. Propose technical controls, governance checks, and reporting practices that preserve analytic value while protecting user data.
Sample Answer
Situation: As a BI analyst supporting personalization, I must deliver actionable user-level insights (e.g., conversion lift by cohort) while ensuring GDPR/CCPA compliance and protecting PII.Approach / Principles:- Minimize: only use data necessary for the analytic purpose.- De-identify early: transform PII at ingestion into stable pseudonyms and store raw identifiers separately in a locked vault.- Purpose & consent enforcement: honor user consent flags in pipelines and exclude or roll-up opted-out users.Technical controls:- Ingestion: apply deterministic hashing + per-environment salt for user_id; store mapping in an encrypted secrets-managed vault with strict RBAC.- Feature store: expose features for modeling as aggregated or pseudo-identified values; prevent joins back to raw identifiers outside controlled environments.- Aggregation & thresholds: enforce minimum cell sizes (k >= 10) in dashboards; block queries that return small cohorts.- Differential privacy / noise: for high-sensitivity metrics (ad conversion, revenue by micro-cohort) add calibrated Laplacian/ Gaussian noise or use DP libraries to provide provable privacy.- Query governance: implement query cost and privacy guards (query planner rejects personality queries that could re-identify).- Secure compute: run sensitive joins and experiments inside ephemeral secure enclaves/air-gapped analytics clusters with VPC, encryption at rest/in transit, and strict logging.Governance checks:- Data catalog + lineage: label datasets with sensitivity, retention, lawful basis, and allowed uses; require data owner sign-off for new personalization metrics.- DPIA & risk review: for new personalization features run DPIA and privacy threat model; involve legal and privacy teams.- Access controls & approvals: role-based access to user-level datasets; just-in-time access with manager approvals and expiry.- Audit & monitoring: log all dataset accesses, query patterns, and exports; run periodic access reviews and anomaly detection for exfil patterns.Reporting practices (BI-specific):- Default to aggregated dashboards (trend, cohorts) with user-count thresholds; provide drilldowns only to anonymized cohorts.- Create "privacy-aware" dashboard templates: automated masking of PII, removal of raw IDs, and warnings when exporting.- Provide contextual metadata: show consent coverage, sample rate, noise parameters, and confidence intervals so analysts understand analytic limits.- Self-serve safe analytics: offer pre-approved metrics and parameterized reports via governed Looker/Tableau dashboards; gate custom ad-hoc user-level queries to a data steward process.- Incident playbook: predefine steps for suspected breaches, including revoking tokens, rotating salts, and notifying privacy office.Result / Rationale:These controls preserve analytic value—stable pseudonymous IDs allow longitudinal analysis, aggregated features enable personalization modeling—while reducing re-identification risk, ensuring legal compliance, and keeping transparency for stakeholders. As a BI analyst, I’d operationalize templates, implement gating rules in dashboards, and collaborate with data engineering and privacy to keep the balance between insight and protection.
Unlock Full Question Bank
Get access to hundreds of Privacy-Preserving Analytics and Experimentation interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.