Privacy-Preserving Analytics and Experimentation Questions

Doing measurement and data science without over-collecting or exposing individuals: privacy-preserving experiment design, aggregate and on-device measurement, and privacy-respecting attribution. Covers techniques for analytics and A/B testing that limit personal-data use and honor consent. Includes reconciling measurement quality with privacy constraints.

HardTechnical

72 practiced

Critically analyze the following strategy: centrally collect raw event logs, anonymize user IDs with a salted hash, and then add random noise to aggregates before release. Identify realistic attack vectors that could re-identify users and propose robust alternatives with reasoning.

Sample Answer

High-level critique- The pipeline (central raw logs → salted-hash IDs → noise on aggregates) is a weak privacy posture: central collection keeps full fidelity data vulnerable; salted hashing is reversible under brute-force or linkage if salt leaks or attacker guesses; adding “random noise to aggregates” without formal guarantees risks insufficient protection against reconstruction or differencing attacks.

Realistic attack vectors1. Linkage with auxiliary data: adversary with external records (timestamps, IP ranges, rare event patterns) can match salted-hashed IDs to known profiles by correlating event sequences.2. Salt brute-force / leakage: if salt is static, short, or stored near data, attackers can brute-force common user IDs (emails, phone numbers) and recover mappings.3. Frequency / uniqueness attacks: unique sequences of events (e.g., one user did X at 03:07 then Y) act as fingerprints; aggregates that preserve rare cells allow re-identification.4. Differential differencing (intersection) attacks: repeated releases with small noise let attackers average out noise and recover true counts or identify presence/absence of a user.5. Reconstruction attacks on high-dimensional data: central raw logs permit an attacker (insider or via breach) to run record linkage or machine-learning reconstruction to de-anonymize.

Robust alternatives (with reasoning)1. Avoid central raw collection when possible: collect client-side or use local differential privacy (LDP) to perturb at source, so raw identifiers never leave device — reduces breach impact.2. Apply strong cryptographic hashing with per-record secret (HMAC) only for necessary linking, rotate keys, and store keys in HSMs. But treat hashed IDs as pseudonymous, not anonymous.3. Adopt differential privacy (DP) for all released aggregates: use calibrated noise (Laplace/Gaussian) with a global privacy budget, accounting for composition over repeated releases. DP provides provable bounds against differencing and reconstruction.4. Aggregate before centralization: bucketize or generalize rare attributes (k-anonymity-like pre-processing) and suppress low-count cells to reduce uniqueness before DP.5. Access controls + monitoring: limit raw log access via RBAC, encryption-at-rest, query auditing, and use secure enclaves for sensitive computations.6. Synthetic data + DP: when sharing datasets, produce DP synthetic datasets rather than naively noisy aggregates; this preserves utility while providing formal guarantees.7. Threat modeling and red-team testing: simulate linkage attacks using realistic auxiliary datasets to tune DP parameters and suppression thresholds.

Recommendation (practical steps)- Stop relying on salted hashes as the primary protection. Implement DP for releases with a documented ε budget, move to LDP for client telemetry where feasible, reduce raw centralization, and enforce strict key management and access controls. Quantify utility vs privacy trade-offs with experiments and publish a privacy spec stakeholders can audit.

MediumSystem Design

86 practiced

Design a privacy-preserving analytics pipeline to compute daily active users (DAU) and 7-day retention without storing raw per-user event logs centrally. Provide high-level architecture components, dataflows, and discuss guarantees and trade-offs for accuracy and privacy.

Sample Answer

Requirements & constraints:- Functional: compute daily active users (DAU) and 7-day retention (cohort retention) each day.- Non-functional: no central raw per-user event logs; preserve privacy (limit re-identification); reasonable accuracy for product decisions; scalable, auditable.

High-level architecture:1. Client SDK (mobile/web) - Locally aggregates events into daily summaries per user (e.g., boolean active, day-of-week, cohort id). - Optionally compute Bloom filter or hashed identifiers for dedup checks.2. Local Differential Privacy (LDP) layer on client - Applies randomized response or randomized hashing to sensitive fields (or perturbs counts) before upload.3. Secure Aggregation Gateway - Collects encrypted/perturbed per-client summaries; uses secure multi-party aggregation (e.g., threshold homomorphic encryption or secure aggregation protocol) so server only sees aggregated totals, not per-user messages.4. Aggregation & Analytics service - Decrypts aggregated sums once threshold met, computes DAU and rolling 7-day retention from aggregated cohort counts. - Stores aggregated time-series (not raw events).5. Monitoring & Privacy Accounting - Tracks total privacy budget (epsilon), data quality metrics, and sampling rates.

Dataflow (example for DAU & 7-day retention):- Client: for each day, compute active_flag = 1 if any relevant event; cohort_day = signup_day; package {cohort_day, day_index (0-6), active_flag}.- Apply LDP: flip active_flag with probability derived from epsilon or add noise to count.- Secure-aggregate encrypted vectors across clients → server obtains noisy sums per (cohort_day, day_index).- Analytics cleans (debias LDP noise), computes DAU = sum active_flags per day, 7-day retention = aggregated count of cohort_day = t and day_index = 7 divided by cohort size estimate.

Guarantees and trade-offs:- Privacy guarantees: - LDP provides per-user privacy with parameter epsilon; secure aggregation prevents the server from seeing individual messages before aggregation. - Combining both reduces risk of re-identification.- Accuracy trade-offs: - LDP introduces noise; accuracy improves with larger N (law of large numbers). Small cohorts suffer high relative error. - Secure aggregation requires threshold of participants; delays or missing participants can bias daily metrics. - Bloom filters/hashing help dedupe but add false positives/negatives.- Operational trade-offs: - Complexity: implementing secure aggregation and privacy accounting increases engineering cost. - Freshness vs. privacy: lower epsilon (stronger privacy) → noisier/less precise metrics; can mitigate by smoothing, Bayesian debiasing, hierarchical aggregation. - Auditability: keep reproducible aggregated pipelines and synthetic test datasets for validation without exposing raw logs.

Practical recommendations:- Use hybrid approach: secure aggregation + mild LDP (epsilon ~1–2) to balance privacy and utility.- Use cohort minimum-size thresholds; suppress or flag small cohorts.- Debias aggregated LDP noise analytically and provide confidence intervals.- Instrument monitoring for participation rates and privacy budget burn.- Start with DAU and 7-day retention prototypes on aggregated data, validate against internal non-production raw logs for calibration, then roll out to production.

MediumTechnical

88 practiced

Design an experiment-level privacy policy and consent UI for running A/B tests that include personalization. Specify the consent language, the granular choices users should have, and how consent toggles map to data collection and processing flows.

Sample Answer

Situation: You need an experiment-level privacy & consent UI for A/B tests that use personalization while remaining GDPR-compliant and giving users clear control.

Consent language (short, plain English):- Title: “Personalized experience & experiments”- Body: “We run experiments to improve features and may personalize content using data about your activity. You can choose what we use: essential signals only, analytics, or full personalization which may include building models that use your anonymized profile. Refuse or change choices anytime.”- Purpose list (bullet): “1) Essential product function; 2) Performance & analytics; 3) Personalization & experiments; 4) Third-party services.”

Granular UI choices (toggles; default = only Essential ON):1. Essential (required) — collects device, session cookies for security/stability. No opt-out.2. Analytics — collects aggregated/hashed event data (pageviews, clicks) for telemetry and A/B measurement.3. Personalization & Experiments — allows use of behavioral signals + persistent pseudonymous ID to serve personalized variants and to train/validate models.4. Third-party sharing — permits sending data to vendors (CDP, analytics platforms). Off by default.

Mapping toggles → data flows- Essential ON: session_id, error logs → routing to app telemetry; no experiment assignment that personalizes.- Analytics ON: anonymous_event(user=null) → storage in analytics DB; used to compute experiment metrics (CTR, conversion) in aggregate; no profile building.- Personalization & Experiments ON: - Create pseudonymous_id (rotateable) stored in first-party cookie or localStorage. - Collect event stream with pseudonymous_id + hashed identifiers + feature vectors → feature store (access controlled). - Model training pipeline consumes only pseudonymous records; outputs model parameters used to assign variants or personalize content. - Experiment assignment service reads pseudonymous_id + model rules → deterministic bucketing.- Third-party ON: same payloads forwarded to named vendors with contractual DPAs.

Technical & compliance details- Store explicit consent timestamp + version in consent DB; include experiment-level tag. Check consent before any pipeline enqueues events.- Enable “soft-sampling”: when user revokes personalization, stop using pseudonymous_id in future assignments and anonymize (delete) their feature vectors after a configurable retention (e.g., 30 days) or immediately on request.- Retention & minimization: keep raw event data for X days for analytics (default 90), store aggregated metrics indefinitely. Document deletion workflow for data subject requests.- Logging & audit: every experiment run includes consent-filtering logs; ties experiment results to consent cohorts (consented vs non-consented).- Legal basis: use consent for personalization and third-party sharing; use legitimate interest or contract for Essential/Performance only with clear justification.

UX details- Present concise modal on first visit with “Manage choices” link.- Per-experiment banner when personalization affects a visible change (“This experience uses personalization — change settings”).- Easy revoke in account settings with one-click “Stop personalization and remove profile”; show consequences (less personalized experience).- Provide “How your data is used” link with diagram and vendor list.

Example pseudocode (consent gating):

python

def ingest_event(event, user):
    consent = consent_store.get(user.id)  # or cookie-based
    if not consent.essential:
        return  # reject
    if event.type in ANALYTICS_EVENTS and consent.analytics:
        analytics_queue.enqueue(anonymize(event))
    if consent.personalization:
        pseud_id = ensure_pseudo_id(user)
        personalization_queue.enqueue(attach_id(event, pseud_id))

Why this works- Users get clear, granular control and meaningful defaults (privacy-first).- Data pipelines enforce consent at ingestion and preserve model utility via pseudonymization.- Retention, deletion, audit, and vendor controls ensure regulatory compliance while allowing experiments to run on appropriate cohorts.

MediumTechnical

129 practiced

As part of vendor risk assessment you must audit an analytics provider who claims to pseudonymize and delete data. List the technical and process checks you would perform, give sample queries or tests to validate deletion, and recommend contractual protections.

Sample Answer

Approach: split checks into technical (data, models, infra) and process (policies, people, evidence). For each, include concrete tests/queries and recommended contract clauses.

Technical checks & tests- Data mapping & lineage: obtain schema, ingestion pipelines, and sample row IDs. Verify where identifiers are stored and transformed.- Pseudonymization verification: - Confirm transformation is one-way (HMAC/hashed with secret salt) and salt/key management practices. - Test determinism: send two records with same PII and verify pseudonyms are identical (if claimed deterministic) or different (if claimed non-deterministic). - Re-identification test: sample 100 records, attempt to link pseudonyms back to public datasets or via small-variance attacks.- Deletion verification: - Logical deletion: check flags/soft-delete columns and downstream propagation. - Physical deletion: verify removal from primary store, backups, analytics DBs, logs, caches. - Sample SQL/API tests:

sql

-- Verify record absent by original identifier
SELECT COUNT(*) FROM analytics.events WHERE original_email = 'user@example.com';

-- If pseudonym stored, check absence by pseudonym value
SELECT COUNT(*) FROM analytics.events WHERE pseudonym = 'abc123hash';

bash

# Verify API returns 404/empty for deleted user
curl -i -X GET "https://vendor/api/users/user@example.com" -H "Authorization: Bearer ..."

- Hash consistency checks:

python

# If vendor provides HMAC function details, recompute on sample and compare
import hmac, hashlib
h = hmac.new(b'secret_salt', b'user@example.com', hashlib.sha256).hexdigest()

- Backup & snapshot checks: request snapshot manifests and sample restoration attempts to confirm deleted records are not present in retained backups. Ask vendor to run a restore-to-temp and run the SQL queries there.- Metadata & logs: request audit logs showing deletion events (timestamp, actor, request id) and verify immutability (WORM storage or append-only logs).- Model artifacts: ensure models trained on data are either retrained after deletion or that features do not store reversible identifiers. Run feature-store queries for presence of identifiers.

Process checks- Policies: review data retention, deletion, key rotation, access control, and incident response docs.- Personnel: review roles, least privilege, background checks, subcontractor use.- Testing & attestation: check frequency of internal audits, third-party SOC2/GDPR DPIA reports, and deletion proof procedures.- SLA & breach handling: verify time-to-delete, notification timelines, and remediation steps.

Contractual protections (recommended clauses)- Data Processing Addendum (DPA) mandating pseudonymization standards (e.g., HMAC with managed keys), key management, and prohibition on re-identification.- Deletion SLA: specify exact deletion windows (e.g., 30 days), scope (primary, backups, caches, logs), and required proof (audit logs and signed attestations).- Right to audit: on-site/remote audit rights, ability to require vendor to run deletion verification on demand, and restore-to-temp test rights.- Certifications & reporting: require SOC2 Type II / ISO27001 / annual penetration tests and delivery of DPIA.- Remedies & indemnity: financial penalties, breach notification timelines, and indemnity for regulatory fines arising from vendor failure.- Data return/portability and secure destruction procedures with certificate of destruction.

Result & reasoning: these checks validate both technical enforcement (can data be reconstructed or is it truly deleted) and governance (are practices reliable). As a data scientist, prioritize tests that prove residual identifiers don't influence models and that deletion is end-to-end (including backups and derived features).

HardTechnical

83 practiced

Derive the Renyi differential privacy (RDP) accounting for one epoch of DP-SGD with Poisson subsampling and Gaussian noise. State necessary assumptions, compute RDP at order alpha for a single step, and show how to aggregate across steps to obtain final (epsilon,delta). You may start from the standard Gaussian RDP formula.

Sample Answer

Assumptions- Adjacent datasets differ by one example (add/remove); per-example gradient clipping enforces l2-sensitivity 1 for each sampled example.- Poisson subsampling with rate q (each example included independently with prob q) per SGD step.- Gaussian noise with std dev σ added to the summed (and scaled) clipped gradients.- Steps are independent; there are T gradient steps in the epoch (or across epochs you can multiply T).

Single-step RDP at order αStart from the Gaussian RDP formula for a mechanism that adds N(0,σ^2) noise to a query with ℓ2-sensitivity 1: RGD(α) = α/(2σ^2). For Poisson subsampling, use the standard moment-accounting expansion (Wang, Balle, Kasiviswanathan; Mironov variants). The per-step RDP at integer order α ≥ 2 satisfies:

R_step(α) = (1/(α−1)) · log(1 + ∑_{j=1}^{α} C(α,j) q^j [exp(j(j−1)/(2σ^2)) − 1] )

where C(α,j) = binomial(α, j). (For non-integer α one can replace binomial with Gamma-based generalization; practical code uses the same series with α integer or uses tight bounds in literature.)

Explanation: subsampling mixes the distribution between “sampled” and “not sampled” cases; the j-th moment contribution corresponds to j sampled neighboring contributions producing a shift of magnitude j, yielding factor exp(j(j−1)/(2σ^2)). The log and normalization (1/(α−1)) give Rényi divergence of order α.

Composition across stepsRDP composes additively: for T independent steps (same q,σ), total RDP at order α is

R_total(α) = T · R_step(α).

Convert RDP → (ε,δ)Given R_total(α), for any α>1 and target δ∈(0,1) we get an (ε,δ)-DP guarantee:

ε(α) = R_total(α) + (log(1/δ))/(α−1).

Choose the α that minimizes ε(α) (numerically search α over integers or continuous α>1) to produce the tightest ε.

Edge notes / practicalities- For small q or large σ, the series converges quickly; for implementation use log-sum-exp for numerical stability.- There exist tighter closed-form bounds (e.g., using Poisson subsampled Gaussian bounds in Balle & Wang) but the series above is standard and implementable.- If gradients are averaged rather than summed, adjust sensitivity and σ accordingly (scaling factor).- For non-integer α use interpolation or continuous formulas from the literature; in practice privacy accountants search α in a dense grid.

Unlock Full Question Bank

Get access to hundreds of Privacy-Preserving Analytics and Experimentation interview questions and detailed answers.

Join thousands of developers preparing for their dream job.