Privacy-Enhancing Technologies and Anonymization Questions
Technical safeguards that reduce identifiability: anonymization, pseudonymization, tokenization, differential privacy, and related privacy-enhancing technologies. Covers the difference between anonymized and pseudonymized data, re-identification risk, and when each technique is appropriate. Includes evaluating the privacy-utility tradeoff of a given technical control.
MediumTechnical
36 practiced
In Python with PyTorch, implement a function dp_aggregate(per_example_grads, clipping_norm, noise_multiplier, seed) that performs per-example gradient clipping and returns the differentially-private noisy aggregated gradient using Gaussian noise. Describe numerical stability, vectorized implementation, and memory trade-offs.
Sample Answer
To compute a DP noisy aggregated gradient in PyTorch you:1) clip each example's gradient to L2 norm <= clipping_norm,2) sum clipped gradients across the batch,3) add Gaussian noise scaled by noise_multiplier * clipping_norm,4) divide by batch size to return the noisy average gradient.Assumptions: per_example_grads is a list/tuple of gradient tensors, one tensor per parameter with shape (batch_size, *param.shape). Implemented in a fully vectorized way (no python loop over batch), stable numerics, reproducible via seed.Key points / reasoning:- Norm computed by summing squared flattened elements across parameters yields exact per-example L2 norms.- eps prevents division-by-zero for zero gradients (numerical stability).- Broadcasting factor avoids Python loops over batch; only loop over parameters (usually far fewer than batch).- Noise added to the summed gradient then divided by batch size matches DP-SGD convention: noise N(0, (sigma*C)^2) added to the sum, then average. Here sigma = noise_multiplier.- Reproducibility: using a torch.Generator seeded ensures deterministic noise on CPU; for CUDA, seed global CUDA RNG separately if required.Time & space:- Time: O(B * P) where P = total parameters, dominated by operations on per-example grads.- Memory trade-offs: storing per-example gradients requires O(B * P) memory; this can be prohibitive for large models / batches. - Alternatives: compute per-example gradient norms via gradient hooks + per-microbatch accumulation, or compute per-example gradients one microbatch at a time (memory-time trade-off). - Another alternative: compute norms only (cheap) then recompute clipped gradients in microbatches to avoid storing full per-example tensors.Edge cases:- Very small clipping_norm or large noise_multiplier may drown signal.- Zero gradients: handled by eps.- Mixed precision: convert to float32 for norm computations to avoid underflow/overflow.Alternatives:- Compute per-example norms using double precision for stability when parameter count is large.- Use per-parameter independent clipping (less common) or group clipping.
python
import torch
def dp_aggregate(per_example_grads, clipping_norm, noise_multiplier, seed=None, eps=1e-6):
"""
per_example_grads: list of tensors, each shape (B, *param_shape)
clipping_norm: float, max L2 norm per example
noise_multiplier: float, sigma (noise = N(0, (sigma*C)^2))
seed: optional int for reproducibility
returns: list of tensors same shapes as param (averaged noisy gradient)
"""
if seed is not None:
# reproducible across CPU/GPU
g_cpu = torch.Generator(device='cpu').manual_seed(seed)
# for CUDA, user can set torch.cuda.manual_seed_all(seed) outside if needed
batch_size = per_example_grads[0].shape[0]
device = per_example_grads[0].device
# 1) compute per-example squared norms in a vectorized way
# For each param, flatten per-example and compute squared sum
sq_sums = None
for g in per_example_grads:
# view as (B, -1) then sum squares over dim=1
s = g.view(batch_size, -1).pow(2).sum(dim=1)
sq_sums = s if sq_sums is None else sq_sums + s
norms = torch.sqrt(sq_sums.clamp_min(0.0)) # shape (B,)
# 2) compute clip factors: min(1, C / (norm + eps))
clip_factors = (clipping_norm / (norms + eps)).clamp(max=1.0) # (B,)
# 3) apply clip factors and sum; vectorized by broadcasting
clipped_sums = []
for g in per_example_grads:
# reshape clip_factors to broadcast to g: (B, 1, 1, ...)
factor_shape = [batch_size] + [1] * (g.dim() - 1)
factors = clip_factors.view(*factor_shape).to(g.dtype).to(device)
clipped = g * factors
summed = clipped.sum(dim=0) # sum over batch -> same shape as param
clipped_sums.append(summed)
# 4) add Gaussian noise per-parameter (noise added to the summed gradient)
noise_std = noise_multiplier * clipping_norm
noisy_aggregate = []
for s in clipped_sums:
if seed is not None:
# use CPU generator to create a tensor, then move to device to keep reproducible
noise = torch.normal(
mean=0.0,
std=noise_std,
size=s.shape,
generator=g_cpu,
device='cpu'
).to(device, dtype=s.dtype)
else:
noise = torch.randn_like(s) * noise_std
noisy = (s + noise) / float(batch_size) # return average noisy gradient
noisy_aggregate.append(noisy)
return noisy_aggregateMediumTechnical
41 practiced
Explain differential privacy (DP) for ML practitioners using the 4-part structure: (1) define DP simply (epsilon, delta intuition), (2) step-by-step how DP-SGD or output perturbation add noise and bound influence, (3) give real use-cases (federated learning, analytics), (4) discuss privacy-utility trade-offs and practical deployment challenges.
Sample Answer
1) Define DP simplyDifferential privacy (DP) formalizes the idea that a single individual's data should not noticeably change the output of an algorithm. Intuitively, an algorithm is (ε, δ)-DP if, for any two datasets that differ by one record, the probability of any output changes by at most a multiplicative factor e^ε plus δ slack. ε (epsilon) controls privacy strength: smaller ε → stronger privacy (less distinguishability). δ is a tiny failure probability allowing rare larger leaks.2) How DP-SGD / output perturbation add noise and bound influence- DP-SGD: During training, gradients computed on micro-batches are clipped to a fixed norm (bounds each example’s influence), then Gaussian noise proportional to the clipping norm is added to the averaged gradient before the optimizer step. A privacy accountant (e.g., moments or Rényi accountant) accumulates the per-step ε contribution to give a final (ε, δ).- Output perturbation: Train normally or compute a statistic, then add calibrated noise (Laplace/Gaussian) to model parameters or outputs proportional to the function’s sensitivity (max change caused by one record). Both mechanisms ensure individual contributions are masked by noise, providing a provable bound on information leakage.3) Real use-cases- Federated learning: Clients compute local gradients; DP-SGD or client-level DP prevents reconstruction of a client’s data from model updates.- Analytics & query release: Private histograms, counts, or aggregate statistics released with Laplace/Gaussian noise.- ML model release / fine-tuning: Differentially private fine-tuning (e.g., private BERT) to protect training examples in sensitive domains (health, finance).4) Privacy-utility trade-offs & deployment challenges- Trade-off: More noise (smaller ε) reduces utility/accuracy; you must choose ε based on legal/ethical requirements and acceptable performance loss. Tuning clipping norm, batch size, and noise multiplier affects both privacy cost and convergence.- Challenges: setting meaningful ε (stakeholder communication), accounting for composition across multiple releases, utility degradation for rare classes, hyperparameter tuning under DP, extra compute cost, and auditing/secure aggregation to prevent side-channel leaks. Practical deployments combine DP with secure aggregation, careful ε budgeting, and monitoring to balance privacy and model performance.
HardSystem Design
33 practiced
Design a privacy-preserving federated learning architecture to support 1,000,000 devices, target 10 training rounds per day, and keep model accuracy within 5% of centralized training. Include client selection, secure aggregation, DP accounting, bandwidth constraint assumptions (200 kbps upload), intermittent connectivity, and GDPR compliance considerations.
Sample Answer
Requirements & constraints:- Scale: 1,000,000 devices, 10 federated training rounds/day.- Accuracy: within 5% of centralized training.- Bandwidth: 200 kbps upload per device.- Connectivity: intermittent, variable latency.- Privacy: secure aggregation, differential privacy (DP), GDPR compliance.High-level architecture:- Orchestrator (cloud): schedules rounds, client selection, global model store, privacy accountant, aggregator coordinator.- Regional aggregators (edge data centers): collect encrypted client updates, perform secure-aggregation rounds (Bonawitz-style), forward aggregated updates to central aggregator.- Secure-aggregation service: multi-party protocol that never exposes individual updates.- DP accountant service: computes cumulative (ε, δ) using moments accountant / RDP.- Client runtime: local trainer, update compressor, secure-agg client, DP clipping/noise, retry/heartbeat module.Client selection:- Stratified, availability-aware sampling to reduce bias: sample by geography, device type, data distribution; oversample underrepresented strata until modeling objectives met.- Per round target: assume effective participation of N_eff ≈ 50k clients per round (empirical trade-off to reach accuracy within 5% for large models). Use importance sampling (clients with higher data diversity get higher probability).- Handle churn: maintain a pool of warm clients (pre-authorized, recently available) and a cold pool; use short enrollment windows (~5–10 min) per round.Bandwidth & update-size strategy:- Assume model baseline 10M parameters. Use sparsification + quantization + sketching: - Top-k sparsification (e.g., 0.5–2% updates) + 8-bit quantization + entropy coding → typical upload ≈ 50–200 KB. - At 200 kbps (25 KB/s) worst-case, allow upload window ~10s–60s; use staggered upload windows and regional aggregators to smooth load.- If model too large, use layer-wise freezing or per-round partial updates (e.g., fine-tune last layers most rounds, full model every few rounds).Secure aggregation & DP:- Use Bonawitz et al. secure-aggregation with ephemeral keys and Shamir-sharing to ensure server cannot decrypt individual updates. Implement hierarchical secure-aggregation: clients encrypt to regional aggregator groups, aggregators combine and forward masked sums to central.- On-device DP: per-client L2 clipping of local model delta to norm C, then add Gaussian noise scaled to C and noise multiplier σ before encryption. This minimizes communication of raw updates.- DP accounting: centralized RDP accountant that consumes participation rates and noise parameters to track cumulative ε. Target total ε ≤ target (e.g., ε ≤ 8 with δ = 1e-6) to stay within desired privacy budget while preserving accuracy; tune C and σ empirically.Intermittent connectivity & robustness:- Asynchronous fallback: allow late-arriving updates up to a staleness threshold; weight updates by staleness (decay) during aggregation.- Retries & partial uploads: resume-capable uploads and chunked transfers; clients send signed metadata so partial updates can be validated.- Client heterogeneity: adaptive local epochs by compute capability, and normalize contributions by number of local samples/epochs.Scalability:- Hierarchical aggregation reduces central bandwidth and computation: 1M devices → regional groups of ~10k–50k; regional aggregators perform secure-agg then forward partial aggregates.- Autoscaling orchestrator and aggregator services; use load-based sharding and queueing.- Monitoring: per-round participation, convergence metrics, privacy budget, and network load.Accuracy retention strategies:- Pretrain global model centrally where possible to give strong initialization.- Use more clients per round and tune local epochs and LR schedules; periodic full-model rounds (every K rounds) to reduce drift.- Personalization: mixed objective (global + local fine-tuning) to improve on-device performance without harming global accuracy.- Validate by hold-out centralized evaluation dataset and on-device validation signals.GDPR & legal considerations:- Data never leaves device—data minimization and purpose limitation.- Explicit, revocable opt-in consent for training; store consent logs and TTLs.- Right to be forgotten: orchestrator marks device/user id; future rounds exclude device, and model updates can be adjusted with selective unlearning strategies (influence-function-based or re-training partial).- DPIA, records of processing activities, subprocessors list, and Data Processing Agreement (DPA) with regional aggregator operators.- Pseudonymization of client IDs; minimize retention of metadata; encrypt logs at rest and in transit; audit trails.Trade-offs & tuning:- Privacy vs accuracy: higher σ reduces accuracy; tune noise/clipping and increase client count per round to compensate.- Bandwidth vs speed: heavier compression reduces fidelity; use adaptive compression based on client uplink.- Latency vs robustness: shorter enrollment windows speed rounds but reduce participation; use staggered scheduling.Validation & metrics:- Track central validation loss/accuracy vs centralized baseline; target ≤5% relative gap.- Track per-round ε, participation rate, effective sample size, staleness distribution, and average upload time.This design balances privacy (secure-aggregation + DP + GDPR practices), scalability (hierarchical aggregators, warm client pools), bandwidth limits (compression + partial updates), and robustness to intermittent connectivity while providing levers to meet the 10 rounds/day and accuracy targets.
HardTechnical
67 practiced
Discuss the challenges and practical considerations of applying differential privacy to pretraining large transformer language models. Cover per-step noise requirements, gradient clipping at scale, impact on downstream utility, compute cost, and practical alternatives such as private fine-tuning or PATE.
Sample Answer
Situation summary: Applying differential privacy (DP) to full-scale pretraining of transformer LLMs is technically possible but fraught with practical trade-offs — the two main axes are privacy cost (epsilon) which scales with training steps and the heavy engineering/compute burden of per-example gradient operations.Per-step noise & privacy accounting- DP-SGD adds Gaussian noise scaled to a noise multiplier σ and requires privacy accounting across all gradient-update steps (use Rényi DP or moments accountant). Pretraining runs for billions of tokens and millions of steps; given the composition theorem, either you end up with a very large cumulative epsilon (weak privacy) or you must set σ very large (which degrades utility).- Sampling rate (batch_size / dataset_size) and number of steps dominate the privacy budget. For long pretraining, to keep epsilon small you'd need very high noise or huge batch sizes to reduce the number of steps — both hurt convergence.Gradient clipping at scale- DP requires per-example (or per-microbatch) gradient clipping to bound sensitivity. Computing per-example gradients for transformers is memory- and compute-intensive: activations must be retained and backward passes executed per example or per microbatch.- Workarounds: microbatching + gradient accumulation, vectorized per-example gradients where possible, or approximate clipping (per-layer clipping, or use of "ghost clipping" approximations). These reduce privacy fidelity or add bias.- Large models exacerbate variance: clipping across many parameters can zero-out signal for some examples; adaptive clipping schemes help but add hyperparameters and complexity.Impact on downstream utility- High noise and aggressive clipping reduce pretraining signal, harming alignment of representations and downstream performance. Empirically, models trained with strong DP pretraining show degraded perplexity and downstream task accuracy unless model size, dataset size, or compute are substantially increased.- Mitigation: increase model capacity and dataset size to recover utility, tune clipping norm carefully, or restrict DP to sensitive subsets of the corpus only.Compute & engineering cost- Expect multix compute overhead: per-example gradient computation, larger batches, more steps to converge, and privacy accounting. Memory footprint grows (activations retention for microbatches), training time increases significantly (often 3–10x or more depending on implementation).- Implementation complexity: efficient per-example grad libraries, secure aggregation for distributed DP-SGD, robust privacy accounting pipelines, and careful hyperparameter sweeps.Practical alternatives- Private fine-tuning: Pretrain on public or non-sensitive data non-privately, then apply DP-SGD to fine-tune on the sensitive dataset (or only tune small adapters). This dramatically reduces number of DP steps and improves utility/cost trade-off. Use techniques: DP-Adam/DP-AdamW for optimizers, low-rank adapters or LoRA to limit parameter count.- PATE (Private Aggregation of Teacher Ensembles): Can work for classification-style supervision; for language generation it's harder but possible to apply to synthetic-labeling subcomponents (e.g., classifiers or reward models). PATE gives strong privacy guarantees but requires many teacher models and ensemble infrastructure.- Synthetic data with DP mechanisms: Train a DP model to produce synthetic data and then train a non-DP LLM on that data — utility depends heavily on synthetic quality.- Selective DP: only enforce DP on data known to be sensitive; keep public web data unnoised.Practical recipe (recommended)- Prefer DP for fine-tuning/adapters rather than full pretraining unless you have massive compute and an appetite for high epsilon.- If pretraining must be private: use very large batch sizes, increase model and dataset scale, implement efficient microbatch clipping, use RDP accounting, and target realistic epsilon (document trade-offs — values like ε≈1 are extremely costly; many practical systems accept ε in low double-digits).- Evaluate downstream utility across tasks and perform privacy/utility ablation; invest in adaptive clipping and noise scheduling; consider hybrid approaches (private fine-tuning + selective DP pretraining on sensitive shards).Takeaway: Full DP pretraining is theoretically straightforward but practically expensive and utility-damaging at current scales. For most production use-cases, private fine-tuning or adapter-level DP and PATE/synthetic-data approaches offer far better privacy/utility/compute trade-offs.
MediumTechnical
70 practiced
Explain k-anonymity, l-diversity, and t-closeness for tabular data anonymization. For a healthcare dataset with quasi-identifiers age, zip code, and gender, give a concrete example of a transformation to achieve k-anonymity and discuss the utility loss and re-identification risk.
Sample Answer
k-anonymity, l-diversity and t-closeness are progressive defenses against re‑identification in tabular data:- k-anonymity: each record is indistinguishable from at least k−1 others on the set of quasi-identifiers (QIs). Prevents direct linkage attacks that match QIs to external datasets.- l-diversity: within each k-anonymous equivalence class, the sensitive attribute(s) must have at least l “well-represented” distinct values, reducing attribute disclosure when attacker narrows to a class.- t-closeness: the distribution of a sensitive attribute in any equivalence class must be within distance t (e.g., Earth Mover’s Distance) of the attribute’s global distribution — protects against skew and semantic closeness attacks.Concrete transformation (QIs = age, ZIP, gender), target k=5:- Generalize age into 10-year buckets: 20–29, 30–39, …- Truncate ZIP to 3 digits (e.g., 94107 → 941**)- Suppress rare genders if needed or keep as-isExample equivalence class after transform:{ age:30–39, ZIP:941**, gender:Female } contains 5 patient records → k=5.Utility loss:- Numeric precision lost (age ranges instead of exact age) and geographic granularity reduced (ZIP3). Model features sensitive to exact age or fine-grained location (e.g., risk scores, local outbreak analyses) will degrade; measured by information loss metrics (e.g., classification AUC drop, normalized entropy).Residual re-identification risk:- Reduced but not eliminated. If an attacker has additional attributes (clinic visit dates, rare diagnosis) or combines with external datasets, linkage may still occur. l-diversity and t-closeness would further reduce attribute disclosure: for example require each ZIP3/age/gender block to contain at least l distinct diagnoses and ensure diagnosis distribution in the block matches the overall population within threshold t.Trade-offs:- Increasing k, l, or stricter t reduces disclosure risk but increases utility loss. Choose parameters based on risk model, downstream tasks, and regulatory requirements; validate with empirical utility tests and simulated linkage attacks.
Unlock Full Question Bank
Get access to hundreds of Privacy-Enhancing Technologies and Anonymization interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.