Privacy-Enhancing Technologies and Anonymization Questions

Technical safeguards that reduce identifiability: anonymization, pseudonymization, tokenization, differential privacy, and related privacy-enhancing technologies. Covers the difference between anonymized and pseudonymized data, re-identification risk, and when each technique is appropriate. Includes evaluating the privacy-utility tradeoff of a given technical control.

HardTechnical

39 practiced

Prototype a privacy-preserving k-nearest-neighbors (kNN) search where the client holds a query vector and the server holds a dataset of indexed vectors. Using additive secret sharing as your primitive, implement a simulated server-side protocol that computes Euclidean distances without the server learning the client's query and without revealing dataset vectors to the client beyond the returned k nearest items. Provide code for share generation, pairwise distance computation using shares, and result reconstruction. Discuss communication and computation costs.

Sample Answer

Approach (short): use 2-party additive secret sharing: client splits query q into q0,q1 (q0+q1=q mod p). Server splits each dataset vector x into x0,x1 (or keeps x0 locally and gives x1 to helper). Two non-colluding servers S0,S1 hold shares and use Beaver triples to multiply shares for cross-terms. They compute shares of squared Euclidean distance: ||q-x||^2 = ||q||^2 + ||x||^2 - 2 q·x. All operations on shares; final distance shares returned to client for reconstruction of only k indices.

Code (simulated two servers + client, small finite field p):

python

import random
import heapq

p = 2**61-1  # prime-like for modular arithmetic

def share_vector(v):
    """Client: produce two additive shares"""
    s0 = [random.randrange(p) for _ in v]
    s1 = [(vi - s0i) % p for vi, s0i in zip(v, s0)]
    return s0, s1

def beaver_triple(d):
    """Generate Beaver triple shares for vectors of dim d.
    Returns (a0,a1),(b0,b1),(c0,c1) with c = a*b elementwise.
    """
    a = [random.randrange(p) for _ in range(d)]
    b = [random.randrange(p) for _ in range(d)]
    c = [(ai*bi) % p for ai,bi in zip(a,b)]
    a0, a1 = share_vector(a)
    b0, b1 = share_vector(b)
    c0, c1 = share_vector(c)
    return (a0,a1),(b0,b1),(c0,c1)

def beaver_mul(u_sh, v_sh, triple):
    """Two-party Beaver multiplication on vectors.
    u_sh = (u0,u1) locally known shares per party; we'll simulate both parties.
    triple = ((a0,a1),(b0,b1),(c0,c1))
    Returns (w0,w1) additive shares of elementwise product u*v
    """
    # simulate both parties' local computations
    a0,a1 = triple[0]
    b0,b1 = triple[1]
    c0,c1 = triple[2]
    # reconstruct a,b,c globally (simulator)
    d = len(a0)
    a = [(a0[i]+a1[i])%p for i in range(d)]
    b = [(b0[i]+b1[i])%p for i in range(d)]
    c = [(c0[i]+c1[i])%p for i in range(d)]
    # players compute e = u - a, f = v - b on their shares, then reconstruct e,f (requires open)
    u = [(u_sh[0][i]+u_sh[1][i])%p for i in range(d)]
    v = [(v_sh[0][i]+v_sh[1][i])%p for i in range(d)]
    e = [(u[i]-a[i])%p for i in range(d)]
    f = [(v[i]-b[i])%p for i in range(d)]
    # product = c + e*b + f*a + e*f
    prod = [ (c[i] + e[i]*b[i] + f[i]*a[i] + e[i]*f[i])%p for i in range(d)]
    # now split prod into shares
    w0, w1 = share_vector(prod)
    return w0, w1

# Example protocol: compute squared distance shares between query q and dataset X (n x d)
def server_side_distance_shares(q, X):
    # client shares q -> q0,q1
    q0,q1 = share_vector(q)
    # server splits each x into x0 (kept) and x1 (sent to helper) -> simulation: share_vector
    X0 = []
    X1 = []
    for x in X:
        x0,x1 = share_vector(x)
        X0.append(x0); X1.append(x1)
    # Precompute ||x||^2 shares locally for server halves
    x_norm_shares = []
    for x0,x1 in zip(X0,X1):
        # compute elementwise squares and sum without revealing: we can locally compute squares on shares via Beaver
        # For simplicity in this simulated server-side protocol, compute full norm and then split (server knows x)
        x = [(a+b)%p for a,b in zip(x0,x1)]
        norm = sum((xi*xi)%p for xi in x) % p
        n0, n1 = share_vector([norm])
        x_norm_shares.append((n0[0], n1[0]))
    # client would provide q shares to servers; we simulate both shares available
    # compute ||q||^2 shares similarly
    q_norm = sum((qi*qi)%p for qi in q) % p
    qn0, qn1 = share_vector([q_norm])
    # compute q·x via beaver multiplications per vector
    d = len(q)
    dot_shares = []
    for x0,x1 in zip(X0,X1):
        # u_sh = (q0, q1), v_sh = (x0, x1)
        triple = beaver_triple(d)
        prod0, prod1 = beaver_mul((q0,q1),(x0,x1), triple)  # elementwise products shares
        # sum elements to get dot product shares
        s0 = sum(prod0)%p
        s1 = sum(prod1)%p
        dot_shares.append((s0,s1))
    # Now distance share per server: dist = ||q||^2 + ||x||^2 - 2*dot
    dist_shares = []
    for (x_n0,x_n1),(s0,s1) in zip(x_norm_shares, dot_shares):
        # party0 share:
        d0 = (qn0[0] + x_n0 - (2*s0)%p) % p
        d1 = (qn1[0] + x_n1 - (2*s1)%p) % p
        dist_shares.append((d0,d1))
    return dist_shares  # list of (share0, share1) per data point

def reconstruct_and_select(dist_shares, k):
    # client reconstructs full distances and selects k smallest indices
    dists = [ (d0+d1)%p for d0,d1 in dist_shares ]
    # convert modular to integers (assume values < p/2)
    dists = [di if di <= p//2 else di-p for di in dists]
    idxs = heapq.nsmallest(k, range(len(dists)), key=lambda i: dists[i])
    return idxs, [dists[i] for i in idxs]

# quick demo
if __name__ == "__main__":
    q = [3,4]
    X = [[0,0],[3,4],[5,5],[2,1]]
    shares = server_side_distance_shares(q,X)
    print(reconstruct_and_select(shares, k=2))

Key points / reasoning:- Use identity ||q-x||^2 = ||q||^2 + ||x||^2 - 2 q·x, so only one secure dot-product per candidate.- Beaver triples enable secure multiplications for elementwise products; shares keep raw values hidden.- In practice precompute triples offline (OT-based generation) to reduce online cost.

Costs:- Communication: client sends q shares (O(d) field elements). For n dataset vectors, servers must open e,f during Beaver multiplication (opening costs O(d) per dot product) — overall online communication O(n*d) field elements (can be optimized via packing/OT extension).- Computation: per data point need d secure multiplications (via Beaver) → O(n*d) multiplications and O(n*d) local adds. Preprocessing (triple gen) is heavy but offline.- Client reconstruction: receives 2 shares per distance (or one party sends its shares) → O(n) elements, then selects k smallest.

Security notes:- Requires two non-colluding servers or honest majority; leakages occur when opening e,f in Beaver—these are masked by triples. Client learns only returned k indices and distances; server never reconstructs q.- For production, use fixed-point encoding, batching, and optimized triple generation (OT extension or homomorphic hybrid).

MediumSystem Design

35 practiced

Design a monitoring and governance system to track privacy-budget (ε, δ) consumption across multiple ML services and teams in production. Include APIs or SDK calls for services to log privacy-consuming operations, a central privacy ledger, dashboards, alert rules for budget exhaustion, enforcement mechanisms (rate-limiting or automatic throttling), and audit trails for compliance. Discuss trust boundaries and how to prevent teams from bypassing the ledger.

Sample Answer

Requirements:- Track cumulative (ε, δ) per team/service, per dataset and per user cohort; strong audit trail; real-time alerts and enforcement; tamper-evident ledger; low overhead SDK/API.

High-level architecture:- SDKs (Python/JS/gRPC) embed into services → Privacy Gateway (sidecar or central) → Central Privacy Ledger (append-only DB + ledger signer) → Policy Engine → Alerts/Dashboard/Enforcer → Audit store (WORM).

APIs / SDK (example):

python

# python SDK
from privacy_sdk import PrivacyClient
pc = PrivacyClient(service="recsys", team="ads")
# log a DP query consuming (epsilon, delta)
pc.record_dp_op(operation="gaussian_mechanism", epsilon=0.2, delta=1e-6, dataset="user_events", tags=["batch_job"])

Central privacy ledger:- Append-only store (e.g., immutable logs on cloud storage + signed Merkle-tree roots in KMS-backed signer). Each record: timestamp, service, team, dataset, op, ε, δ, request id, signature.

Policy Engine & Enforcement:- Maintains per-scope budgets, windows, and aggregation rules. On record, computes remaining budget; returns OK, WARN, or REJECT.- Enforcement modes: advisory (log/alert), soft-throttle (rate-limit new DP ops), hard-block (return error). Implement as sidecar proxy or API gateway plugin for low latency.

Dashboards & Alerts:- Real-time metrics (spent vs budget), per-team drilldown, trend forecasting (burn rate). Alert rules: threshold (%) and forecasted exhaustion within window. Integrate with PagerDuty/Slack.

Audit trails & Compliance:- Store signed ledger entries, immutable snapshots, exportable reports, queryable by compliance team. Include proofs (Merkle proofs) to detect tampering.

Trust boundaries & anti-bypass:- Place Privacy Gateway as required network hop (service mesh or firewall rules). Enforce via infrastructure policy: only gateway URL allowed in VPC, IAM roles restrict direct writes to data stores. Require signed tokens issued by CI/CD identity to call downstream DP libraries; SDKs verify ledger-ack signatures.- Monitor for suspicious raw access patterns (direct DB queries) via data access logs and DLP agents; integrate with IAM to revoke keys for offenders.- Use attested runtimes (e.g., mTLS, mutual TLS, or workload identity) and signed SBOM for services allowed to call ledger.

Preventing bypass examples:- Sidecar injection by platform ensures all egress routed through gateway.- Network policies + egress firewall deny external endpoints.- Periodic reconciliations: compare ledger consumption against observed noise levels or differential privacy reconciliations from randomized audits; flag mismatches.

Trade-offs:- Strong enforcement increases latency and operational complexity; advisory mode eases adoption. Start with audit/advisory, then migrate to hard enforcement.

MediumTechnical

43 practiced

You need to train a DP model on a dataset with rare but high-value classes. DP's clipping and noise can disproportionately hurt rare classes. Propose practical strategies (e.g., importance weighting, per-class clipping, class-aware augmentation, synthetic examples, separate privacy budgets) to preserve utility for rare classes while maintaining DP guarantees. Discuss privacy implications of each approach and how you'd evaluate fairness and utility trade-offs.

Sample Answer

Situation: We must train a differentially private model on imbalanced data where rare classes are high-value; standard DP-SGD (per-sample clipping + noise) often shrinks rare-class gradients causing poor utility for those classes.

Solution (practical strategies + privacy implications):

1) Importance weighting in loss- Approach: multiply per-example loss by a precomputed class weight (e.g., inverse freq) so rare-class gradients have larger magnitude before clipping.- Privacy implication: weighting changes sensitivity — if weights are public and fixed, DP accounting remains valid but noise must be calibrated to the maximum weighted gradient norm. Do not derive weights from private labels at training time unless computed with DP.

2) Per-class clipping thresholds- Approach: set separate clipping norms C_c for each class and apply per-example clipping using the threshold for that example’s class.- Privacy implication: allowed if thresholds are fixed in advance (or computed from public data). If thresholds are adaptively learned from private data, that step must be made DP. When using multiple C_c, the effective sensitivity is max_c C_c; noise calibrated accordingly or per-class noise added with composition accounting.

3) Class-aware augmentation & synthetic examples- Approach: augment rare-class examples (transformations) or generate synthetic examples via DP-trained generative models (DP-GAN/DP-VAE).- Privacy implication: naive augmentation is fine if transformations are deterministic/public. Synthetic generation must be DP (train generative model with DP-SGD or use PATE) — otherwise you risk leaking membership. DP synthetic data reduces direct training exposure but consumes privacy budget.

4) Separate privacy budgets / PATE and teacher ensembles- Approach: allocate more privacy budget to rare-class teacher votes or use PATE to privately label synthetic/unlabeled data so student learns better rare-class patterns.- Privacy implication: splitting budget is valid but reduces budget for other parts; PATE provides tight per-class accounting but requires careful teacher ensemble design and may need supplementary unlabeled data.

5) Two-stage training / fine-tuning- Approach: train a DP base model, then privately fine-tune on a rebalanced dataset (with DP-SGD using adjusted clipping/noise) or privately reweight gradients for rare classes.- Privacy implication: composition accumulates privacy loss; use advanced/Rényi DP accounting to track cumulative epsilon.

How to maintain DP guarantees in practice- Make any hyperparameters derived from private data computed via DP mechanisms or set from public sources / domain knowledge.- Use per-sample gradients + DP-SGD; calibrate noise to the true sensitivity considering weights/clips.- Use Rényi DP / moments accountant to track (epsilon, delta) across all steps (training, synthetic generation, hyperparameter tuning if private).

Evaluation strategy (utility vs fairness trade-offs)- Utility: report per-class metrics (precision, recall, F1), AUC per class, calibration (ECE) for rare classes, and overall loss.- Fairness/resilience: measure disparity metrics (difference in recall/TPR across classes), minimum-class performance, and business-specific utility (cost-weighted error).- Privacy-utility curve: sweep noise multiplier and clipping thresholds and plot per-class metrics vs epsilon to visualize trade-offs.- Statistical significance: bootstrap per-class metrics and test whether improvements for rare classes are real.- Monitoring for leakage: conduct membership inference and training-data-attribution tests on rare classes to verify no undue leakage.- Operational checks: track model confidence on synthetic vs real rare examples and perform a holdout DP evaluation (if possible).

Example practical plan- Fix class weights from public/business rules.- Choose per-class clipping C_c from prior experiments on public/auxiliary data; calibrate noise with moments accountant.- Optionally train a DP generative model (DP-GAN) to expand rare class, then train final DP classifier on combined data using DP-SGD.- Evaluate per-class recall and privacy-utility curves; iterate while ensuring all private computations are accounted for in final epsilon.

This combination preserves DP guarantees if every private operation (weight estimation from private data, threshold selection, synthetic generation) is covered by DP mechanisms; trade-offs are explicit: higher rare-class utility requires either larger sensitivity (hence more noise) or more privacy budget.

MediumSystem Design

36 practiced

Design a federated learning pipeline for 100 hospitals where raw data never leaves hospital boundaries. Requirements: (1) use secure aggregation so the central server cannot see individual updates, (2) apply differential privacy so the aggregated model has a provable privacy budget, (3) handle connection unreliability and stragglers. Describe communication protocol, cryptographic primitives, orchestration, failure handling, and how you'll compute and monitor the overall privacy budget across rounds.

Sample Answer

Requirements clarification:- 100 hospitals, raw data never leaves site.- Secure aggregation so server can't see individual updates.- Differential privacy with provable budget across rounds.- Robust to flaky connections and stragglers.

High-level approach:- Federated averaging with secure aggregation (Bonawitz et al. 2017 style) + centralized DP noise added in aggregated space (via distributed noise or client-side DP) and accountant (RDP) tracking.

Communication protocol & authentication:- gRPC over mTLS + mutual auth; protobuf messages carrying model version, clipped update, and small metadata.- Round orchestration: central coordinator (orchestrator) selects subset (or all) hospitals each round, announces round-id and model checkpoint, clients respond with readiness.

Cryptographic primitives:- Secure aggregation via additive masking + pairwise key agreements (Diffie-Hellman / ECIES) to produce pairwise masks; clients mask updates and send masked vectors. Server can only recover sum after masks cancel.- Use threshold-resilient mask-reveal (Bonawitz) that tolerates dropouts: clients secret-share their mask keys (Shamir secret sharing) to allow reconstruction only if enough participants remain.- Optionally hybrid: use lightweight homomorphic encryption for small models or verification.

Differential privacy:- Per-client clipping: each client clips gradient/update to L2 norm C before masking.- Two options for noise: 1) Central DP: clients add no noise; after secure-aggregate yields sum, a set of designated noise-contributing clients (or a distributed noise generation protocol) add calibrated Gaussian noise via secure aggregation protocol so server only sees noisy sum. This avoids trusting server. 2) Local DP: each client adds calibrated Gaussian noise to its clipped update before masking — simpler but higher noise.- Use Gaussian mechanism; compute per-round sigma given target per-round (ε_r, δ_r) via analytical Gaussian mechanism.

Privacy accounting and monitoring:- Use Rényi DP accountant (RDP) to compose rounds efficiently. Maintain per-client ledger of participation probabilities and per-round (α, ε_RDP) contributions; convert to (ε, δ) for reporting.- Track: number of participating clients per round, clipping norm C, noise sigma, sampling rate q = participants/100, and update cumulative RDP -> convert to ε at target δ (e.g., 1e-5).- Expose dashboards with current cumulative ε, per-round config, and alerts when ε budget near threshold.

Orchestration & ML pipeline:- Coordinator schedules rounds, maintains model versioning and checkpoints.- Client agent responsibilities: - Load local model, compute gradients/updates on local dataset. - Clip updates, optionally add DP noise (if local), perform secure-aggregation mask protocol (pairwise keys + secret shares), send masked update. - Persist unfinished work; resume across reconnects.- Server flow: - Wait until threshold T or timeout. If T reached, reconstruct sum via mask cancellation and secret-share reveals; add any additional distributed noise; update global model via average; publish model. - If not enough clients (below threshold) within timeout, either proceed with fewer participants (if allowed) or abort round and reschedule. T should be chosen to balance privacy (sampling) and reliability.

Failure handling & stragglers:- Use participant deadlines + partial aggregation: set a deadline per round; include only contributors who completed before deadline.- Design secure-aggregation to support dynamic dropout: each client secret-shares mask seeds at start so server can reconstruct masks for dropouts only if threshold met; this prevents single-client mask leaks.- Retries & backoff: clients retry key-establishment; exponential backoff for intermittent connectivity.- Checkpointing: clients and server persist state (model version, secret shares) to allow resumed protocol after transient failures.- Validation & fraud detection: per-client range checks on update norms (after decryption sum, server can sanity-check average magnitude); outlier detection can request re-run of round with flagged participants (requires careful privacy/ops process).

Practical considerations and trade-offs:- Bandwidth: sending full model vectors 100x is heavy — use model compression (quantization, sparse updates) and secure-aggregation compatible encodings (e.g., fixed-point encoding).- Compute & latency: Bonawitz-style secure aggregation has O(n^2) pairwise key ops at setup; mitigate with efficient crypto libraries and batching keys.- Privacy vs utility: local DP simpler but higher noise; distributed noise with secure aggregation gives better utility while preserving trustlessness.- Threshold selection: choose threshold (e.g., 60–80) to balance dropout tolerance vs privacy sampling rates.

Monitoring & audits:- Central logging of per-round metadata (no raw updates): participants, clipped norm, sigma, q, cumulative ε.- Periodic audits: cryptographic proofs of correct aggregation (verifiable computation or secure enclaves) for compliance.- Alerts if a hospital participates unusually often/rarely (affects privacy accounting).

Summary of key steps per round:1. Coordinator selects participants, broadcasts model and round-id.2. Clients compute clipped updates, establish pairwise keys, secret-share mask seeds.3. Clients mask and send masked updates over mTLS.4. Server waits until threshold or timeout; reconstructs sum via mask cancellation and secret-share reveals for dropouts.5. Server (or distributed protocol) adds calibrated Gaussian noise if central DP; compute updated model.6. Update RDP accountant with q, sigma, compose to update cumulative ε; publish metrics.

This design satisfies: raw data stays local; secure aggregation prevents server from viewing individuals; DP accountant provides provable budget; protocol tolerates dropouts via threshold secret-sharing and timeouts; monitoring ensures operational privacy visibility.

HardTechnical

32 practiced

A critical vulnerability is discovered in a third-party MPC library used in production. As the responsible ML engineer, outline an operational plan: immediate mitigations (patch, rollback, isolate affected services), steps for secure forensic analysis (collect logs, preserve evidence), communicating with legal/compliance and customers, assessing data exposure, and long-term actions (dependency policies, vendor risk controls, alternative libraries).

Sample Answer

Situation: While on-call I learned a CVE was published for an MPC (multi-party computation) library used by our model-serving pipeline. The library handled secure aggregation of feature vectors across partners; exploitability could expose model inputs/outputs.

Task: As the ML engineer responsible for production, I needed to contain risk immediately, preserve evidence for a forensic investigation, assess data exposure, notify stakeholders (legal, compliance, partners, customers), and implement long-term prevention.

Action:- Immediate mitigations (first 0–4 hours) - Convened an incident war room with SRE, security, and product owners and declared an incident. - Applied an emergency mitigation matrix: if a vendor patch was available and quickly verifiable, prioritized staged patching to canary nodes first; if no safe patch, initiated rollback to the last known-good container image and feature-flagged affected functionality. - Isolated affected services by removing them from load balancers and disabling inter-service RPCs to limit lateral risk. - Implemented compensating controls: rate limits, stricter auth between services, increased logging and alert thresholds.

- Secure forensic analysis (4–24 hours) - Coordinated with security/forensics to snapshot affected hosts, collect volatile memory dumps, container images, and network captures; followed chain-of-custody procedures to preserve evidence. - Centralized relevant logs (application, access, orchestration, network) and created immutable exports; tagged and retained storage with strict access controls. - Performed targeted revocation/rotation of credentials that could have been exposed (service keys, API tokens) and audited for anomalous use. - Reproduced exploit in a controlled sandbox to understand attack surface and observable indicators.

- Communication (first 24–72 hours) - Notified Legal & Compliance immediately with a concise risk brief (components affected, potential data types exposed, mitigations underway), providing timelines for updates. - Engaged vendor for technical details, timeline for patches, and CVSS context. - Drafted an external communication plan with PR/legal: a fact-based customer notification if there was a risk of sensitive data exposure, with guidance on steps we took and recommended customer actions. - Kept internal stakeholders updated via daily incident reports and an incident timeline.

- Assessing data exposure (24–72 hours) - Mapped data flows and inventories to determine which datasets touched the vulnerable library, prioritizing PII/sensitive partner data and model training labels/weights. - Queried logs for anomalous access patterns, unusual volumes, or unknown egress destinations during the vulnerability window. - If confirmed exposure, quantified records impacted, documented evidence, and worked with Legal/Compliance on regulatory reporting obligations.

- Long-term actions (post-incident) - Implemented stricter dependency policies: enforce SBOM for all services, automated vulnerability scanning in CI/CD, and mandatory staging/third-party library approval workflows. - Established a vendor risk program: contractual SLAs on security disclosures, patch timelines, and right-to-audit clauses; maintain an approved-vendor list and periodic security reviews. - Increased runtime protections: sidecar sandboxes for sensitive libraries, least-privilege service identities, and network micro-segmentation. - Evaluated and tested alternative libraries; created migration plans and modularized code to reduce coupling to any single third-party component. - Performed a postmortem within 7 days, produced an action item backlog with owners, and tracked completion metrics (time-to-patch, time-to-detect, mean-time-to-remediate).

Result / Learning:- Containment and rollback prevented confirmed data exfiltration. The incident highlighted gaps in dependency visibility and patch automation; within a quarter we closed those gaps, reduced median time-to-patch from days to hours, and improved stakeholder trust through transparent communication and measurable process changes.

Unlock Full Question Bank

Get access to hundreds of Privacy-Enhancing Technologies and Anonymization interview questions and detailed answers.

Join thousands of developers preparing for their dream job.