Meta Research Scientist Interview Preparation Guide - Staff Level (12+ Years)

1) Prioritize findings into opportunities (Week 0–1)

• Owner: Research lead (me) + PM
• Action: Map findings to user pain points and product areas; rank by impact/feasibility
• Success metric: Top 3 opportunities agreed by stakeholders
• Checkpoint: Weekly kickoff meeting (30 min)

2) Define hypotheses & experiments (Weeks 1–3)

• Owner: Research scientist + UX researcher + engineer
• Action: For each opportunity write 1–2 testable hypotheses and experiment designs (A/B, prototype, offline eval)
• Success metric: Completed experiment specs with sample sizes and metrics
• Checkpoint: Weekly design review

3) Build MVP prototypes (Weeks 3–7)

• Owner: Eng lead + ML engineer + UX
• Action: Implement lightweight models or mockups and instrumentation
• Success metric: Working MVPs deployed to test environment
• Checkpoint: Weekly demo + blocker log

4) Run experiments & collect data (Weeks 7–11)

• Owner: Data scientist + SRE
• Action: Launch experiments, monitor data quality, run statistical analyses
• Success metrics: Primary metric lift (e.g., +X% accuracy/engagement) and p < 0.05
• Checkpoint: Weekly results review with dashboard

5) Evaluate & decide (Weeks 11–12)

• Owner: Research lead + PM + Product Director
• Action: Synthesize results, compute ROI, and recommend integrate/iterate/stop
• Success metric: Clear go/no-go decisions for each opportunity
• Checkpoint: Decision meeting; minutes circulated

6) Roadmap integration & handoff (Weeks 12–16)

• Owner: PM (roadmap) + Engineering manager (delivery) + Research for transfer
• Action: Translate validated experiments into epics, acceptance criteria, and timelines
• Success metrics: Epics added to roadmap with owners and estimated delivery dates
• Checkpoint: Weekly execution standups for first quarter of delivery

Throughout: maintain a shared tracker (owners, timelines, experiments, metrics), weekly 30–45min syncs, and a living dashboard so stakeholders always see “what’s next.”

Direct answer: "Most impactful" is a claim about scale, reach, or durability of a change, not automatically the project with the single biggest percentage. Come with a short comparison across two or three candidate projects on a common yardstick (people affected, durability of the fix, or how core the process was), and be ready to justify why that yardstick and not just report a number.

A framework for ranking impact across projects

When you don't have hard numbers

• Use proxy metrics: adoption rate, ticket volume, "still in use N months later," or direct stakeholder feedback.
• State explicitly that it's a proxy, not a causal measurement, rather than dressing it up as a precise result.
• Reach and durability are often easier to state honestly than a precise causal percentage, and they're still a legitimate basis for "most impactful."

Worked example (illustrative, arithmetic shown)

Two candidate projects: Project A fixed a rare edge-case bug, reducing its error rate from an estimated 3% to under 1% on the narrow path it affected. Project B rebuilt the new-user onboarding flow that every signup passes through; its effect on conversion wasn't cleanly isolated, but it has been in production for 12 months and the product runs roughly 2,000 signups a month. Reach comparison: Project B touches 2,000 x 12 = 24,000 users over that period, versus Project A's narrow edge case affecting a small estimated fraction of a much smaller baseline. Project B is presented as "most impactful" on reach and durability grounds, even though Project A has the cleaner percentage, and that trade-off is named explicitly rather than hidden.

Trade-offs and pitfalls

• Picking the project with the single biggest reported percentage without checking how narrow its scope was is a common overclaim.
• Confusing "impactful to me personally" with "impactful to the business" weakens the answer under questioning.
• Presenting a proxy metric as if it were a measured causal result erodes credibility once challenged.
• Failing to acknowledge a plausible rival project when asked invites doubt about the whole answer.

Reproducible workflow (high level)

1. Define objective & success metrics (business and technical).
2. Prepare dataset + lineage snapshot (hashes, extraction queries).
3. Version code + infra (git commit, container image).
4. Run tracked experiments (experiment DB with seeds, hyperparams).
5. Produce deterministic training artifact (model weights, RNG state).
6. Evaluate with standard scripts and generate model card.
7. Package environment and handoff bundle; include automated smoke tests.

Required artifacts and why they matter

• Dataset manifest & lineage (URIs, extraction SQL, hashes, sampling script)

  • Why: Engineering and product can recreate, audit privacy/compliance, and estimate cost of retraining. Reduces surprises from data drift.

• Data schema & preprocessing code

  • Why: Ensures feature parity in production; prevents training-serving skew.

• Training code repo + exact commit and container image

  • Why: Engineers can reproduce build, debug failures, and optimize for infra.

• Experiment log (hyperparams, random seeds, metrics, hardware)

  • Why: Enables reproducibility, fair comparison, and capacity planning.

• Evaluation scripts and raw results (including unit and stress tests)

  • Why: Verifies performance claims and gives QA deterministic tests for CI.

• Model artifact (weights, tokenizer/vocab, conversion scripts)

  • Why: Direct input to deployment pipelines; avoids reimplementation.

• Environment spec (Dockerfile, conda/pip freeze)

  • Why: Prevents dependency drift; simplifies deployment and security scans.

• Model card & risk assessment (intended use, limitations, fairness tests)

  • Why: Non-research stakeholders (PM, legal, ops) get concise guidance on safe use and monitoring needs.

• Smoke tests and CI recipe (inference tests, latency/throughput benchmarks)

  • Why: Quick verification during integration; sets SLAs.

Each artifact maps to a tangible stakeholder need (repro, compliance, maintainability, deployment). Together they convert a research prototype into an engineering-ready, auditable, and low-friction deliverable.

Approach
Use the normal-approximation two-proportion sample size formula for equal allocation. Compute pooled variance under alternative pooled p̄ = (p1 + p2)/2 and use z-scores for alpha and power. Return ceiling to integer.

Code

import math
from scipy.stats import norm

def compute_sample_size(p_control, p_treatment, power=0.8, alpha=0.05):
    # z for two-sided alpha and for power
    z_alpha = norm.ppf(1 - alpha / 2)
    z_beta = norm.ppf(power)

    p1 = p_control
    p2 = p_treatment
    delta = abs(p2 - p1)
    if delta == 0:
        raise ValueError("p_control and p_treatment must differ to compute sample size")

    # pooled proportions for variance under H1 (conservative): p1*(1-p1)+p2*(1-p2)
    var = p1 * (1 - p1) + p2 * (1 - p2)

    # sample size per group (normal approximation)
    n = ((z_alpha + z_beta) ** 2) * var / (delta ** 2)

    return int(math.ceil(n))

Key reasoning

• Uses sum of variances of two independent binomials for equal allocation.
• Conservative, standard approach for two-sided tests.

Complexity & edge cases

• O(1) time and memory.
• Edge cases: delta ~ 0, p values outside [0,1], extremely small/large p require larger n; consider continuity corrections or exact methods (Fisher/poisson) for small samples.

Alternative
Use statsmodels.stats.power.NormalIndPower().solve_power for built-in functionality.

Direct answer. A shape-mismatch error tells you two tensors disagreed in dimension somewhere in the forward pass, but the traceback often points at the operation that FAILED, not the operation that introduced the wrong shape several layers earlier, so the debugging process is really about walking the shape forward from the input until it diverges from what you expect.

Step-by-step approach.

1. Print the input shape first, and compare it against what the first layer actually expects. A surprising number of shape bugs are simply "the input isn't shaped the way I assumed," not a bug in the model at all.
2. Use a model summary tool (or manually print .shape after each layer in a quick forward pass) to see the shape at every stage in one pass, rather than binary-searching by commenting out layers one at a time.
3. Add explicit shape assertions inside custom layers, at the point where a specific shape is assumed (assert x.shape[-1] == self.expected_dim, f"got {x.shape}"). This turns a downstream, confusing shape error into an immediate, precisely-located one the next time the bug is triggered, which pays for itself the first time someone else hits a variant of the same bug.
4. Write a small unit test with a known, fixed input shape that exercises just the suspect layer or block in isolation, rather than the whole model, so you can iterate on the fix without paying the cost of a full forward pass through everything else.

A concrete example of why step 1 matters. A very common real case: a model expects batch-first input (batch, seq_len, features) but receives (seq_len, batch, features) from a data loader or a different framework's convention. The shapes are individually valid tensors, nothing crashes until several layers in when a dimension that "coincidentally" matched for a while finally doesn't, at which point the error message points at a layer far from the true cause (the data loader).

The unit test that prevents recurrence. Something as small as:

def test_encoder_output_shape():
    x = torch.randn(4, 10, 32)  # (batch=4, seq_len=10, features=32), the CONTRACT this layer expects
    out = encoder(x)
    assert out.shape == (4, 10, 64), f"expected (4, 10, 64), got {out.shape}"

run in CI on every change to the layer or anything upstream of it, catches this class of bug the moment a shape contract is violated, rather than three deploys later when someone finally notices predictions look wrong.

Direct answer

Define a custom exception when a caller needs to programmatically distinguish and handle a specific failure mode; reuse a built-in (ValueError, TypeError, KeyError) when the failure is a generic, well-understood violation with no library-specific handling to offer. Keep except clauses as narrow as the exception you can actually recover from, never bare except:. Use raise NewError(...) from original whenever you translate a low-level exception into a library-level one, so the original traceback and type are preserved for debugging instead of discarded.

Structured elaboration

When to define a custom exception type

• Define one when a caller might reasonably want to catch this specific failure and do something different for it than for other failures (retry, fall back, surface a specific user-facing message). If every caller would handle it the same way as a generic ValueError, a custom type adds ceremony without adding value.
• Root the hierarchy at a single package-level base (for example MyLibError(Exception)), with specific failures subclassing it (ModelLoadError(MyLibError), InvalidDatasetError(MyLibError)). This lets a caller catch the single base class to mean "anything this library can go wrong in" without having to enumerate every subclass, while still allowing narrower catches where useful.
• Name exceptions for what failed semantically, not for the internal mechanism that detected it; a caller should be able to catch ModelLoadError without knowing or caring whether the implementation currently reads checkpoints from disk, S3, or a database.

How narrow an except clause should be

• Catch the most specific exception type you can actually do something about. A library function should generally let exceptions it cannot meaningfully handle propagate (or wrap them in its own type), rather than catching broadly and hiding the failure.
• Bare except: (or except Exception: used as a catch-all) at the library level is almost always wrong: it catches things like KeyboardInterrupt-adjacent control-flow signals in the case of bare except:, and in the case of except Exception: it hides programming errors (a typo causing an AttributeError) behind the same handling path as an expected, recoverable failure.
• Application-level code (the outermost layer, close to a user or an operator) is where broader catches are more defensible, specifically to provide a fallback, a user-facing error message, or a metric increment, since at that point there is nowhere further up to propagate to.

Exception chaining with raise ... from ...

• raise ModelLoadError(...) from original_exc sets original_exc as the new exception's __cause__, so the traceback shown to a developer includes both: "the following exception occurred while handling this one," preserving the original type, message, and traceback instead of losing them.
• Omitting from (raise ModelLoadError(...) inside an except block) still implicitly chains the original as __context__, shown as "during handling of the above exception, another exception occurred"; explicit from is preferred when the translation is intentional, since it documents the relationship as deliberate rather than incidental. raise ... from None suppresses the chain entirely, which is appropriate only when the original exception is genuinely irrelevant noise (rare in a library boundary).

Worked example

class LibraryError(Exception):
    '''Base class for all errors raised by this library.'''

class ModelLoadError(LibraryError):
    '''Raised when a model checkpoint cannot be loaded.'''

def load_checkpoint(path):
    raise FileNotFoundError(path)

try:
    load_checkpoint("/tmp/does-not-exist.pt")
except FileNotFoundError as e:
    raise ModelLoadError(f"could not load checkpoint at {e}") from e

This raises ModelLoadError: could not load checkpoint at /tmp/does-not-exist.pt, and the traceback CPython prints includes both exceptions, joined by the line "The above exception was the direct cause of the following exception:", with the original FileNotFoundError and its own traceback shown first. A caller who only knows about this library's API can catch ModelLoadError (or the LibraryError base) without needing to know the failure originated from a missing file rather than, say, a corrupted checkpoint format; a developer debugging the failure still sees the full original traceback via the chained __cause__.

Trade-offs & pitfalls

• A hierarchy that is too deep (many single-use subclasses that no caller ever catches individually) adds API surface for no behavioral benefit; a hierarchy that is too flat (one exception type for every failure) forces every caller to parse the message string to distinguish cases, which is fragile and not something the standard library's own conventions encourage.
• Catching broadly "to be safe" inside a library function is the single most common way debugging information gets lost: an unrelated bug (a typo, an off-by-one) gets silently reclassified as the same expected failure the except clause was written for, and the real bug ships unnoticed.
• Changing which exception type a public function raises (or removing a subclass from the hierarchy) is a breaking API change for any caller who catches it specifically; treat the exception hierarchy itself as part of the library's versioned public contract, not as an implementation detail.

Direct answer

The core move in an incident explanation to a non-technical audience is separating three layers up front: what happened (in plain terms, no root-cause mechanism), what it meant for them (impact, in terms they already track), and what's being done about it, then deliberately leaving out anything that doesn't serve one of those three. Below is an incident where I did that under time pressure, including delivering it live to a mixed engineering-and-business audience.

What to include, what to leave out, and how to decide

• Lead with impact, not sequence. Legal, sales, and executives care about what happened TO THEM first, which customers, how long, what's the exposure, the technical timeline is useful evidence, not the headline.
• Deliberately exclude logs, stack traces, and internal service names; they add authority for an engineering audience and add nothing but confusion for this one. A useful test: if a detail doesn't change what the listener should do next, leave it out.
• Give the cause in one plain sentence with no jargon, something like "a recent configuration change made one of our systems too slow to respond to a partner service in time," rather than either omitting cause entirely (which reads as evasive) or over-explaining the mechanism.
• When delivering this live rather than in a written report, whether it's a hallway update or presenting a postmortem verbally to a room that mixes engineers and business stakeholders, pause after the impact statement for questions before moving to cause. People worried about impact can't absorb a root-cause explanation until that worry is addressed first.

Worked example

Situation: during a high-traffic sales period, our payment service began intermittently failing checkout requests for roughly ninety minutes. Legal, sales leadership, and the executive team needed an explanation quickly.

Task: explain what happened clearly enough for them to act, communicate with affected customers, assess any obligations, decide on immediate next steps, without either alarming them with irrelevant detail or minimizing the impact.

Action: I opened with impact, in the terms they track: which customers were affected, for roughly how long, and that the issue was fully resolved and being watched closely. I gave the cause in one sentence: a recent configuration change made our payment service too slow to respond to our external payment gateway in time, causing some checkout attempts to fail. I described what we did in plain terms (reverted the change, increased how long we wait before giving up on a slow response, added an automatic circuit breaker so a slow dependency can't cascade into a wider outage) and what we were doing next (a deeper review, with a fuller technical writeup available to anyone who wanted it). I left out the specific error codes, service names, and configuration parameter, none of which changed what legal, sales, or the executives needed to do next. I paused for questions right after the impact statement, before moving on, and answered a legal question about customer notification obligations directly instead of routing it back to engineering jargon.

Result: legal and sales left with a clear, accurate picture of exposure and could communicate confidently with affected customers; the executive team approved the follow-up work (the circuit breaker and review) without needing to dig into implementation detail themselves, and a fuller technical postmortem was made available separately for the engineering team that wanted the mechanism-level explanation. I learned that pausing for questions right after the impact statement, before cause, kept people from tuning out a cause explanation they weren't ready to hear yet.

Trade-offs and pitfalls

Leaving out technical detail can read as evasive if you do it silently; I said "I'm not going to walk through the technical internals here, I'm glad to share those separately" so the omission was visible on purpose rather than hidden. The other pitfall is understating severity to keep the room calm, that erodes trust the moment the real scope becomes clear later. State the honest impact even when it's uncomfortable, and let the "what we're doing about it" section carry the reassurance instead of the impact statement itself.

Direct answer

Treat a single surprising segment finding, mobile shows a significant lift while the overall test is flat, as a hypothesis to validate, not a result to act on. Work through data-integrity checks, a formal interaction test with a multiplicity correction (since this segment was very likely noticed after the fact rather than pre-specified), and a set of robustness checks; then use an explicit decision process that weighs the statistical uncertainty against the business value and cost of being wrong, rather than a pure significance threshold, to choose between shipping to that segment, running a confirmatory follow-up, or abandoning the finding.

Structured elaboration

Step 1: verify the data before trusting the effect

• Check assignment balance within mobile specifically: treatment and control counts, and balance on key covariates, within the mobile slice alone, not just in aggregate.
• Check for instrumentation differences: missing events, a different SDK version, or a different exposure window on mobile that could produce a spurious effect having nothing to do with the treatment.
• Check for timing issues: did the mobile rollout start at the same time as the rest of the experiment, and is there any cross-over where a user appears in both device buckets across the test window.

Step 2: test the interaction formally

Fit an interaction model rather than comparing the mobile-only conversion rate to the mobile-only control rate informally:

import statsmodels.formula.api as smf

df["treat"] = df["assignment"].map({"control": 0, "treatment": 1})
model = smf.logit("conversion ~ treat + mobile + treat:mobile + signup_channel", data=df).fit()
print(model.summary())

Illustrative output (a hypothetical summary row, not a real run) would show a coefficient, standard error, z-value, and p-value for each term; the row that matters most here is treat:mobile. A row reading something like treat:mobile coef = 0.18, p = 0.02, alongside a treat main-effect coefficient close to zero and non-significant, is the pattern that supports a genuine mobile-specific effect: the interaction term carries the real signal while the main treatment effect alone looks flat, consistent with the original observation that the overall test showed no lift. A significant coefficient on treat:mobile is what actually supports "the effect really differs by device," rather than the mobile-only point estimate on its own, which can look large purely from within-mobile noise.

Step 3: correct for multiplicity honestly

Ask directly whether mobile was a subgroup chosen before the test ran or one noticed afterward because it happened to look interesting. If it was not pre-specified, and in practice it usually was not when this kind of question comes up, apply a multiplicity correction appropriate to however many segments were actually eyeballed (even informally) before mobile stood out, or at minimum treat the raw p-value as an optimistic upper bound on how surprising this finding really is.

Step 4: check power on the mobile slice itself

Compute the sample size and event count within mobile alone and the confidence interval width on its effect estimate. A wide interval or a small mobile sample means the "significant" reading is fragile, and this matters even more when mobile is a genuinely small-traffic segment (a specific device class or platform with limited volume) rather than merely a smaller slice of a large population: in that case a confirmatory follow-up restricted to the same segment may take a long time to reach adequate power, or may never fully reach the same statistical bar as the overall test, which is itself part of the decision, not a reason to ignore the finding.

Step 5: robustness checks

• Look at related metrics (engagement, retention, complaint or refund rate) to see whether they move in a direction consistent with the primary metric's mobile-specific lift, or whether the primary metric is moving alone in a way that is harder to explain.
• Check whether the effect is stable over the test window or concentrated in a short burst of days.
• Check finer sub-slices of mobile (iOS versus Android, OS version) to rule out the effect actually being driven by one narrow slice within "mobile" rather than the device class as a whole.
• Re-run with alternative covariate adjustment and see whether the interaction coefficient is stable.

Worked example: the decision process

Rather than a bare "p < 0.05 so ship it" rule, weigh four inputs explicitly: how strong the statistical evidence is after the checks above, how large and reliable the resulting business value would be if the effect is real, how costly it is if the segment is shipped and the effect turns out not to be real, and how long a confirmatory follow-up on that segment alone would realistically take to reach adequate power given the segment's own traffic volume.

• Strong evidence, low cost of being wrong, fast to confirm: ship a small, reversible rollout to the segment while a confirmatory read continues, since the downside of being wrong is small and quickly detected.
• Moderate evidence, or the segment is small enough that a proper confirmatory test would take a long time to reach power: this is the case worth naming explicitly, since waiting for full statistical certainty may never be practical for a genuinely small segment. Here, the decision becomes an explicit risk-tolerance call: state the estimated cost of shipping on an unconfirmed finding versus the estimated cost of never acting on a real effect because the segment could never generate enough data to confirm it on its own, and make that trade-off visible to the decision-maker rather than deferring it to a p-value the segment may structurally never be able to produce.
• Weak evidence, or a moderate cost of being wrong: run a dedicated, pre-specified confirmatory experiment targeted at the segment before making any production change, treating the original finding purely as the hypothesis that justified the follow-up.
• Evidence disappears after the data-integrity and robustness checks: abandon the finding and document why, so the same slice does not get re-litigated the next time someone happens to look at it.

Trade-offs & pitfalls

• Treating an unadjusted subgroup p-value as decisive. The interaction test plus a multiplicity correction is what separates a real segment effect from one of several plausible slices that happened to look significant.
• Waiting indefinitely for a small segment to reach the same statistical bar as the overall test. For a genuinely low-traffic segment, that bar may not be reachable on a useful timeline; the decision framework needs to say what happens in that case rather than defaulting to inaction.
• Ignoring instrumentation as a candidate explanation. A device-specific logging or SDK difference is a mundane but common cause of an apparent segment effect and should be ruled out before any statistical machinery is trusted.
• Shipping on a single significant slice with no plan to re-check it. Even a reversible segment rollout should carry a defined follow-up read, not be treated as a closed decision the moment it ships.

Direct answer

Do not treat the metric win and the UX risk as opposing bets. Before building anything, agree with design and product on the primary success metric and on explicit guardrail metrics chosen specifically to catch the kind of harm the primary metric would not see, then validate cheaply with a prototype or a small qualitative test before committing to a live experiment sized to detect both.

Structured elaboration

Agree on what "good" means before anyone builds

The primary metric, say a conversion or engagement number, tells you if the change works on its own terms. Guardrail metrics are chosen specifically because they would catch harm the primary metric is blind to, such as task completion, return usage a week later, or support-ticket volume. Naming guardrails upfront, with agreed thresholds, prevents "we'll know it if we see it" arguments after the fact.

Validate cheaply before going live

A clickable prototype or a small moderated usability session can surface confusion or trust issues that the metric alone cannot catch, at a fraction of the cost of a live experiment. This is not a substitute for the experiment, it is a cheap filter that catches the worst ideas before they reach real users.

Run a bounded experiment, not a full rollout

Start with a small slice of traffic, watch both the primary metric and the guardrails, and decide the stopping rule, meaning what result on which metric ends the test, before the test starts, not after you see the numbers.

Decide and communicate together

If the primary metric improves but a guardrail moves the wrong way, that is a real finding, not a technicality to explain away. Whether to ship, iterate, or drop the idea is a joint call between design, product, and whoever owns the guardrail metric, made against the thresholds agreed upfront.

Worked example

Design proposes reordering a list of recommended items to increase click-through rate. The concern is that users may have learned to expect a stable, predictable order, and reordering it could hurt their ability to quickly find what they are looking for on repeat visits, something click-through rate would not show because a user can click more and still be more frustrated.

Before building, the group agrees the primary metric is click-through rate, and the guardrails are task completion rate (did the user's search end in the outcome they were after) and a return-usage check at one week out. A moderated usability test with a handful of participants on a clickable prototype surfaces that new users find the reordered list fine, but a couple of returning participants mention it "looks different" and take longer to find what they normally click first. That is a signal, not a stop sign: the team ships the change to a small slice of traffic, watches both metrics for an agreed window, and only expands the rollout if task completion holds steady alongside the click-through gain.

Trade-offs and pitfalls

Over-instrumenting every change with a full guardrail suite slows teams down and trains people to skip the process for anything that feels small. Guardrails should be chosen deliberately for the specific risk in question, not applied as a blanket checklist.

The sharpest failure mode is agreeing on guardrails in principle but not on thresholds, so when a guardrail moves slightly, the debate about whether it is a real regression happens after the data is already in and someone has already committed emotionally to shipping. Fixing the threshold before the test removes that fight.

Answer (Research Scientist perspective)

Overview / approach
I would train with DP-SGD: compute per-example gradients, clip each to norm C, aggregate, add calibrated Gaussian noise, then update model. Use a moments accountant (or Rényi DP accountant) to track (ε, δ) across iterations.

Key steps

• Per-example gradient clipping: for each sample i compute g_i, scale g_i ← g_i * min(1, C / ||g_i||2). This bounds sensitivity to C so noise calibration is valid.
• Noise addition: add Gaussian noise N(0, σ^2 C^2 I) to the summed/clipped gradient; σ chosen to meet desired (ε, δ) via accountant.
• Privacy accounting: use moments accountant or Rényi DP to compose per-step privacy loss tightly and convert to (ε, δ). Track cumulative privacy budget.

Trade-offs

• Lower ε (stronger privacy) requires larger σ → more utility loss. Smaller C reduces noise scale but may bias gradients if many g_i exceeded C.
• Batch size matters: larger batches reduce relative noise but increase per-step privacy composition speed; epochs and learning rate interact nonlinearly.
• Architecture/overparametrization often increases vulnerability to leakage, requiring stricter DP.

Empirical membership-leakage evaluation plan

1. Train two models (DP-SGD with target ε and baseline non-DP) on same data and seeds.
2. Attack: implement membership-inference attacks (black-box confidence-based, and white-box loss-based). For each, collect scores for members vs non-members.
3. Metrics: AUC-ROC, true positive rate at low FPR, precision@k, and attack advantage.
4. Ablations: vary ε, C, σ, batch size, and training epochs; report utility (test accuracy) vs leakage curves.
5. Statistical significance: repeat trials, report confidence intervals and effect sizes.

I would supplement results with calibration plots and per-class analyses to understand where DP helps or degrades utility most.