Google Research Scientist (Junior Level) - Comprehensive Interview Preparation Guide

Direct answer

Convert a sample-size target into a run duration in two steps: compute the required sample size per variant with the standard two-proportion test formula, then divide that by how much daily traffic actually lands in each variant (daily uniques times allocation), rounding up. The formula gives you a headcount; the traffic split is what turns a headcount into a calendar.

Structured elaboration

Step 1: the sample-size formula

For a two-sided test comparing baseline conversion $p_1$ against a target conversion $p_2$, at significance $\alpha$ and power $1-\beta$:

$n = \frac{\left(z_{1-\alpha/2}\sqrt{2\bar p(1-\bar p)} + z_{1-\beta}\sqrt{p_1(1-p_1)+p_2(1-p_2)}\right)^2}{(p_2-p_1)^2}, \quad \bar p = \frac{p_1+p_2}{2}$

This is per-variant sample size; assumes a two-sided test with no sequential peeking (one look at the end) and independent, one-conversion-per-user data. Sequential monitoring and multiple-comparisons corrections change this number and are a separate design decision, not part of the raw duration estimate.

Step 2: convert daily traffic into per-variant daily volume

$\text{daily per variant} = \text{daily unique visitors} \times \text{allocation share}$

Step 3: convert sample size into days

$\text{days} = \frac{n}{\text{daily per variant}}$

Round the sample size up (never down, since undershooting trades away the power you asked for) and round the resulting day count up to a whole day. If the metric has meaningful weekday/weekend variation, round further up to a whole number of weeks so every arm sees the same mix of weekdays and weekends; that decision is a separate seasonality question with its own reasoning (see the seasonality-planning answer), not something the raw formula above accounts for.

Assumptions and rounding choices worth stating out loud

• Two-sided test with a single, pre-planned final look; no interim peeking.
• Users are independent and contribute one conversion event each; no repeated exposure double counting.
• All intermediate values kept to about five significant figures before rounding the final answer, and the final day count always rounds up, not to the nearest day.
• A buffer of roughly 10-20% is a common practical addition on top of the raw day count to absorb data loss (bot filtering, QA holds, instrumentation gaps); state it as a buffer, not as part of the statistical requirement.
• If daily traffic is volatile rather than a single stable number, use a conservative (lower) daily estimate for the duration calculation rather than the average, since under-running the test is far more costly than over-running it by a day or two.

Worked example

Inputs: 60,000 daily unique eligible visitors, 50/50 allocation, baseline conversion $p_1 = 3.0\%$, desired minimum detectable effect of 8% relative, $\alpha=0.05$ two-sided ($z_{1-\alpha/2}=1.9600$), power 80% ($z_{1-\beta}=0.8416$).

Absolute MDE: $\Delta = 0.03 \times 0.08 = 0.0024$, so $p_2 = 0.0324$, $\bar p = 0.0312$.

$z_{1-\alpha/2}\sqrt{2\bar p(1-\bar p)} = 1.9600 \times \sqrt{2 \times 0.0312 \times 0.9688} = 1.9600 \times 0.2459 = 0.4819$

$z_{1-\beta}\sqrt{p_1(1-p_1)+p_2(1-p_2)} = 0.8416 \times \sqrt{0.03\times0.97 + 0.0324\times0.9676} = 0.8416 \times 0.2459 = 0.2069$

$n = \frac{(0.4819+0.2069)^2}{(0.0024)^2} = \frac{0.4744}{0.00000576} \approx 82{,}376 \text{ users per variant}$

Daily per variant: $60{,}000 \times 0.5 = 30{,}000$.

$\text{days} = \frac{82{,}376}{30{,}000} \approx 2.75 \rightarrow \text{round up to } 3 \text{ days minimum}$

Because this crosses a weekday/weekend boundary either way, the practical duration recommendation would be at least 7 days (one full week) rather than the bare 3-day statistical minimum, so weekday and weekend behavior are represented in the same proportion in both arms.

Trade-offs & pitfalls

• MDE sensitivity. The required sample size scales with the inverse square of the MDE, so halving the effect you want to detect roughly quadruples the required sample and the resulting duration; a stakeholder asking for a smaller MDE "just to be safe" is asking for a much longer test, not a marginally longer one.
• Binary-outcome formula does not transfer to continuous metrics. Revenue-per-user or time-on-task outcomes use the outcome's variance, not $p(1-p)$, in the same general formula shape; plugging a conversion-rate formula into a continuous metric silently understates or overstates the required sample.
• Shortcutting the calendar rounding. A raw day count under 7 does not mean the test is safe to run for that literal number of days; the weekly-cycle rounding matters as much as the raw arithmetic and is a common place teams cut a real corner under launch pressure.
• Traffic volatility. A single "daily uniques" number hides day-to-day swings; a duration plan built on a lucky high-traffic day will run short in practice.

Definition (brief)
Satisficing is choosing a solution that meets predefined acceptable criteria (good enough) rather than exhaustively seeking the theoretical optimum. In research prioritization it means allocating effort to approaches that achieve sufficient scientific or business value within constraints (time, compute, risk), instead of pursuing the marginal gains of an optimal but costly path.

When I prefer satisficing

• Under tight time-to-value or deliverable deadlines.
• When uncertainty or diminishing returns make further optimization costly relative to benefit.
• When the goal is to triage many ideas quickly to find promising directions for deeper work.

Two corporate research-lab scenarios

1. Productization baseline: I need a robust, explainable model to ship an internal prototype for partner evaluation in 6 weeks. I select a well-understood architecture and tuned hyperparameters that meet accuracy and latency thresholds rather than exploring new architectures that might squeeze out 1–2% accuracy but risk delay.

2. Broad exploration phase: Given 20 novel algorithmic ideas, I run low-cost, small-scale experiments to filter the top 3 candidates. I accept “good-enough” results to prioritize follow-up research; exhaustive optimization on every idea would waste compute and slow discovery.

These choices balance scientific rigor with practical impact and enable focused investment where the payoff is highest.

Direct answer

The core review comment is that a Python-level per-row loop over a table is both slower and more error-prone than a vectorized operation (one that operates on a whole column at once instead of cell by cell), and this specific diff also has a real correctness bug: dividing by a zero-span constant column, plus a dtype issue I reproduced directly. I'd ask for the loop to be replaced with a vectorized min-max scale and a test covering the constant-column edge case.

Structured elaboration

Performance

Iterating with row-by-row indexing inside a Python for loop touches the table one cell at a time through the library's indexing machinery on every iteration, instead of letting the underlying vectorized engine operate on the whole column as a single operation. The fix is a vectorized subtract-and-divide across the columns at once, computed as one operation on the underlying array rather than one Python-level operation per row.

Readability

The loop version buries the actual formula (min-max scaling) inside indexing boilerplate; a reviewer has to mentally simulate the loop to recover what the code is doing. The vectorized version reads as the formula itself, which is what a reviewer, and a future maintainer, actually wants to verify is correct.

Edge cases

• Constant column (max equals min): the formula divides by zero. I verified this concretely: an unguarded vectorized version on a constant column produces a missing-value marker for every entry rather than raising, which is silent and easy to miss. This needs an explicit decision (skip the column, fill with a default, or raise a clear error) rather than an accidental one.
• Integer-typed columns: I actually ran the loop version against integer-typed columns and it raised a type error, because assigning a fractional result back into an integer column cell by cell isn't allowed. This is a real, reproducible bug in the diff as written, not a hypothetical: any caller passing an integer-typed feature column hits it immediately. The vectorized version needs the same fix: cast the columns to a floating-point type before scaling.
• Missing values already present: neither version currently guards against a missing value in the input propagating through the min, max, and arithmetic; worth a comment asking whether that's intended.

Concrete suggestion

import numpy as np

def normalize_features(df, columns):
    df = df.copy()
    df[columns] = df[columns].astype(float)
    col_min = df[columns].min()
    col_max = df[columns].max()
    span = (col_max - col_min).replace(0, np.nan)  # avoid divide-by-zero; NaN signals "cannot scale a constant column"
    df[columns] = (df[columns] - col_min) / span
    return df

Test suggestions

• A normal case with known min/max per column, asserting the scaled values match hand-computed expected values
• A constant column, asserting it comes out as a missing-value marker (or whatever the team agrees on) rather than raising or silently returning garbage
• An integer-typed input column, asserting it doesn't raise, which would have caught the dtype bug above

Worked example

I ran both the original loop version and the suggested vectorized version against the same float-typed sample data to confirm they produce identical results, and separately confirmed the two edge cases above.

import pandas as pd
import numpy as np

def normalize_features_loop(df, columns):
    df = df.copy()
    for col in columns:
        col_min = df[col].min()
        col_max = df[col].max()
        for i in range(len(df)):
            val = df.iloc[i][col]
            df.iloc[i, df.columns.get_loc(col)] = (val - col_min) / (col_max - col_min)
    return df

def normalize_features_vectorized(df, columns):
    df = df.copy()
    col_min = df[columns].min()
    col_max = df[columns].max()
    span = (col_max - col_min).replace(0, np.nan)
    df[columns] = (df[columns] - col_min) / span
    return df

df = pd.DataFrame({
    'feature_a': [10, 20, 30, 40],
    'feature_c': [1, 2, 3, 100],
}).astype(float)

loop_result = normalize_features_loop(df, ['feature_a', 'feature_c'])
vec_result = normalize_features_vectorized(df, ['feature_a', 'feature_c'])
print(np.allclose(loop_result[['feature_a', 'feature_c']], vec_result[['feature_a', 'feature_c']]))
print(vec_result)

Actual output:

True
   feature_a  feature_c
0   0.000000   0.000000
1   0.333333   0.010101
2   0.666667   0.020202
3   1.000000   1.000000

Both versions agree exactly on this data. I also confirmed the dtype pitfall is real: calling the loop version on the original integer-typed columns, without the cast to float first, raises a type error naming the specific fractional value it couldn't place into an integer column, on the very first row where the scaled value isn't a whole number.

Trade-offs and pitfalls

• A vectorized fix that skips the dtype cast just moves the same bug from "raises inside a loop" to "raises, or silently truncates depending on the library version, on the whole-column assignment"; the fix has to address the dtype, not only the loop
• Silently turning a constant column into a missing-value marker is defensible but must be a decision the team agrees on, since a downstream model silently receiving missing features can fail in a much more confusing way, far from this code
• Vectorization sometimes trades a small amount of readability for a large amount of speed on genuinely large data, but here the vectorized version is also more readable, so there's no real trade-off to accept in this specific case

Direct answer

The tight worst-case bound here is $\Theta(n)$ time and $O(1)$ auxiliary space, not the $\Theta(n \log n)$ that the doubling inner loop might suggest at first glance: for a fixed outer value i, the inner loop runs only about $\log_2(n/i)$ times, and summing that quantity over all i turns out to telescope to a linear total, not a linearithmic one, once carried through carefully. If the outer loop body itself did $O(n)$ work instead of $O(1)$, the total becomes $O(n^2)$, since that O(n) cost is now paid once per outer iteration, n times, dominating the inner loop's own (still linear) total cost.

Structured elaboration

Step 1: count inner-loop executions for a fixed i

Starting from j = i, each inner iteration replaces j with 2j + 1. In closed form, after m iterations, $j_m = 2^m(i+1) - 1$. The loop stops as soon as j >= n, so the number of executions $t(i)$ for a given i is the smallest m with $2^m(i+1) - 1 \ge n$, which gives
$t(i) = \left\lceil \log_2\!\left(\frac{n+1}{i+1}\right) \right\rceil$

Step 2: sum across all outer iterations

$T(n) = \sum_{i=0}^{n-1} t(i) = \Theta\!\left(n + \sum_{i=0}^{n-1} \log_2\!\left(\frac{n}{i+1}\right)\right)$
(the added $n$ term accounts for the ceiling and the outer loop's own O(1) per-iteration bookkeeping). The sum splits cleanly:
$\sum_{i=0}^{n-1} \log_2\!\left(\frac{n}{i+1}\right) = \sum_{k=1}^{n} \left(\log_2 n - \log_2 k\right) = n\log_2 n - \log_2(n!)$

Step 3: this is where the naive intuition goes wrong, and Stirling's approximation resolves it

It is tempting to stop at "a sum of n logarithmic terms is $\Theta(n \log n)$" without simplifying $\log_2(n!)$ further, but $\log_2(n!)$ is itself $\Theta(n \log n)$, and the two $n \log_2 n$ terms above very nearly cancel. Using Stirling's approximation in natural-log form,
$\ln(n!) = n\ln n - n + O(\ln n)$
and converting to base 2 ($\log_2 x = \ln x / \ln 2$):
$\log_2(n!) = n\log_2 n - \frac{n}{\ln 2} + O(\log n)$
Substituting back:
$n\log_2 n - \log_2(n!) = n\log_2 n - \left(n\log_2 n - \frac{n}{\ln 2} + O(\log n)\right) = \frac{n}{\ln 2} + O(\log n) = \Theta(n)$
so the total is
$T(n) = \Theta(n)$
Intuitively: the inner loop runs many times only for the small handful of i near the very start (i close to 0 needs close to $\log_2 n$ iterations), and that count drops off so quickly as i grows that the sum across all i stays linear in n rather than growing to n times the average log factor.

Step 4: verifying the derivation against a direct operation count

Since this is a derived claim about growth rate, it is worth checking numerically before trusting it, by literally counting how many times the inner loop body executes:

import math


def count_inner_iterations(n: int) -> int:
    """
    Counts total O(1)-work executions of the inner while loop across all
    outer iterations, for direct comparison against the analytic bound.
    This counts operations, not wall-clock time.
    """
    total = 0
    for i in range(n):
        j = i
        while j < n:
            total += 1
            j = j * 2 + 1
    return total


if __name__ == "__main__":
    for n in [1_000, 10_000, 100_000, 1_000_000]:
        counted = count_inner_iterations(n)
        predicted = n * math.log2(n) if n > 1 else 0
        ratio = counted / predicted if predicted else float("nan")
        print(f"n={n:>8}  counted={counted:>9}  n*log2(n)={predicted:>12.1f}  ratio={ratio:.3f}")

Running this prints:

n=    1000  counted=     1994  n*log2(n)=      9965.8  ratio=0.200
n=   10000  counted=    19995  n*log2(n)=    132877.1  ratio=0.150
n=  100000  counted=   199994  n*log2(n)=   1660964.0  ratio=0.120
n= 1000000  counted=  1999993  n*log2(n)=  19931568.6  ratio=0.100

The counted total divided by n converges to almost exactly 2 as n grows (1.994, 1.9995, 1.99994, 1.999993), while the counted total divided by $n\log_2 n$ keeps shrinking toward 0 rather than settling at a constant. A quantity that is truly $\Theta(n \log n)$ would have a roughly constant ratio against $n \log_2 n$; a quantity that is truly $\Theta(n)$ has a ratio against $n \log_2 n$ that shrinks toward 0 as n grows, which is exactly the pattern above, confirming the $\Theta(n)$ derivation (the limiting ratio against n itself, about 2, is consistent with $1/\ln 2 \approx 1.44$ plus the O(1) per-outer-iteration bookkeeping folded in).

Complexity

Time: $\Theta(n)$, tight (both upper and lower bound, not just an upper bound). Space: $O(1)$ auxiliary, since only i and j are tracked regardless of n.

Edge cases

• n=0: the outer loop body never runs, so the total work is trivially $\Theta(1)$ (or 0, depending on how the base case is counted), consistent with the formula's leading term.
• i=0 is the single most expensive outer iteration, taking close to $\log_2 n$ steps; i near n-1 costs only 1 step (j starts already close to n).
• If the doubling step were instead j = j * 2 (without the +1), the same derivation applies with a one-off adjustment to the closed form for $j_m$, and the asymptotic result is unchanged.

Trade-offs & pitfalls

The single biggest pitfall on this exact problem is stopping the derivation one step early: summing n terms that are each individually $O(\log n)$ and concluding $O(n \log n)$ overall, without carrying through what $\sum \log_2(n/i)$ actually simplifies to via Stirling's approximation. That intuition is wrong here specifically because the terms in the sum shrink rapidly (as $\log_2(n/i)$ for growing i), rather than staying near their largest value the way they would if the inner loop's iteration count did not depend on i at all. This is exactly the kind of derivation the reproducibility standard requires showing step by step, and confirming numerically, rather than asserting from a memorized shape ("doubling inside a loop looks like $\log n$, so nested with an outer loop must be $n \log n$") that does not actually hold once the per-i cost is summed out. If the outer loop body itself does $O(n)$ work in addition to the inner while loop, that new cost is paid once per outer iteration regardless of the inner loop's behavior, adding $n \times O(n) = O(n^2)$ to the total, which now dominates the inner loop's own $\Theta(n)$ contribution; the overall complexity becomes $O(n^2)$.

Direct answer
Few-shot and meta-learning approaches are built specifically for the "only a handful of examples per new category" problem, learning a general adaptation MECHANISM ahead of time so a genuinely new category can be handled from just a few examples, without the retraining cycle transfer learning or aggressive augmentation would otherwise require.

Structured elaboration
Prototypical networks: learn an embedding space where each class is represented by the MEAN (prototype) of its few labeled support examples; classifying a new example is simply finding its nearest prototype. This is simple, fast at inference (a nearest-neighbor lookup once embeddings are computed), and a strong default for genuinely frequent, fast-changing category sets.
MAML: learns a set of INITIAL parameters specifically chosen so that a FEW gradient steps of fine-tuning on a new task's few examples reaches good performance; more flexible than prototypical networks (it can adapt the whole model, not just a distance computation), but requires actually running those adaptation steps at serving time, adding latency prototypical networks avoid.
Training regime, EPISODIC training: both approaches are trained by repeatedly simulating the few-shot scenario itself during training (sampling small "episodes" of N classes with K support examples each), so the model is directly optimized for the ADAPTATION task it will actually face in production, not just for ordinary classification.

Deployment trade-offs versus alternatives: transfer learning (fine-tuning a classifier head per new category) is simple but couples adding a new category to a RETRAINING cycle, which is exactly the wrong shape for categories that change frequently, and risks catastrophic forgetting of earlier categories unless carefully managed. Aggressive data augmentation can stretch a FEW real examples further, but it has a hard ceiling for a genuinely NOVEL category, since synthetic transforms of a handful of real examples cannot manufacture true semantic variety the category doesn't already exhibit in its few real examples.

Worked example
A concrete, practical hybrid architecture: a strong PRETRAINED encoder (from ordinary transfer learning, giving good general features) feeding into a PROTOTYPICAL classification head on top; adding a genuinely new category becomes as cheap as computing and storing a new prototype vector from its few examples, no retraining of the encoder at all, while still benefiting from the encoder's strong pretrained representations. For production serving, precomputed prototype (and support) embeddings are stored in a vector index (FAISS, HNSW-based), giving sub-millisecond nearest-neighbor classification even as the category set churns.

Trade-offs & pitfalls
A common mistake is choosing MAML by default for its greater flexibility without weighing its real serving-time cost; running even a few gradient-descent steps per NEW category at inference time is a genuinely different latency profile than prototypical networks' single embedding lookup, and for a product surface with strict latency requirements, that difference can rule MAML out regardless of its accuracy advantage in an offline benchmark. A second common gap is assuming meta-learning eliminates the need for good REPRESENTATIONS entirely; meta-learning's benefit is specifically the fast-ADAPTATION mechanism, and it still performs meaningfully better when built on top of a strong, well-pretrained encoder rather than one trained from scratch on limited data.

Use STAR-like minimal practices list with one implementation step each:

1. Fairness: practice — group-level metric monitoring. Implementation: compute and alert on parity metrics (e.g., equalized odds, demographic parity) per sensitive group each deployment; block deploy if gap > threshold.
2. Privacy: practice — data minimization and access control. Implementation: enforce role-based access, store only required features, and apply field-level encryption; log and audit accesses.
3. Explainability: practice — model explanations for decisions. Implementation: integrate SHAP or counterfactuals into prediction logs and provide human-readable reasoning for flagged cases.
4. Robustness & testing: practice — pre-deployment tests for bias and performance across slices. Implementation: add CI tests that run fairness, distribution-shift, and residual analyses on holdout slices.
5. Governance: practice — decision record and signoff. Implementation: require a responsible-AI checklist and stakeholder signoff (legal/ethics/product) before production rollout.
   Each practice ties to operational controls (metrics, automation, audits) so engineering teams can embed checks into pipelines.

Assumptions

• Two-sided hypothesis, alpha = 0.05, power = 0.8.
• Observations (accuracy across seeds) are independent and approx. normal.
• Observed SD = 4% (σ = 0.04). The target absolute difference Δ = 2% (0.02).
• Clarify design: independent runs per method (unpaired) vs. paired (same seeds for both methods).

Calculation (approximate, normal-approximation)

• Z_{1-α/2} = 1.96, Z_{1-β} = 0.84, so (Zsum)^2 = (1.96+0.84)^2 = 7.84.

If unpaired (equal n per group):
n_per_group = 2 * (Zsum)^2 * σ^2 / Δ^2
= 2 * 7.84 * (0.04^2) / (0.02^2)
= 2 * 7.84 * 0.0016 / 0.0004
= 62.72 ≈ 63 runs per group (126 total).

If paired (same seeds, measuring difference):
n = (Zsum)^2 * σ_diff^2 / Δ^2

• If differences have same SD ≈ 0.04, n ≈ 31.4 ⇒ ≈ 32 paired runs.

Practical checks and robustness

• Run a pilot (20–30 seeds) to estimate SD of differences (for paired) or per-group SD.
• Check normality of accuracies/differences (QQ plot); if heavy tails, use t-based or nonparametric/bootstrapping.
• Verify independence (no shared randomness besides seed) and homoscedasticity; if variances differ, use Welch correction.
• Use permutation tests or bootstrap CIs as robustness checks.
• Pre-register analysis, avoid peeking; if sequential testing needed, adjust alpha (e.g., alpha-spending).

Direct answer

I use a small, weighted numeric rubric, novelty, product impact, feasibility, and team skill fit, scored independently by more than one person and calibrated against how past projects actually turned out. I treat the score as a structured input to a resourcing conversation, not as an automatic ranking: two ideas half a point apart on a five-point scale are a coin flip, not a verdict.

Structured elaboration

Dimensions and illustrative starting weights (tune per organization; a research-heavy lab should weight novelty higher than a product-focused team would):

Score independently, then reconcile, do not average blindly. Have two or three people score each idea before comparing notes. A wide disagreement on feasibility usually means someone has information the others do not, for instance an engineer who knows a platform gap the researcher does not, and that should surface in a short discussion rather than get smoothed out by averaging.

Calibrate against outcomes, not just intuition. Backtest the rubric against ten to twenty completed projects, scoring them retroactively from the original proposal, and compare the predicted ranking to what actually happened (shipped, published, stalled). If high-novelty, low-impact projects consistently underperformed the product roadmap, raise the impact weight; if high-feasibility projects still stalled, the feasibility criteria were measuring the wrong thing.

Fold in the ongoing cost the idea will create, separately from its value. Research code's long-term maintainability cost is real but hard to score directly, so use a cheap proxy: code churn, meaning how much of the prototype's code is still being rewritten week over week. High churn either means the idea is still genuinely unstable, which is normal early on, or that the eventual production version will require heavy ongoing rework, which should pull the feasibility score down rather than being ignored as noise.

Treat cross-functional effort as a range with its assumption attached, not a single number. When an idea needs another team's engineering time, for example a data pipeline team or an infrastructure team, that team's own effort estimate has to feed the feasibility score, not just the research team's estimate of its own work. State it as a range tied to its source of uncertainty, such as "this depends on whether the data team has capacity for a new pipeline next quarter," rather than compressing it into a single confident number. Presenting a cross-functional estimate as false precision is the most common way a feasibility score turns out wrong later.

Worked example

Two ideas competing for the same slice of the team's time:

• Idea A, a new self-supervised pretraining objective: novelty 5, product impact 2 (no clear shipped feature yet), feasibility 4 (the team has done self-supervised work before), skill fit 5.
• Idea B, fine-tuning an existing model on a new internal dataset to improve support-ticket triage accuracy: novelty 2, product impact 5 (ties directly to a metric the support org already tracks), feasibility 3 (needs a new data pipeline owned by another team), skill fit 3.

Using the illustrative weights above:

Idea A: (5 x 0.30) + (2 x 0.35) + (4 x 0.20) + (5 x 0.15) = 1.50 + 0.70 + 0.80 + 0.75 = 3.75 out of 5.

Idea B: (2 x 0.30) + (5 x 0.35) + (3 x 0.20) + (3 x 0.15) = 0.60 + 1.75 + 0.60 + 0.45 = 3.40 out of 5.

Idea A scores a little higher, driven by novelty, feasibility, and skill fit rather than product impact. A 0.35-point gap on a five-point subjective scale is not decisive, so instead of treating this as "fund A, kill B," I use it to structure the actual conversation: Idea B's feasibility score is depressed by a dependency on another team, so before deciding, I would get that team's own effort range rather than assume the worst. If both hold up, they likely draw from different budget lines anyway, research time for A and roadmap capacity for B, so the rubric's job here is to make the trade-off visible and arguable, not to auto-select a winner.

Trade-offs and pitfalls

• Treating the rubric as more precise than it is. A gap of a few tenths of a point on subjective five-point scales is noise; use it to structure discussion, not to auto-rank.
• Letting the weights become a silent value judgment. A team that always weights novelty low will drift into doing only product work and stop doing research without ever deciding to; revisit the weights explicitly on a schedule instead of letting whichever stakeholders are in the room set them by default.
• Using code churn as a hard scoring input rather than a flag. Early-stage prototypes churn heavily for good reasons; treat high churn as a prompt to ask why, not as an automatic feasibility penalty.
• Rounding a cross-functional effort estimate into a single confident number. Carrying the uncertainty and its source through to the feasibility score is what keeps the score honest when the dependency turns out to be wrong.

Direct answer

"Make the model more robust" is not an instruction, it is a symptom report with the details stripped out. Before running any experiments, ask clarifying questions to find out what triggered the request and which specific failure the reviewer actually observed, then weigh that against the realistic set of robustness dimensions (data drift, adversarial inputs, missing or malformed data, rare segments, and operational load) so the experiments you prioritize target the real problem instead of a generic notion of "robust."

Structured elaboration

Clarify the trigger first. Ask who raised the concern and why: was there a production incident, a new customer segment behaving oddly, a security review, or a demo failure? Ask what specifically went wrong: did the model return a confidently wrong prediction, crash on an input, or degrade slowly over time? The answer to "why now" usually points straight at the relevant dimension.

Know the menu of robustness dimensions before you guess. A model can be fragile in several genuinely different ways, and a fix for one does almost nothing for another:

• Distribution shift or data drift: the input distribution in production has moved away from training data over time.
• Adversarial robustness: the model is deliberately fooled by crafted inputs, relevant mainly when the model faces an adversary (fraud, spam, security).
• Missing, noisy, or malformed inputs: upstream data quality issues, nulls, wrong types, or schema changes reaching the model unfiltered.
• Rare segments and class imbalance: the model works well on average but fails on a small but important subgroup.
• Out-of-distribution inputs: entirely novel inputs the model was never trained to handle, ideally detected rather than silently scored.
• Operational robustness: latency spikes or failures under load, which is an engineering concern more than a modeling one but often gets bundled into "robustness" complaints.

Prioritize experiments against the actual signal, not the whole list. Once you know which dimension the feedback is really about, prioritize experiments that address it directly (for example, input validation and imputation for a data-quality issue, or a drift-monitoring and retraining cadence for distribution shift) before considering the more expensive or unrelated options like adversarial training.

Worked example

A product manager tells the team "the fraud model needs to be more robust" after a customer escalation. Clarifying questions reveal the trigger: a support ticket where a legitimate transaction was flagged, and the customer's payment method was new to the platform, unlike anything represented much in training data. That points to distribution shift and rare-segment weakness, not an adversarial attack and not a general engineering robustness issue. The prioritized experiments become: check how prediction confidence changes for accounts with a new payment method, evaluate performance specifically on that segment rather than only the overall metric, and consider adding monitoring that flags when the share of "new payment method" transactions grows faster than the training data represents. Broad adversarial-training work is explicitly deprioritized because nothing in the clarified feedback points to an adversary crafting inputs.

Trade-offs and pitfalls

Treating "make it more robust" as a blank check to run every robustness technique available (adversarial training, heavy data augmentation, ensembling) wastes effort on dimensions nobody actually reported a problem with, and it delays fixing the real one. Guessing at the dimension without asking the clarifying question first risks solving an imagined problem instead of the reported one. It is also a mistake to treat "robustness" as purely a data-science concern when the actual complaint is operational, such as latency under load; in that case, the right answer routes to an engineering fix, not a modeling experiment.

Direct answer

Keep a single, easy-to-find decision log tied directly to the work it affects: what was decided, the options considered, the reasoning, and who owns it, updated by whoever is making the decision at the moment it is made, not reconstructed later from memory.

Structured elaboration

What belongs in an entry

A short, consistent structure works better than a long one, because people will actually fill it out: a title, the date, who owns it, the context in one or two sentences, the options considered with their trade-offs, the decision itself, and the reasoning behind it in a few bullet points.

Where it lives

The log needs to be one discoverable place, linked from the tickets, docs, or roadmap items it affects, not scattered across meeting notes and chat threads. A shared doc or wiki page with a simple table works; the tool matters less than the discipline of always linking to it.

Who keeps it current

The person who owns the decision, not a rotating scribe with no stake in it, writes or finalizes the entry, ideally right after the decision is made, while the reasoning is still fresh and easy to state accurately.

How it gets used afterward

In retrospectives, revisit decisions that affected the outcome and check whether the original assumptions held. For onboarding, a short list of the most consequential recent decisions gives a new team member the context that would otherwise take weeks of osmosis to pick up.

Worked example

A team is deciding between two ways to notify users of an event: a push notification versus an in-app banner. The entry, once decided, looks like this: title, "Notification channel for event alerts"; date and owner, the decision owner's name and the date; context, users were missing time-sensitive alerts under the current in-app-only approach; options considered, push notification (faster delivery, requires a new permission prompt), in-app banner only (no new permission needed, slower to be seen), and both channels (best coverage, more engineering and support surface); decision, push notification with an in-app banner as a fallback for users who decline the permission; reasoning, the delay in the in-app-only approach was the specific problem being solved, and the fallback covers users who opt out.

Anyone who later asks why the team does not just use an in-app banner, since it is simpler, can read this entry and see the trade-off was already considered, rather than re-litigating it from scratch.

Trade-offs and pitfalls

A log nobody updates is worse than no log: it creates false confidence that the reasoning is captured somewhere, while actually going stale. The fix is keeping entries short enough that updating one takes minutes, rather than requiring a formal write-up every time.

A log can also be used as a weapon later, such as insisting a past decision still holds in a situation where circumstances genuinely changed and revisiting was the right call. The log should record reasoning, not lock in a decision forever; a review date or a note on when to re-evaluate keeps it a living reference instead of a trap.