Lyft Applied Scientist (Entry Level) Interview Preparation Guide

Direct answer

A p-value answers one narrow question: if there were truly no effect, how surprising would data like ours be? A small p-value, commonly under 0.05, means the pattern we saw would be unusual if nothing were really going on, so it counts as evidence worth acting on rather than random noise. Caution in one sentence: a p-value does not tell you the probability the effect is real, or how big or important it is, only how surprising the data would look under a "nothing is happening" assumption.

Structured elaboration

Explaining a statistics term to a non-technical stakeholder is less about simplifying the definition and more about choosing what to omit and what to guard against:

1. Pick one mental model and don't mix in a second. "How surprising the data would be if nothing were going on" is enough; don't also try to explain sampling distributions or the formal null hypothesis by name in the same breath.
2. State the caution as its own sentence, not a footnote. The most common misread is treating the p-value as "the probability we're right." Say plainly what it is not.
3. Separate statistical significance from size. A p-value can be tiny on an effect too small to matter, or the reverse: with a small sample, a real and sizeable effect can fail to look significant. That is where a confidence interval (the range the true effect probably falls in, not just whether it beat a cutoff) and margin of error earn their keep: they answer "how big, and how sure," a different question from "was this surprising."
4. Check understanding by asking the stakeholder to restate it in their own words. If they say "so it proves we're right," the caution hasn't landed yet.

The same "how surprising is this" frame extends past hypothesis tests. If a stakeholder asks how confident a model's predicted probabilities are, model calibration answers a related but different question: not "was this one result surprising" but "when the model says 70%, does that outcome actually happen about 70% of the time." And when a CFO worries a result is a fluke of a small sample, the honest reassurance isn't the p-value at all, it's showing the sample size and how much the estimate would tighten with more data, since a small or biased sample can produce a small p-value that still doesn't generalize.

Worked example

An A/B test on a new checkout flow comes back with a borderline p-value of 0.04. In the room:
"We ran the new checkout against the old one with about 20,000 customers on each side. The lift we saw would only happen by chance about 4 times out of 100 if the new flow actually made no difference, so this is likely real, but it's close to our usual cutoff, not a landslide. Before we roll it out everywhere, I'd want to see the confidence interval, the range of lift the data actually supports, because at this sample size a real-but-small effect and a borderline-noise effect can look similar on the p-value alone. If that range includes 'basically no change,' I'd want one more week of data before we call it, rather than lock in a decision off a single borderline number."
Caution restated: a p-value close to the cutoff isn't something you can round away in either direction, it's a signal to look at the interval and the trend, not settle the question on the spot.

Trade-offs and pitfalls

Dropping the caution sentence to keep the pitch upbeat is the most common mistake, and it's exactly what produces "the data proves it" overconfidence later. Overcorrecting the other way, hedging so hard that a genuinely strong result reads as shaky, costs credibility too. Borderline p-values, roughly 0.03 to 0.07, deserve more nuance than either "significant, ship it" or "not significant, ignore it": show the interval and the trend, not just the single number against the 0.05 line.

Plan overview: treat this as a data/pipeline problem: gather evidence, compare distributions and temporal patterns, then run small isolation experiments (shadow/canary/replay).

1. Gather logs & data

• Raw inputs for affected period (examples, request timestamps, client metadata).
• Feature distributions (per-feature values before/after deploy, encoded categories counts).
• Model inputs (post-featurization vectors), feature-importance or SHAP for batches.
• Ground-truth arrival patterns and labeling delays (timestamps when true labels arrive).
• Prediction outputs, confidence scores, calibration stats, and downstream action logs.
• System logs: latency, errors, dropped requests, preprocessing exceptions.

2. Statistical comparisons / tests

• Population Stability Index (PSI) per feature and overall. Flag >0.1/0.25.
• Two-sample KS test or Mann–Whitney for numeric features.
• Chi-squared for categorical frequency shifts.
• Compare prediction score distribution and calibration (reliability diagrams, Brier).
• Compare label-rate and label-delay distributions (time-to-label survival analysis).
• Feature correlation / covariance changes and multivariate drift (MMD).

3. Small experiments / canary isolations

• Shadow/replay: feed historical inputs through new vs. old model offline; compare predictions and AUC on same labeled backlog.
• Traffic split: route small % to new version with logging; monitor live AUC, score distributions, latency.
• Input-ablation: replace individual features with old-model values to locate problematic feature transformations.
• Replay through full pipeline including preprocessing to detect mismatch between training feature pipeline and production transforms.
• Canary on subset of users/regions or identical hardware to rule out infra/latency effects.
• If label delay suspected, wait/stratify by time windows to see performance recovery.

4. Interpretation & next steps

• If drift in inputs/features -> fix data source or retrain with recent data or apply input normalization.
• If preprocessing bug or schema mismatch -> rollback and patch transformation code, add schema checks/unit tests.
• If calibration change -> consider recalibration or threshold retraining.
• Document findings, create automated drift alerts (PSI/KL), add end-to-end integration tests (replay + shadow) to CI.

This systematic evidence-first approach isolates whether cause is data drift, label issues, code changes, or infra, and guides targeted remediation.

Direct answer

Run a short, repeatable inspection pass over any newly loaded DataFrame: dtypes and non-null counts first (df.info()), then descriptive statistics split by numeric versus categorical columns (since describe() reports very different things for each), then percent missing per column, then a sample of unique values for whichever categorical column you actually care about, usually the one you are about to group, filter, or join on.

Implementation

import pandas as pd

def profile_df(df, cat_col=None, n_unique_sample=10):
    # 1) dtypes and non-null counts
    print("Dtypes and non-null counts:")
    print(df.info())

    # 2) descriptive stats for numeric columns
    print("\nNumeric summary:")
    print(df.select_dtypes(include="number").describe().T)

    # 3) descriptive stats for categorical/text/boolean columns
    # include 'string' alongside 'object' so this also catches columns under
    # pandas 3.0's default string dtype, not just the legacy object dtype
    cat = df.select_dtypes(include=["object", "string", "category", "bool"])
    if not cat.empty:
        print("\nCategorical summary (count, unique, top, freq):")
        print(cat.describe().T)
    else:
        print("\nNo categorical columns found.")

    # 4) percent missing per column
    missing_pct = df.isna().mean() * 100
    print("\nPercent missing per column:")
    print(missing_pct.sort_values(ascending=False))

    # 5) sample unique values for a chosen categorical column
    if cat_col is None:
        cat_col = cat.columns[0] if not cat.empty else None
    if cat_col is not None:
        uniques = df[cat_col].dropna().unique()
        print(f"\nSample unique values for '{cat_col}' (up to {n_unique_sample}):")
        print(list(uniques[:n_unique_sample]))
        print(f"Total unique: {len(uniques)}")
    else:
        print("\nNo categorical column to sample uniques from.")

Worked example (verified against pandas 3.0.3, no warnings emitted)

df = pd.DataFrame({
    "id":      [1, 2, 3, 4, 5],
    "price":   [10.5, 20.1, None, 15.0, 9.9],
    "country": ["US", "UK", "US", "CA", None],
    "active":  [True, False, True, True, False],
})
profile_df(df, cat_col="country")

Actual output (trimmed to the key sections):

Numeric summary:
       count    mean       std  min    25%    50%     75%   max
id       5.0   3.000  1.581139  1.0   2.00   3.00   4.000   5.0
price    4.0  13.875  4.733128  9.9  10.35  12.75  16.275  20.1

Categorical summary (count, unique, top, freq):
        count unique   top freq
country     4      3    US    2
active      5      2  True    3

Percent missing per column:
price      20.0
country    20.0
id          0.0
active      0.0

Sample unique values for 'country' (up to 10):
['US', 'UK', 'CA']
Total unique: 3

price correctly shows count=4 (one missing value excluded from the mean/std), country's describe() correctly reports US as the most frequent value with freq=2, and the missing-percent breakdown immediately tells you price and country are the two columns that need a decision (impute, drop, or carry forward as legitimately missing) before modeling or aggregation.

Key points

• df.info() is the fastest way to see dtypes and non-null counts together, catching an accidentally-all-object numeric column or an unexpectedly low non-null count in one glance.
• describe() reports different statistics depending on dtype: mean/std/quartiles for numeric columns, count/unique/top/freq for text, category, and boolean columns, calling it once on the whole DataFrame silently drops whichever family of columns it was not built for unless you explicitly split by select_dtypes first, as this function does.
• df.isna().mean() * 100 is a fast, vectorized way to get percent-missing per column without writing a loop, and sorting it descending immediately surfaces the columns that need attention first.
• Sampling unique values (rather than printing all of them) is deliberate: it is meant to catch typos and inconsistent category labels ("US" vs "U.S." vs "United States") at a glance, not to enumerate a high-cardinality column in full.

Complexity and edge cases

Complexity: each step here is a single O(n) or O(n * columns) pass, info(), describe(), and isna().mean() are all vectorized, so profiling scales linearly with the size of the DataFrame; nunique()/unique() on a very high-cardinality column is still O(n) but with a larger constant cost than the aggregate statistics.
Edge cases: a DataFrame with no numeric columns makes select_dtypes(include="number").describe() return an empty frame, not an error, worth guarding the print with a check if you want a clean message instead of an empty table. A very wide DataFrame (hundreds of columns) makes the printed describe().T output unwieldy, consider .sample(n=30, axis=1) on the columns or splitting the profile into a few DataFrames of columns at a time. A high-cardinality categorical column (a free-text or id-like column) will make cat.describe()'s unique/top/freq less informative (often unique close to len(df)), and printing "a sample of unique values" for it is far more useful than the full describe() summary, which is exactly why the function separates those two concerns.

BETWEEN is inclusive on both ends: amount BETWEEN 50 AND 500 means amount >= 50 AND amount <= 500.

Structured elaboration

The two forms are exactly equivalent:

-- BETWEEN
SELECT transaction_id FROM transactions WHERE amount BETWEEN 50 AND 500;
-- Explicit form
SELECT transaction_id FROM transactions WHERE amount >= 50 AND amount <= 500;

BETWEEN reads more naturally for a closed range and is the standard idiom; the explicit form is sometimes preferred when the range needs to be half-open (e.g., excluding the upper bound), since BETWEEN has no built-in way to express that.

Worked example

With amounts 49.99, 50.00, 300, 500.00, and 500.01: the query returns the middle three rows (50.00, 300, 500.00). 49.99 and 500.01 are correctly excluded, confirming both bounds are inclusive.

Trade-offs and pitfalls

On floating-point columns, a boundary value that "looks like" 500 might actually be stored as 499.9999999 due to representation error, silently falling outside an inclusive BETWEEN. This is one of several reasons DECIMAL/NUMERIC types are preferred over floating point for money. BETWEEN on TIMESTAMP columns has its own, more common trap: a date-only upper bound like occurred_at BETWEEN '2024-03-01' AND '2024-03-31' is interpreted as '2024-03-31 00:00:00', midnight at the very start of that day, so every transaction later that same day (say, 2024-03-31 18:00:00) is strictly greater than the upper bound and gets silently excluded, even though it clearly belongs in "March 2024" by any reasonable reading. The fix is the same half-open pattern used for numeric ranges: express the upper bound as the start of the NEXT period, exclusive, e.g. occurred_at >= '2024-03-01' AND occurred_at < '2024-04-01', which never depends on knowing the column's exact time precision.

Overview & constraints
Goal: safe 6-week pilot migrating one production-critical TF1.x model to PyTorch + TorchServe with zero-regression for latency/accuracy and clear rollback. I’ll split effort into learning, migration, testing, deployment, monitoring, and criteria.

Week-by-week learning & ramp

• Week 1: Hands-on PyTorch basics, TorchScript, TorchServe docs, convertors (ONNX), run simple model end-to-end on dev GPU.
• Week 2: Reproduce model training/inference locally in PyTorch; workshops with infra/serving engineers.

Migration steps

• Export TF1.x inference graph and validate inputs/outputs.
• Reimplement model in PyTorch reproducing pre/post-processing exactly.
• Option A: Convert TF -> ONNX -> PyTorch for weights where possible; preferred: re-load weights manually if architecture differs.
• Instrument deterministic seeds and numerics (float32) for parity.

Compatibility & numerical parity

• Unit test layer-wise output comparisons using identical inputs (batches, edge cases).
• Compare logits, intermediate tensors, and final probabilities; allow tiny tolerance (e.g., L_inf < 1e-5) and document acceptable deltas.
• If differences > tolerance, trace back to ops (e.g., batchnorm epsilon, padding) and align implementations.

Testing strategy

• Unit: layer/function parity, preprocessing/postprocessing, edge inputs.
• Integration: end-to-end local inference vs TF baseline on recorded production traffic (shadow runs).
• System: containerized TorchServe health, scaling, memory, and latency tests.
• Canary: deploy 1% traffic route to PyTorch model with request/response hashing; compare A/B for accuracy, latency, error-rate for 48–72h.

Rollback & safety

• Keep TF1.x serving live; use traffic-splitting. Automated quick rollback to 0% PyTorch if SLA breaches or accuracy regressions detected.
• Circuit breaker: auto-disable PyTorch if p50/p95 latency or error-rate thresholds exceeded.
• Maintain reproducible build artifacts + versioned weights to redeploy TF or PyTorch.

Success criteria (objective)

• Numerical parity: mean absolute difference in key metric < 1e-4 and classification AUC/accuracy delta ≤ 0.1% on held-out live sample.
• Performance: p95 latency within 10% of TF baseline and memory within tolerances.
• Reliability: zero production errors attributable to model for 72h at 1% traffic and no rollback triggered.
• Operational: automated CI parity tests added and run in pipeline.

Risks & mitigations

• Weight conversion errors → prefer manual weight load + layer tests.
• TF1.x ops missing → implement custom ops or use ONNX fallback.
• Time: prioritize inference parity and safety; defer retraining for full PyTorch until after pilot.

Direct answer

A retrain-trigger policy is a small decision function that watches a handful of signals, data drift, a delayed quality or business metric, or a fixed calendar cadence, and fires a retrain only when a signal has been persistently and significantly out of bounds, not the moment it moves. That means the policy needs both a statistical bar (is this deviation large enough to be real) and a persistence bar (has it stayed that way long enough to not be noise) before it triggers anything, plus a floor cadence so the model never goes too stale even if every drift signal stays quiet.

Structured elaboration

Candidate trigger signals:

Debouncing mechanisms, so a signal blip does not fire a retrain:

• Statistical significance or minimum sample size: compare values with a test that accounts for how much data backs the comparison, since a tiny sample naturally swings more; this stops random noise on a low-traffic day from tripping the trigger.
• Consecutive-window requirement (hysteresis): require the signal to breach its threshold across several evaluation windows in a row, not once; this is what actually separates a blip from a trend.
• Cooldown period: once a retrain has fired, suppress the same trigger from firing again for a fixed minimum interval, since a fresh retrain needs time to take effect and be evaluated before deciding whether it worked.
• Magnitude floor: require the deviation to clear a minimum practically meaningful bar, not just a statistical one, since with enough traffic almost any tiny difference becomes statistically significant.

Composing the signals: use the scheduled cadence as a backstop; treat data drift alone as a low-trust signal that tightens monitoring (for example, shortens the evaluation window) rather than triggering a retrain by itself; reserve an automatic retrain trigger for a persistent, significant move in the delayed quality metric or business KPI; and let a severe data-drift signal alone escalate to a human decision rather than an automatic retrain, since drift without a confirmed quality impact might be a real, permanent shift the model should adapt to, or a one-off event a retrain would simply overfit to.

flowchart LR
  S[Drift and quality signals] --> D{Breached for N consecutive windows?}
  D -->|No| S
  D -->|Yes| C{Cooldown active?}
  C -->|Yes| S
  C -->|No| RT[Trigger retrain]
  RT --> V[Offline validation gates]
  V --> RO[Canary rollout]

Worked example

Suppose a trigger monitors weekly PSI on a key feature with a threshold of $0.10$, and daily values across a week, Monday through Sunday, are: $0.06, 0.12, 0.05, 0.11, 0.13, 0.14, 0.15$. A naive policy of "retrain the moment PSI exceeds 0.10" fires on Tuesday alone, a single-day spike that drops back below threshold on Wednesday. Adding a 3-consecutive-day hysteresis rule changes the outcome: Tuesday's breach does not count toward a streak because Wednesday falls back under threshold, breaking it; Thursday begins a new streak (Thursday $0.11$, Friday $0.13$, Saturday $0.14$), and the policy correctly fires on Saturday, the third consecutive day above threshold, treating Tuesday's spike as noise and the Thursday-onward run as a real, persistent shift.

Trade-offs and pitfalls

A too-sensitive policy causes retrain thrashing: burning compute and, worse, repeatedly resetting the model onto small, temporary populations, which can make it chase noise instead of tracking real drift. A too-conservative policy, long consecutive-window requirements or wide magnitude floors, means the model quietly serves stale weights through a real, sustained shift for longer than necessary. A common wrong turn is using data drift alone as the retrain trigger, since a distribution can shift with no effect on accuracy, or shift and matter greatly, and only the delayed quality signal actually tells those two apart. Scheduled retraining as the only mechanism is safe but wasteful when nothing has drifted, or dangerous when a real regression sits live while the policy waits for the calendar, which is why a floor cadence should be combined with, not replaced by, a faster, debounced signal-driven trigger. A triggered retrain should still be auto-kicked-off, not auto-promoted: the resulting model passes through the same offline validation and canary rollout gates as any other new version before it ever reaches full traffic.

Direct answer: Proving an engineered interaction feature actually helps (rather than just nudging an offline metric by chance) needs a proper held-out evaluation design: a genuine out-of-time holdout (not just a random split, since the feature's benefit needs to generalize forward in time, not just across a random shuffle), a significance test on the improvement, and a pre-agreed bar for what lift is actually worth the feature's ongoing maintenance cost.

Structured elaboration:
The design: train/validate with and without the interaction feature using identical everything else (same model, same hyperparameters, same other features), evaluate both on an out-of-time holdout (data from AFTER the training period, mirroring how the model will actually be used), and run a proper statistical test on the difference (not just eyeballing which number is bigger, since a small improvement on a small holdout can easily be noise). Cross-validation across multiple folds (not a single split) gives a distribution of the improvement rather than one point estimate, which is what lets you distinguish "reliably better" from "got lucky on this one split."

Deciding what lift is "meaningful" has to be set BEFORE looking at the result, ideally tied to the feature's actual maintenance cost: a feature that adds a new upstream dependency, a new failure mode, and ongoing monitoring burden needs to clear a higher bar than a nearly-free derived column, since a marginal, statistically-real-but-tiny improvement may not be worth carrying long-term.

Worked example: A composite recency-times-frequency feature shows a 0.3% AUC (area under the curve) improvement on a random train/test split. Evaluated properly on an out-of-time holdout across five rolling folds, the improvement is inconsistent (positive in three folds, negative in two, with a confidence interval overlapping zero), correctly indicating the apparent gain on the single random split was likely noise, not a real, generalizable improvement, and saving the team from committing to the added maintenance cost of a feature that doesn't reliably help.

Trade-offs and pitfalls: The most common mistake is evaluating only on a random split, which can look favorable for a feature that's actually capturing something that won't hold up going forward in time (a spurious correlation specific to the training period); an out-of-time evaluation is the discipline that catches this before the feature ships.

Direct answer

Shadow traffic means forking real production requests to a candidate model that scores them without its response ever reaching the user: it validates real-world behavior with zero user-facing risk, at the cost of not being able to observe how users would actually react to the candidate's different predictions.

Structured elaboration

The core mechanism: the serving layer duplicates each incoming request, sends one copy to the currently-serving model (whose response goes to the user, as normal) and one copy to the shadow candidate (whose response is logged and compared, but discarded rather than served). This gives you genuine production-scale, production-distribution validation: not a sampled offline test set, but literally today's real traffic: without any risk of a bad candidate actually harming a user.

Benefits: catches problems an offline evaluation set can't (a candidate that errors or times out on a specific real-world input pattern your offline set didn't happen to include), at the full scale and true distribution of production traffic, with zero blast radius if something's wrong.

Operational pitfalls: shadow traffic still consumes real compute: running two models against every request roughly doubles serving-side compute cost for the duration of the shadow test, which needs budgeting, not an afterthought. Privacy pitfalls: the candidate model is processing real user data even though its output is discarded: any data-handling or retention policy that applies to a genuinely serving model still applies here (the fact that the response isn't shown to the user doesn't exempt the request from privacy obligations around processing it).

Worked example

For validating a new ranking model: fork live search traffic to the candidate, log its ranked results alongside the currently-serving model's, and compare offline (rank correlation, overlap in top-K results, latency) without ever showing the candidate's ranking to a real searcher: this catches a candidate that, say, returns wildly different (and plausibly worse) top results for a specific query pattern that the offline evaluation set happened not to cover, before any user ever sees it.

Trade-offs & pitfalls

Shadow testing's structural limitation is that it can only validate the model's OUTPUT given real inputs, never how a real user would have RESPONDED to a different output: a ranking model whose shadow-tested results look statistically reasonable can still turn out to change user behavior in a way only a real canary (where users actually see and react to the new ranking) would reveal. Shadow testing is the right FIRST gate specifically because it's cheap-risk, not because it's sufficient on its own.

Direct answer

Name a specific, real scope cut, tie it to an explicit trade-off you weighed rather than "we just didn't have time," and show what happened to the deferred item afterward instead of letting the story end at the cut.

Structured elaboration

What makes a strong example
A genuine judgment call with a real alternative you rejected, not something trivially unimportant, and not something imposed on you with no input.

Structure

1. The constraint that forced the choice.
2. The options you weighed, including what you rejected and why.
3. The decision criteria you used: risk, cost, user impact, reversibility.
4. What happened to the deferred item afterward: backlog, follow-up ticket, revisited later.

Ownership calibration
State plainly whether this was your call, a joint call, or one you influenced but a stakeholder ultimately made. Overclaiming decision authority is one of the most common traps in this question.

Worked example

Constraint: a fixed launch date for a SaaS product facing repeated web-application exploit attempts, with pressure from product to keep the feature cadence and from finance to keep costs predictable.

Options considered:

Decision: deliberately deprioritized full custom runtime protection everywhere, in favor of the phased approach, and explicitly deferred hardening the harder services to a follow-up quarter.

What happened to the deferred item: tracked as a scoped follow-up with a named owner and a target quarter, and revisited once the managed-WAF pilot proved out.

Result: protection was live in about two weeks instead of two to three months, and the deferred hardening work still shipped on its follow-up timeline rather than disappearing.

Trade-offs & pitfalls

• Describing a cut that was actually forced on you with zero input reads as compliance, not judgment.
• Failing to say what happened to the deferred scope afterward; interviewers specifically probe whether it was truly deferred or silently abandoned.
• Overstating unilateral authority on a decision that was actually a joint call with a manager or stakeholder.

Direct answer

An analysis that only shows the significant subgroup results, with no mention of how many subgroups were tried, is a textbook signature of p-hacking or selective reporting. Audit it by reconstructing the full set of comparisons that were actually run (not just the ones reported), re-testing that full set with a multiplicity correction, and re-running the analysis end to end on the raw data. The reanalysis plan should pre-specify a small, justified set of primary comparisons and report the complete picture, significant and non-significant alike, not a curated slice of it.

Structured elaboration

Concrete audit checks

Robust reanalysis plan

1. Reproduce the original result from raw data and code before doing anything else.
2. List every comparison that was actually run, whether or not it appeared in the report.
3. Apply a multiplicity correction (Bonferroni for a strict guarantee, Benjamini-Hochberg for a more powered but still principled one) across that full list, not just the reported subset.
4. Report the complete table: every tested subgroup, its unadjusted and adjusted p-value, its effect size, and its sample size, so nothing is hidden by omission.
5. Clearly separate confirmatory findings (survive correction) from exploratory ones (interesting, but not proven, and worth a dedicated follow-up test).

Worked example

Suppose 12 subgroup comparisons were actually run internally, but the report only surfaces the one with $p=0.031$. Re-testing the full family of 12 with Holm-Bonferroni (step-down, more powerful than flat Bonferroni but still FWER-controlling):

(with $m=12$ and $\alpha=0.05$; computed directly from the Holm step-down formula)

The smallest p-value in the full family, 0.031, needed to clear 0.00417 to survive Holm-Bonferroni; it doesn't. The "significant" subgroup finding that made it into the report does not survive once the other 11 comparisons that were actually run are accounted for. This is exactly the pattern an audit is designed to catch, and it doesn't require re-collecting any data, only re-testing honestly against the full family.

Trade-offs & pitfalls

• The hardest part of this audit is usually not the statistics, it's getting an honest accounting of how many comparisons were actually tried; without that, no correction can be computed correctly, so insist on the code and logs, not just a verbal assurance.
• Correcting for multiplicity can occasionally kill a genuinely real effect along with the false ones; that's the acknowledged cost of FWER or FDR control, not a reason to skip it, but it's worth flagging any borderline case as "worth a dedicated confirmatory follow-up" rather than dismissing it outright.
• Presenting this audit to the original analysts or to leadership needs care: the goal is fixing the process (pre-registration, full reporting), not accusing anyone of misconduct, since selective reporting this way is frequently an unconscious byproduct of exploratory analysis rather than deliberate manipulation.
• A reanalysis that only tightens the statistics but keeps reporting only the "winning" comparisons repeats the original mistake in a more sophisticated wrapper; the fix has to include reporting the full family, not just applying a correction to the subset that was already selected.