Netflix Data Scientist Interview Preparation Guide - Mid Level (2-5 Years)

Direct answer

I break the timebox into a small number of weekly milestones, each ending in a hands-on artifact tested against a real-shaped task, not a toy exercise. Resources are a mix of official docs or a course plus time paired with someone already experienced, progress is tracked as milestone pass or fail rather than hours spent, and "ready to apply it on the job" is gated by an outside check, a reviewer, a stakeholder demo, or a working comparison against the existing approach, rather than my own self-assessment.

Structured elaboration

Setting the timebox and milestones. Split the window into roughly three to four checkpoints. Each checkpoint ends in something concrete: a working script, a completed module, a small pipeline that runs. Checkpoints are closer together early on, when momentum and motivation both matter most, and further apart later once the skill is being applied to something real.

Hands-on exercises and projects. The first exercise is usually a minimal, contained build to confirm the basic mechanics work. The middle exercise applies the technology to a narrow slice of a real problem, not a fabricated toy, since a toy exercise can be completed without ever exposing the parts that would actually trip you up on the job. The final exercise is closer to production shape: it includes a failure case, a review, or a comparison against the thing it's meant to replace.

Resources. Draw from more than one type: official documentation and a structured course or tutorial for the foundational shape, hands-on proof-of-concept work for the applied shape, and time paired with a colleague who already knows the technology for the judgment calls that documentation never covers. Where the field has a recognized certification and the role actually values it, a certification's syllabus can be a useful way to guarantee coverage of the foundational shape, but it supplements the hands-on work rather than replacing it.

Measuring progress. Track milestone completion, pass or fail against a specific checkpoint, rather than hours studied. Keep a short running list of "still shaky on" items flagged for extra attention before the final week, so nothing quietly slips through untested.

Validating readiness for the job. Self-assessment alone is a weak signal, since a person is a poor judge of their own blind spots. The actual gate should be external: a reviewer with more experience in the technology signing off, a working parallel run compared against the existing approach, or a stakeholder reacting to a real demo.

This same six-part skeleton, timebox, milestones, exercises, resources, progress tracking, external validation, adapts to very different situations:

• A tight two-week ramp on a new tool to deliver a prototype: milestones compress to daily, resources narrow to the tool's own quickstart plus rebuilding one existing artifact, and a couple of quick, visible wins get surfaced early to build stakeholder confidence while the harder pieces are still in progress.
• A thirty-day ramp on an advanced skill area like complex query optimization: each week ends with a concrete deliverable applied to a real reporting need and presented back to the people who'll actually use it for validation, not just self-checked.
• A six-month plan spanning several named skill areas at once, say query performance, spreadsheet modeling, dashboard design, and foundational modeling basics, sequences them by dependency and immediate need rather than running them in parallel, and frames milestones around a measurable business outcome, such as a specific report that got measurably faster or a prototype that shipped, rather than hours studied.
• A twelve-month plan spanning several priority technologies across a longer career trajectory treats each quarter as its own timeboxed mini-version of this same plan, re-prioritized based on what the team actually needed that quarter rather than fixed a year in advance.
• Choosing among several candidate technologies before committing runs the first exercise as a lightweight bake-off, the same small representative task built in each candidate, before investing the full timebox in one, and validates readiness not just by personal comfort but by whether the resulting artifact delivered value the team could actually use.
• Upskilling specifically to support a larger initiative, like a system migration, anchors milestones to the migration's own real checkpoints, pairs with someone already fluent for the trickiest exercise, and treats "ready" as being able to take one real piece of the migration through review independently.
• Inheriting a codebase built in an unfamiliar framework sets the concrete readiness bar as being able to make a small, safe, correctly tested change and get it through review without the reviewer needing to first explain the framework's core patterns.
• Evaluating an unfamiliar managed service toward a proof of concept builds the contingency directly into the plan: request quota or access on day one rather than waiting until blocked, and have a documented fallback, a smaller tier, a local emulator, or simply a written finding that it isn't viable without a specific quota, so one stalled request doesn't stall the whole timebox.

Worked example

A concrete four-week plan for a data engineer picking up a new streaming platform to replace an existing batch job:

• Week 1, foundations and access. Request sandbox or dev-cluster access on day one rather than waiting to need it. Read the official quickstart plus one well-regarded tutorial. Exercise: run the canonical windowed-aggregation example against a small synthetic stream. Milestone: explain, without notes, what a watermark (a marker asserting that no more late-arriving data older than a given point should be expected) and an event-time window (a fixed time span used to group streaming events by when they actually happened, not when they arrived) are and why they matter.
• Week 2, applied to a narrow real slice. Rebuild one narrow piece of the actual batch job, a single aggregation, as a streaming job against a copy of real or replayed production data, pairing for an hour with a colleague who's used the platform before. Milestone: the streaming version's output matches the batch job's output on a fixed historical window, a correctness check, not yet a performance one.
• Week 3, production-shape constraints. Work through failure and restart semantics, checkpointing (periodically saving the job's processing progress so it can resume from that point instead of reprocessing everything after a crash), and backpressure (the system deliberately slowing producers down when consumers can't keep up, to avoid overload). Exercise: deliberately kill the job mid-run and confirm it resumes without losing or duplicating data. Milestone: a clear, documented answer to why the job needs exactly-once versus at-least-once semantics (whether a crash-and-retry can cause an event to be processed more than once, at-least-once, or never more than once, exactly-once) for this specific use case.
• Week 4, validation and go or no-go. Run the new pipeline in shadow alongside the batch job for a fixed period and compare outputs. Write up the findings, including any access or quota issues hit along the way and how they were resolved, and present them to the team as a short internal write-up or talk. The actual readiness gate is a reviewer with more experience on the platform signing off on the migration plan, not personal comfort alone.

Progress is tracked week to week as milestone pass or fail, with a running note of anything still shaky, rather than a vague sense of "going well."

Trade-offs and pitfalls

A timebox with no exit criteria just becomes open-ended tinkering. A plan built entirely from passive resources, reading and courses, without a real applied exercise doesn't actually validate readiness, since comprehension and being production-ready are different things. Scoping the exercise too broadly, "rebuild the whole system", burns the timebox without ever reaching validation. The single most common failure is skipping the external check and self-certifying "I'm ready," since self-assessment is exactly the thing a person is worst positioned to judge accurately. And for anything gated by quota or access approval, not requesting it on day one is a quiet, common way a two-week timebox turns into two weeks of waiting.

Direct answer
Put the ask, the essential context, and the next step in the first two sentences, so a busy reader can act on the email even if they only read the opening before deciding whether to reply now or later.

Structured elaboration

1. State the ask as the first sentence, not buried after several paragraphs of context: "I'd like to request temporary access to X" or "Could you approve Y by Thursday?"
2. Give only the essential context, one or two sentences of why this ask exists, not the full backstory. Include it because it makes the ask easier to say yes to quickly, not because it's interesting.
3. State the next step explicitly: what you need them to do, and by when, so they don't have to infer the deadline or the required action.
4. Use the subject line to state the ask, not just the topic: "Approval needed by Thursday: Q3 budget line" tells the reader more than "Budget question."
5. Keep the whole email short. If the request genuinely needs more context, put the essential ask up top and the detail below it, rather than making the reader wade through detail to find the ask.

Worked example
Subject: "Access request: prod DB read access, needed by Wednesday"

Body: "Could you grant me temporary read access to the orders table in prod? I'm investigating a customer-reported data discrepancy (ticket #4821) and need to check actual row values, which I can't do in staging since the issue only reproduces with real production data. Happy to have this access time-boxed to a few days and revoked afterward if that's easier to approve."

The ask (temporary read access) and the deadline context (needed by Wednesday) are in the subject line alone; the body confirms the specific ask, gives the minimum context needed to approve it, and proactively offers a constraint (time-boxed) that makes approval easier.

Trade-offs and pitfalls

• Leading with a long justification before the ask is the single most common failure; a reader has to hold the whole paragraph in their head waiting to find out what you actually want.
• Too little context can also fail: an ask with zero justification can force the reader to ask a clarifying question back, which is slower than including the one sentence of context that would have let them approve it immediately.
• For sensitive or high-stakes asks (a large budget approval, access to something risky), a slightly longer, more carefully justified email is worth the extra length; the "front-load the ask" principle still applies, it just means front-loading a well-justified ask rather than skipping justification entirely.

Direct answer

The merge key has to be a value the retry itself can recompute identically: either the source's natural business key combined with a change sequence or event identifier, or a deterministic hash of the record's stable fields, never something regenerated per attempt like a random identifier or the write's own timestamp. A partial write is handled by landing the batch in a staging area first and promoting it into the target with a single atomic merge or replace operation, so the target either reflects the whole batch or none of it, and a crash mid-write leaves nothing to reconcile beyond simply retrying the same staged batch again.

Structured elaboration

What the merge key needs. It must be derivable from the event itself rather than from the write attempt, so a retry computes the exact same key every time; it must be unique per logical record (a business key, or a business key paired with a change sequence number for records that carry multiple versions); and it must exist before the write happens, so it cannot depend on something the target assigns, like an autoincrementing surrogate key.

Options and their failure modes. An idempotency token or event identifier attached upstream is the cleanest, if the upstream can guarantee it. A composite business key (for example customer identifier plus event time plus event type) works without upstream cooperation but can collide if the business key genuinely repeats. A deterministic hash of stable fields is a fallback when neither is available, with a (small) collision risk to weigh against how much precision the use case needs.

Partial-write safety. Never write row-by-row directly into the target with autocommit; land the batch in staging first, then run one transactional merge that promotes it. If the loader dies between "staged" and "merged," the target is untouched, retrying is just re-running the same staging-to-merge path from scratch. If it dies mid-merge on an engine where merge is itself transactional, the engine's own commit semantics mean the merge either fully lands or fully rolls back, so again nothing is left half-applied.

What actually happens on a partial write. Because the merge key is deterministic, re-running an already-committed batch through the same path is a no-op, the merge matches existing rows and updates them to identical values (or skips them), and reprocessing a batch that never committed is now simply its first successful application. No bespoke cleanup step exists or is needed as long as the key is deterministic and the promote step is atomic.

Worked example

A batch of 500,000 events lands in staging keyed by a deterministic dedupe key (a hash of source table, business key, and source log position). The loader crashes after writing 300,000 of the 500,000 rows to staging:

$$\text{rows staged before crash} = 300{,}000, \qquad \text{batch size} = 500{,}000$$

On retry, staging for this batch is fully rewritten (all 500,000 rows, not appended to the partial 300,000), because the batch is treated as one idempotent unit. One merge statement then runs, matching target.dedupe_key = staging.dedupe_key. If a prior, unrelated attempt had actually gotten all 500,000 rows merged into the target before a different failure occurred elsewhere, this retry's merge simply re-matches and no-ops those rows; if the target had nothing from this batch yet, the merge inserts all 500,000. Either way, the target ends with exactly one row per dedupe key for this batch:

$$\text{final distinct keys in target for this batch} = 500{,}000 \quad \text{(never } 300{,}000+500{,}000=800{,}000\text{)}$$

Trade-offs & pitfalls

A merge key built from something the target assigns can never be idempotent, because the retry doesn't know what value the previous attempt got. A key that omits ordering information (a bare business key with no version or sequence) means the last writer for that key silently wins; if a retry applies an older payload after a newer one has already landed, the row regresses unless the merge's update condition explicitly compares versions and skips out-of-order writes. At large scale, running the merge against the entire staging set every retry gets expensive; pre-deduplicating staging to one row per key (keeping the latest by ingestion time) before the merge keeps the match step cheap. This matters most concretely in a payments context: a duplicate row there means double-counting a charge or a payout, not just a slightly noisy dashboard number, so the idempotency discipline described here is the difference between a rounding error and a real financial discrepancy.

Direct answer

The standard ladder, from fastest to slowest growth, is constant $O(1)$, logarithmic $O(\log n)$, linear $O(n)$, linearithmic $O(n \log n)$, and quadratic $O(n^2)$, where $n$ is the size of the input. Each class matches a signature access pattern: a direct lookup, halving a search space each step, one pass over the data, one pass paired with a logarithmic step (typical of good sorting), and comparing every pair of elements. Reporting only the worst-case bound without checking best- and average-case behavior can hide an algorithm that is usually fast but degrades badly on adversarial or already-sorted input, which is exactly the kind of surprise that shows up as a production incident rather than a benchmark number.

Structured elaboration

The five classes, with a concrete example each

Why best, average, and worst case are separate questions

The same algorithm can sit in different classes depending on which case you're asking about:

• Linear search: best case $O(1)$ (the target is first), worst case $O(n)$ (the target is last or absent).
• Insertion sort: best case $O(n)$ on already-sorted input, worst case $O(n^2)$ on reverse-sorted input.
• Quicksort: average case $O(n \log n)$ with a reasonable pivot strategy, worst case $O(n^2)$ if the pivot choice is adversarial (e.g., always picking the first element on already-sorted input).

Judging whether code is "fast enough" means matching the case that actually describes your expected input, not defaulting to whichever case is easiest to state. A cache-friendly, average-case-only justification is fine for random production traffic; it is not fine if an attacker can choose the input (hash-flooding a hash table into its worst case is a real, exploitable pattern).

Ranking functions by growth

Given a set of functions, you rank them by how they behave as $n \to \infty$, not by their value at any one $n$. For example, rank $\{\log n,\ \sqrt{n},\ n,\ n \log n,\ n^2\}$: substituting $n = 16$ gives $\log_2 16 = 4$, $\sqrt{16} = 4$ (a tie at this specific $n$), $n = 16$, $n \log n = 64$, $n^2 = 256$. The tie at $n=16$ is exactly why you don't rank by a single sample point: as $n$ grows, $\sqrt{n}$ pulls permanently ahead of $\log n$, so the correct asymptotic order is $O(\log n) < O(\sqrt n) < O(n) < O(n \log n) < O(n^2)$.

Big-O versus empirical profiling

Big-O describes an asymptotic trend as input size grows without bound; it deliberately ignores constant factors, lower-order terms, and hardware effects. Profiling measures what actually happens for the input sizes you really see, including those ignored factors (memory allocation, cache behavior, branch prediction, garbage collection). Neither replaces the other: Big-O tells you how a design will scale if the input grows 10x or 100x; profiling tells you where time is actually going today. A change that looks neutral or even negative in Big-O terms (e.g., adding a small constant-time cache) can be the dominant real-world win, and Big-O analysis alone would never surface that.

Why a smaller-Big-O algorithm can still be slower in practice

Big-O hides the constant multiplier in front of the growth term. An $O(n \log n)$ algorithm with heavy per-element overhead (extra allocations, function-call indirection, poor cache locality) can lose to a simple $O(n^2)$ algorithm until $n$ passes some crossover point, because the actual operation count is closer to $c_1 \cdot n \log n$ versus $c_2 \cdot n^2$, and if $c_1 \gg c_2$ the crossover point can be far larger than the $n$ you ever encounter. This matters most on hardware with tight memory bandwidth and small caches (mobile devices, embedded targets): the "better" complexity class only pays off once you actually reach the sizes where the growth term dominates the constant.

Worked example

Comparing raw operation counts at $n = 1{,}000{,}000$: $\log_2 n \approx 19.93$, so $n \log_2 n \approx 1{,}000{,}000 \times 19.93 \approx 19{,}931{,}569$, while $n^2 = 1{,}000{,}000{,}000{,}000$. The linearithmic count is about 20x the linear count, but the quadratic count is a million times the linear count, which is the concrete reason quadratic algorithms are the ones that visibly fall over first as data grows, while linearithmic ones stay usable much further out.

Trade-offs & pitfalls

• Quoting a single Big-O figure without naming which case (best, average, worst) it describes is the most common way this topic goes shallow in an interview; always attach the case.
• Big-O comparisons are only meaningful in the limit; for small, fixed input sizes, the algorithm with the better asymptotic class is not guaranteed to be faster once constants and hardware are accounted for.
• Average-case reasoning assumes something about the input distribution (often "roughly random"); that assumption breaks down for adversarial or attacker-controlled input, where the worst case is the one you actually need to defend against.
• Treating two algorithms in the same Big-O class as interchangeable ignores the constant factor, which is often the entire practical difference between them.

Direct answer
A blameless postmortem, an agile retrospective, a fault-finding root-cause investigation, and a live incident review all look at 'what happened,' but they differ in scope, timing, and intent. A postmortem is a single-incident, after-the-fact analysis focused on system-level causes and prevention. A retrospective is a periodic, team-process review across a sprint or cycle, not tied to one specific failure. A blame-assigning RCA investigates to find individual fault, often for disciplinary or legal reasons. A live incident review happens while the incident is still active and is about coordinating response, not analysis.

Structured elaboration

• Postmortem: triggered by a specific incident, usually within days of it; output is a document with root cause, contributing factors, and owned action items; audience is the team plus stakeholders affected by that specific incident; explicitly blameless in framing.
• Retrospective: triggered by the calendar (end of sprint or cycle), not by a specific failure; covers a broader set of process questions (what went well, what didn't, what should change) across many small things, not one deep causal chain; often lighter-weight and less evidence-heavy than a postmortem.
• Blame-assigning RCA: rare, and appropriate only when there's a genuine question of misconduct, negligence, or a formal compliance or legal obligation to identify an accountable individual, for example a regulator requiring named accountability after a security breach; explicitly distinct from, and should not replace, the internal blameless process, which should run in parallel or afterward.
• Live incident review: happens during the incident itself, focused on 'what do we do right now' (mitigation, escalation, communication), not on root cause; a postmortem follows once the incident is resolved and uses this review's timeline as raw material.

When to use each: run a postmortem after any incident above your severity threshold; run retrospectives on a fixed cadence regardless of incidents; reach for a blame-assigning RCA only under genuine legal, regulatory, or integrity concerns, and keep it structurally separate from the team's learning process; the live review is not optional, it's what's actually happening during the incident and simply precedes the postmortem.

Worked example
A payments outage happens on a Tuesday. During the outage (live incident review): the on-call engineer coordinates mitigation, escalates to a second responder, and posts status updates, no root-cause discussion yet. Two days later (postmortem): the team reconstructs the timeline, finds the root cause was a missing input validation check, and assigns an action item. At the end of the sprint (retrospective): the team separately discusses that on-call load has been unusually high this cycle and agrees to rebalance the rotation, a process observation unrelated to any single incident. If it later emerges the outage exposed customer payment data, a formal, blame-assigning investigation may run in parallel, focused narrowly on whether any individual violated policy, kept separate from the blameless technical postmortem which still runs to find the systemic fix.

Trade-offs and pitfalls
A common mistake is collapsing the postmortem into the retrospective (only discussing incidents once a sprint, long after memory and urgency have faded) or collapsing it into the live review (treating the in-the-moment coordination notes as if they were the finished causal analysis, when they usually aren't).

Direct answer
Represent the search space as a tree: the top-level choice is the model family, and each branch carries only that family's own hyperparameters; the search strategy needs to be one that natively supports this conditional structure (a tree-structured Parzen estimator, or a hierarchical Bayesian optimization variant) rather than a flat search that would otherwise waste trials sampling irrelevant hyperparameters for families that weren't even chosen.

Structured elaboration
A flat random or grid search over the UNION of all families' hyperparameters (treating "model family" as just another categorical dimension) wastes a large fraction of every trial: for any given trial, only the hyperparameters belonging to the CHOSEN family actually matter, the rest are simply ignored, meaning your effective sample size per family is much smaller than your total trial count suggests. A hierarchical/conditional search strategy instead samples the top-level choice first (which family), then samples only from that family's own relevant hyperparameter distribution, so every trial's information is fully "used" rather than partially wasted.
Tree-structured Parzen Estimators handle this natively, since they build (implicitly) a tree-shaped generative model of the search space and can represent "hyperparameter B only exists conditional on hyperparameter A = value X" directly. A Gaussian-Process-based Bayesian optimizer, by contrast, doesn't naturally represent this kind of conditional structure and needs either a specialized kernel or a completely separate GP per branch, one surrogate per model family, sharing only the cross-family comparison of best-found scores.

Worked example
Tuning a pipeline that chooses between logistic regression, random forest, and a small neural network: logistic regression's own hyperparameters are just C and the penalty type; random forest's are n_estimators, max_depth, min_samples_leaf; the neural network's are learning_rate, hidden_dim, dropout. A hierarchical search first samples which family to try (perhaps still guided by which family has looked most promising so far), then samples only that family's relevant hyperparameters, so a trial exploring the neural network branch never wastes any of its "information" on an irrelevant random-forest max_depth value.

Trade-offs & pitfalls
Running effectively separate surrogates per branch (in the GP case) means each family gets less cross-informed data than a single unified model would in a non-conditional space; with a very limited total trial budget split across 3+ model families, each family may end up with too few trials to tune its own hyperparameters well, a real cost of pipeline-level search that a single-model search doesn't face.

Direct answer

Missing values are not one problem: they are a symptom, and the right handling depends on the mechanism behind each field's missingness, not on a single blanket rule for the dataset. I would look at income, last_login, and plan_type separately, because a numeric field missing due to a survey skip, a timestamp missing because the event never happened, and a category missing because of an integration gap all call for different treatments.

Structured elaboration

The common thread: decide whether the fact that a value is missing carries information before deciding how to fill it in. Missingness that is itself predictive (an MNAR pattern, missing-not-at-random) should stay visible to the model as a flag; missingness that looks closer to random can be imputed more mechanically without much risk.

Worked example

Concretely, before finalizing any of the three choices above, I would run this check: compare the target churn rate for rows where last_login is missing against rows where it is present.

• If the two groups have similar churn rates, the missingness in last_login is probably closer to random and treating it lightly (e.g. simple imputation) carries lower risk.
• If missing-last_login rows churn dramatically more (or less) than the rest, that gap is signal, and collapsing it into an imputed "typical" login date would erase the single most predictive fact in the dataset. In that case the never_logged_in flag is not an optional nicety, it is likely one of the strongest features in the model.

I would run the same comparison for income and plan_type rather than assuming the table above holds for every dataset; the mechanism has to be checked, not guessed.

Trade-offs and pitfalls

• Imputing on the full dataset before splitting leaks information from validation/test rows into the training-time imputation statistic (e.g. a median computed across all rows). Fit any imputer on the training split only, then apply it unchanged to validation and test.
• Dropping rows is the easiest option and the most dangerous: if missingness correlates with the outcome (which is exactly the case that matters most for churn), dropping those rows biases the remaining dataset toward the "well-behaved" users and the model will underperform precisely on the segment it most needs to catch.
• A missingness flag with no real fill value change can still help even for fields you do impute: keeping income_missing as a separate column after filling income with the median lets a model use "the value" and "whether the value was actually observed" as two independent signals, instead of collapsing them into one filled number that looks identical to genuinely observed data.
• Validate by segment, not just in aggregate: after choosing a treatment, compare model calibration and error rates across the missing-vs-present subgroups specifically, since an averaged validation metric can look fine even while the model is quietly wrong on exactly the rows that had missing data.

Direct answer

A p-value is the probability, computed assuming the null hypothesis $H_0$ is true, of observing a test statistic at least as extreme as the one actually observed. It is a statement about the data given the null, not a statement about the probability that the null (or the alternative) is true. A small p-value is evidence against $H_0$, but by itself it says nothing about how large or important the underlying effect is.

Structured elaboration

Formal definition

For an observed test statistic $T_{obs}$ computed from the data:

$$p = P\big(T \ge T_{obs} \ \big|\ H_0 \text{ true}\big)$$

(one-sided; a two-sided p-value sums both tails). The p-value depends on the chosen test statistic and the definition of "extreme," so two valid tests on the same data can report different p-values.

Common misconceptions

What to report instead of a p-value alone

Pair every p-value with an effect size (the actual magnitude of the difference, in business units) and a confidence interval (the range of plausible effect sizes given the data). The p-value answers "is this compatible with pure chance," the effect size and interval answer "how big is it, and how sure are we."

Worked example

Two groups, $n = 10{,}000$ per group, common standard deviation $\sigma = 1$, observed mean difference $= 0.10$.

Standard error of the difference:

$$SE = \sqrt{\frac{2\sigma^2}{n}} = \sqrt{\frac{2(1)^2}{10{,}000}} = 0.014142$$

Test statistic and two-sided p-value:

$$z = \frac{0.10}{0.014142} = 7.0711 \quad\Rightarrow\quad p \approx 1.5\times 10^{-12}$$

(verified with scipy.stats.norm.sf: 2 * norm.sf(7.0711) = 1.537e-12)

95% CI for the difference: $0.10 \pm 1.96(0.014142) = [0.0723,\ 0.1277]$.

Cohen's d (standardized effect size) is $0.10 / 1 = 0.10$, conventionally a small-to-negligible effect.

Correctly-reported sentence: "Treatment increased the metric by 0.10 units (Cohen's d = 0.10, a small effect), 95% CI [0.072, 0.128], p < 0.001; the effect is real but modest, so ship/no-ship should hinge on whether a 0.10-unit change is worth the cost, not on the size of the p-value."

Trade-offs & pitfalls

• With enough data, essentially any non-zero effect becomes statistically significant, so at large $n$ the p-value stops discriminating between "trivial" and "important" and the effect size has to do that work.
• Reporting p-values without an effect size and interval invites exactly the "significant equals important" misreading this answer warns against.
• The one-sided vs. two-sided choice, and the definition of "extreme," must be fixed before looking at the data. Changing either after seeing results inflates the true error rate above the stated p-value.

Direct answer

GROW is a four-stage, question-led coaching structure: Goal (what success looks like), Reality (the current state), Options (possible paths forward), and Way forward (specific commitments). Applied to a 1:1 with someone who wants to grow a specific skill, it turns a vague aspiration into a concrete next step, and the same question-led habit also works inside a work review, not only a scheduled conversation.

Walking through the four stages

• Goal. Get specific: "What would 'better at this' actually look like, concretely, and how would you know it happened?"
• Reality. Surface the current state without judgment: "Tell me about a recent situation where this was hard, what made it hard?"
• Options. Generate paths rather than prescribing one: "What could you try next, and who or what could help?"
• Way forward. Get a specific, small commitment: "Which one thing will you actually do before we talk again, and what support do you need from me?"

Socratic questioning is the companion technique that runs through all four stages: instead of stating the answer, ask a question that leads the person to notice the gap themselves ("what did you expect to happen there, versus what actually happened?"). It works well when there's time to let someone arrive at the insight; it works poorly when someone is genuinely blocked and just needs the direct answer.

Extending this into reviewing someone's work

The same question-led approach makes a review of someone's work (code, a document, a design, an analysis) constructive rather than purely corrective. Concrete techniques: a review template that separates "must fix" from "worth considering" from "just for your awareness," so feedback doesn't read as one undifferentiated pile of criticism; annotated examples that show a better version alongside the original with a short reason, not just a comment naming the problem; and a Socratic question left in the review itself ("what happens here if this is empty?") instead of stating the bug outright, when the goal is teaching and there's no urgency forcing a direct fix.

Worked example

In a 1:1, a mentee said they wanted to get better at making structural decisions independently instead of always checking first. Goal: they described what "independent" would look like in practice (making a defined class of calls without asking). Reality: walking through a recent case, they could explain their reasoning but hadn't trusted it enough to act without confirmation. Options: they proposed trying it on a low-stakes decision first and reviewing the reasoning after the fact rather than before. Way forward: they committed to making the next reversible decision on their own and bringing the reasoning to the following session, with an explicit offer of support if it went wrong.

Trade-offs and pitfalls

A common mistake is treating GROW as a rigid script and marching through all four stages regardless of what the person actually needs that day. A stronger approach holds the structure loosely: skip Reality if it's already obvious, compress stages under time pressure, and know when the moment calls for direct answers instead of more questions, especially if something is safety-critical or urgent. Inside reviews specifically, overusing Socratic questions when someone is genuinely stuck can read as withholding rather than teaching, so it's worth pairing questions with a clear direct answer once the teaching moment has been made.

Direct answer

For MNAR data where sicker patients are systematically less likely to have follow-up labs recorded, advanced strategies include pattern-mixture models (modeling the outcome separately for the missing-versus-observed groups, then combining), selection models (explicitly modeling the missingness mechanism itself alongside the outcome), and sensitivity analysis (testing how much your conclusions would change under a range of plausible MNAR assumptions, since the true mechanism is fundamentally unverifiable from the observed data alone).

Structured elaboration

Pattern-mixture models: stratify the analysis by missingness pattern (patients with the lab recorded, versus without), model the outcome within each stratum separately, then combine the stratum-specific results, explicitly acknowledging that the two groups may have a genuinely different relationship between features and outcome rather than assuming they don't.

Selection models: jointly model the outcome of interest AND the probability of the lab being recorded, allowing the two models to share information (a patient's severity affects both whether they get the lab, and the outcome itself), which can partially recover the MNAR-induced bias if the SHARED severity signal is at least partly captured by other observed variables, even though the missingness depends on the unobserved lab value itself.

Sensitivity analysis: since MNAR's true mechanism can't be verified from the data alone (that's definitionally what makes it MNAR rather than MAR), run the analysis under several different plausible assumptions about how the missing values relate to severity (a range of assumed shifts), and report how much the conclusion changes across that range, giving stakeholders an honest picture of how much the result depends on an assumption that can't be directly checked.

Evaluating effectiveness statistically and ethically: statistically, compare the pattern-mixture or selection-model results against a naive (MCAR-assuming) imputation baseline on any AVAILABLE ground truth (a subset where follow-up data was eventually obtained through some other channel, if one exists) to see whether the more sophisticated approach actually reduces measurable bias; ethically, be explicit in any reporting or downstream clinical use that the sicker, unmeasured patients are systematically underrepresented in the "observed" data, and that any point estimate carries real, quantifiable uncertainty from this source that a single imputed number would otherwise hide, since silently treating an MNAR-derived estimate as certain risks a clinical decision built on a systematically-biased foundation without anyone realizing it.

Worked example

If sicker patients missing follow-up labs are, say, twice as likely to have a poor outcome based on other observed variables (comorbidities, admission severity scores) as those with the lab recorded, a pattern-mixture model that fits the outcome relationship SEPARATELY within the "lab recorded" and "lab missing" strata, rather than pooling them under a shared imputed value, avoids diluting that severity signal into a single number that under-represents how sick the missing-lab group actually is.

Trade-offs and pitfalls

The honest, uncomfortable trade-off in all of this: no technique here TRULY solves MNAR, since by definition the missingness depends on something you can't observe, and every method above is a way of being more explicit and more careful about the resulting uncertainty, not a way of eliminating it; a stakeholder expecting a single, fully-corrected number needs to understand that MNAR fundamentally limits how confident any single point estimate can honestly be.