Netflix Machine Learning Engineer Interview Guide - Senior Level

First split the problem into two buckets

1. True optimization issues: the model is mathematically harder to train at larger scale.
2. Distributed systems issues: the parallel run is changing the effective training behavior.

Things I would investigate

• Effective batch size and learning rate scaling. A bigger world size often means a larger batch, which can require LR tuning.
• Data sharding. I would verify that each sample is seen once, not duplicated or skipped.
• Gradient synchronization. I would check that all ranks contribute the same gradients and that there are no stale or dropped updates.
• Precision and numerics. Mixed precision can introduce overflow, underflow, or different rounding behavior at scale.
• Stateful layers like batch normalization, which can behave differently across replicas.

How I would separate them
I would compare small-scale and large-scale runs with the same global batch and fixed seeds, then inspect gradient norms, loss curves, and sample hashes.

Worked example
If 8 workers train with a batch of 256 each, the global batch becomes 2048. If loss gets noisier only after that change, it may be an optimization tuning problem. If the run only degrades when messages cross nodes, it is more likely a communication or synchronization problem.

That distinction tells me whether to tune the optimizer or fix the distributed stack.

Direct answer

Roll back when the signals show the blast radius is growing or user-facing impact is severe and the fix isn't yet proven; keep debugging in place when the signals are stable or narrowing and you have a low-risk path to more evidence. The rubric is about trend and confidence, not just the current error rate: a flat 2% error rate you understand is safer to sit with briefly than a 0.5% rate that's climbing and whose cause is still unknown.

Structured elaboration

Concrete signals that push toward rollback:

• Error rate or latency is still climbing, not plateaued, meaning the blast radius is actively growing while you investigate.
• The failure touches a critical path (payments, auth, data writes) where continued exposure risks data integrity or revenue, not just degraded UX.
• You cannot yet state a specific hypothesis for the cause. If you don't know what's wrong, you can't bound how bad it might get, and every minute of uncertainty is a minute of unbounded risk.
• The most recent deploy is a plausible cause and rolling it back is cheap (a well-tested, low-risk rollback path exists). When rollback is cheap and the cost of being wrong is high, the asymmetry favors rolling back even before the cause is confirmed.

Signals that support continuing to debug in place:

• The metric is flat or already recovering on its own (e.g., a transient dependency blip that's clearing), meaning the risk is bounded and shrinking.
• You have a specific, testable hypothesis and the evidence to confirm or reject it is minutes away, not hours.
• The failure is isolated to a non-critical path or a small, known user segment, so continued exposure has a low, well-understood ceiling.
• Rolling back has its own real cost (loses a needed migration, reintroduces a different known bug, or the rollback path itself is unproven), making rollback riskier than the current state.

Two more factors that belong in the rubric alongside the raw signals: your technical confidence in a proposed quick patch, and your monitoring capability while you wait. A patch you are highly confident in, that you can deploy and verify within minutes, tips the balance toward fixing forward even at a moderate error rate; a patch you're guessing at does not, no matter how appealing "just ship the fix" feels under pressure. Separately, if your monitoring can only tell you the aggregate error rate but not WHO is affected or WHY, you are debugging half-blind, and that itself is a reason to prefer the safer, well-understood state (rollback) over continuing to poke at a system you can't fully observe.

Worked example

A rubric applied in production: after a deploy, error rate on the checkout endpoint rises from 0.1% to 1.5% and holds flat for four minutes while you check logs. The rate is not climbing further, the failures are concentrated in one specific edge case (carts with a discount code applied), and a log line points directly at a null-handling bug in the new discount logic. Decision: continue debugging, because the blast radius is bounded, isolated, and you have a specific hypothesis you can confirm in the next two minutes. Contrast: if that same 1.5% had climbed to 6% over those four minutes with no clear pattern in the failing requests, the correct call flips to immediate rollback, because the trend is the dominant signal, not the absolute number.

Trade-offs and pitfalls

The rubric fails when applied to a single snapshot instead of a trend: a rate that looks acceptable in isolation can be five minutes from becoming a major incident, or a rate that looks alarming can already be resolving itself. The discipline is to always ask "is this getting better, worse, or staying the same" before deciding, and to treat "I don't have a hypothesis yet" as itself a strong vote toward rollback, since it means you cannot bound the risk of waiting.

Direct answer

Approach it as a narrowing funnel: confirm the growth is real and reproducible, use tracemalloc to find where allocations are coming from, then use the gc module (and, for reference-graph questions, objgraph) to find what is holding those objects alive and why they were not freed. Check the common Python-specific leak patterns first, since most long-running-process leaks are one of a small set of shapes: unbounded caches/global registries, unclosed resources (sessions, file handles, connections), reference cycles combined with objects the cyclic collector cannot promptly reach, and closures or long-lived threads capturing large objects.

Structured elaboration

Step 1, confirm and reproduce: watch process resident memory (via psutil or your platform's monitoring) over a representative load pattern (a load test or an accelerated replay), not just a hunch that "it feels like it's growing." A leak that only shows up after days needs either patience or an accelerated repro before you spend time instrumenting it.

Step 2, find allocation sites with tracemalloc: take a snapshot, run the suspect workload, take another snapshot, and diff them. This ranks allocations by where in the source they happened, which narrows "memory is growing" down to specific lines or call sites.

Step 3, find what's holding things alive with gc: gc.get_objects() lists every object the garbage collector currently tracks (container types capable of participating in a cycle; simple scalars like int and str are not tracked, they cannot hold references to other objects). Counting instances by type over time shows which type is accumulating. gc.get_referrers(obj) walks backward from a suspect object to whatever is holding a reference to it, which is how you find the actual retaining path (a global dict, a closure, a still-registered callback) rather than just knowing the type is growing.

Step 4, objgraph for visual reference-graph tracing: objgraph.show_most_common_types() gives the same per-type counts as the gc.get_objects() counter above, and objgraph.show_backrefs([suspect], max_depth=...) renders the actual reference chain keeping an object alive, which is much faster to read than manually following gc.get_referrers chains by hand once the graph gets more than one or two hops deep.

Common Python-specific leak patterns, roughly in the order to check them:

1. Unbounded module-level or global caches/registries (a plain dict used as a cache with no eviction).
2. Unclosed resources with finalizers (an aiohttp.ClientSession, a database connection, an open file) that are only freed when the GC eventually runs their __del__, not promptly.
3. Closures or callbacks capturing a large object and being registered somewhere long-lived (an event bus, a scheduler) and never unregistered.
4. Reference cycles, especially now that __del__-bearing cycles are collected safely (since Python 3.4), the more common failure mode is not "uncollectable," it's "not yet collected" because the object landed in an older GC generation that is scanned less frequently, or gc was disabled somewhere in the process.

Worked example

Verified on CPython 3.12. tracemalloc, showing where repeated allocations are coming from:

import tracemalloc

tracemalloc.start(10)
leaked = []

def leaky_step():
    leaked.append(bytearray(10_000))   # simulates an unbounded cache append

snap1 = tracemalloc.take_snapshot()
for _ in range(50):
    leaky_step()
snap2 = tracemalloc.take_snapshot()

for stat in snap2.compare_to(snap1, "lineno")[:1]:
    print(stat)

Output (path shortened): ...leaky_step():8: size=491 KiB (+491 KiB), count=101 (+101), average=4983 B, pointing straight at the leaked.append(...) line as the growth source. (The exact byte counts scale with the object sizes and counts chosen here; what matters is that the tool attributes growth to a specific line, not the specific number.)

gc, counting live instances of a specific type to confirm growth (using a class instance, not a plain bytearray, since bytearray and other simple scalar types are not tracked by the cyclic collector and would not show up in gc.get_objects() at all):

import gc, collections

class CacheEntry:
    def __init__(self, payload):
        self.payload = payload

leaked = []
before = collections.Counter(type(o).__name__ for o in gc.get_objects())["CacheEntry"]
for _ in range(500):
    leaked.append(CacheEntry(bytearray(100)))
after = collections.Counter(type(o).__name__ for o in gc.get_objects())["CacheEntry"]
print("before:", before, "after:", after)

Output: before: 0 after: 500, confirming 500 CacheEntry instances are alive and tracked, matching the 500 appended.

Fix pattern, weak references so a cache does not keep entries alive on its own:

import weakref

cache = weakref.WeakValueDictionary()

class Big:
    pass

obj = Big()
cache["k"] = obj
print("in cache before del:", "k" in cache)   # True
del obj
import gc; gc.collect()
print("in cache after del:", "k" in cache)    # False: the cache no longer keeps it alive

Trade-offs & pitfalls

• Complexity: tracemalloc adds per-allocation bookkeeping overhead proportional to the number of stack frames you ask it to keep (the 10 above); deeper stacks cost more, both in memory and in snapshot time, so dial it back once you have narrowed the search.
• Edge case: a consumer or handler that raises an exception before releasing a resource (closing a session, calling a cleanup callback) leaks that resource on every failure path, not just the happy path; audit except branches, not just the normal flow.
• Common wrong turn: assuming a growing gc.get_objects() count for a type means a reference cycle. It often just means ordinary, uncycled objects are being kept alive by a single strong reference somewhere (a cache, a list) that never gets cleared; gc.get_referrers will show a direct reference in that case, not a cycle, and no amount of gc.collect() will free it, since it is not garbage, it is genuinely still reachable.
• Restarting long-running processes periodically is a legitimate operational mitigation for a slow leak while the root cause is investigated, but it masks the underlying bug rather than fixing it, and hides real regressions if used as a permanent substitute for finding the leak.
• Edge case: native (C extension) memory (for example, GPU buffers or malloc'd data used by an image or ML library) is invisible to tracemalloc and gc, both only see Python-level objects; a leak in that layer needs an external tool, not more digging in gc/tracemalloc.

Direct answer

Gradient clipping caps the magnitude of gradients before the optimizer update to prevent occasional very large gradients (from an unstable loss landscape, a bad mini-batch, or numerical issues) from causing a destructively large parameter update; the two common strategies are global-norm clipping (rescale the entire gradient vector if its overall norm exceeds a threshold) and value clipping (clamp each individual gradient component to a fixed range independently).

Structured elaboration

• Global norm clipping: compute the L2 norm of the concatenated gradient across all parameters; if that norm exceeds a threshold max_norm, rescale every gradient component by the same factor max_norm / norm so the overall gradient direction is preserved but its magnitude is capped. This is the more common choice for deep networks (especially RNNs and transformers) because it preserves the gradient's relative direction across parameters, only shrinking its overall size.
• Value clipping: clamp each individual gradient component independently to a fixed range (e.g. [-c, c]), regardless of the overall norm. This is simpler and cheaper to compute (no need to first compute a global norm across all parameters), but can distort the gradient's direction, since components near the clip boundary get truncated disproportionately relative to smaller components, unlike norm clipping's uniform rescaling.
• Impact on training dynamics: clipping prevents a single unstable step from derailing training (a common cause of NaN/divergence in RNNs and transformers, especially early in training or with an aggressive learning rate), at the cost of somewhat biasing the effective update whenever clipping actually triggers (the update is smaller and, for value clipping, differently-directed than the "true" unclipped gradient would have produced).

Worked example

A gradient vector with an unusually large spike (norm 50, versus a typical norm around 2) and a clip threshold max_norm=1.0: global norm clipping rescales every component by 1.0/50 = 0.02, preserving the gradient's direction while capping magnitude to exactly 1.0; value clipping with a threshold c=0.1 instead clamps any individual component exceeding 0.1 to exactly 0.1 regardless of the others, which for a gradient with most components small but a few very large ones changes the update's direction, not just its magnitude, since the large components get disproportionately truncated relative to the small ones.

Trade-offs & pitfalls

Norm clipping's threshold needs to be chosen relative to the model's typical (unclipped) gradient norm during stable training, not an arbitrary small number; a threshold set far below typical gradient norms clips essentially every step, meaningfully slowing learning by systematically shrinking updates that weren't actually problematic.

Direct answer

Developing independent judgment, not just correct outputs, requires repeated exposure to the same class of decision with you gradually receding from it, and requires the person to narrate their reasoning, not just report their choice. You know it's internalized, not just imitated, when their reasoning transfers to a situation you never coached them on directly, ideally one you weren't even present for.

How judgment gets built and verified

Coach the decision class, not the individual decision. A one-off answer to "should we do X" teaches them what to do this time. Judgment comes from recognizing the same underlying trade-off recurring in different clothes, which means you have to name the pattern explicitly rather than just resolving each instance.

Recede deliberately in stages. Start by explaining your own reasoning out loud when a decision comes up. Then ask them to predict what you'd decide, and why, before you weigh in. Then let them make the call and explain their reasoning to you after the fact. Then stop reviewing it at all. Each stage removes a layer of your safety net.

Make them narrate the criteria, not just the outcome. If someone can only say "I did X because I figured that's what you'd want," they've pattern-matched to you specifically, not internalized the underlying principle. You're listening for whether their stated reasoning would still hold up in a case where the "obvious" answer is actually wrong.

Verify with a novel or unobserved case. The strongest signal is watching them apply the same reasoning to a situation they haven't seen before, particularly one where you weren't in the loop and only heard about the decision afterward.

Worked example

Someone you're mentoring kept bringing you a specific recurring trade-off as if it were a one-off question each time: whether to fix a flaky, intermittently-failing test or ship a feature that was ready and waiting on it. Each time, you could have just answered the immediate question. Instead you treated it as a judgment gap and built a repeatable heuristic with them: is the flake masking a real intermittent bug or is it environment noise, what's the actual blast radius of shipping with it unresolved, and is there a way to quarantine the test that unblocks delivery without hiding the underlying risk.

Weeks later, a similar trade-off came up and they handled it without asking you first, only mentioning the decision afterward along with their reasoning. Their stated criteria matched the heuristic you'd built together, but in their own words, applied to a case with a different shape than the original one. That, not their confidence in the moment, was the signal it had actually internalized rather than just been remembered.

Trade-offs and pitfalls

Asking someone "do you understand?" tells you almost nothing; people say yes regardless of whether it's true. The only real test is watching the reasoning survive a situation you didn't script.

A subtle failure mode: rewarding a decision because it matches what you personally would have done, rather than evaluating whether the reasoning behind it was sound. If the original case was genuinely a coin toss, insisting they land on your exact answer trains obedience, not judgment.

The deeper trade-off is time and tolerance for being wrong. Actually receding means letting them face real stakes without a safety net, which means tolerating some decisions that turn out wrong in hindsight. That's not a bug in the process; it's the cost of judgment actually being tested rather than simulated.

A mentor who never truly recedes, who keeps reviewing every instance of the decision "just to be safe," never actually finds out whether the judgment transferred, because it's never been tested without the net.

Direct answer: Amdahl's law bounds the maximum SPEEDUP from parallelizing a FIXED-size workload: if a fraction $p$ of the work is parallelizable and $(1-p)$ is inherently serial, speedup with $N$ processors is capped at $\frac{1}{(1-p) + p/N}$, approaching a hard ceiling of $\frac{1}{1-p}$ as $N \to \infty$ - even infinite processors can't beat the serial portion's fixed cost. Gustafson's law reframes the question for a GROWING workload (the common real-world case: given more processors, you solve a BIGGER problem in the same time, not the same problem faster), giving speedup that scales roughly linearly with $N$ instead of hitting Amdahl's fixed ceiling, because the serial fraction typically doesn't grow with problem size.

Structured elaboration

• Amdahl's law: $\text{Speedup}(N) = \frac{1}{(1-p) + p/N}$. The key insight: even a SMALL serial fraction imposes a hard ceiling. With $p=0.95$ (95% parallelizable, seemingly excellent), the maximum possible speedup as $N \to \infty$ is $\frac{1}{1-0.95} = 20\times$ - no matter how many processors you throw at it, you cannot exceed 20x speedup on this FIXED workload, because the remaining 5% serial work eventually dominates as parallel processors reduce the parallel portion's time toward zero.
• Gustafson's law: $\text{Speedup}(N) = N - (1-p') \times (N-1)$ (using $p'$ as the parallel fraction of the SCALED-UP workload) - reframes speedup as: given $N$ processors, how much MORE work can be done in the same fixed time, relative to 1 processor? Since the serial portion is typically a fixed, small overhead independent of problem size (not a fixed FRACTION that necessarily grows), scaling up the problem size lets the parallel portion dominate more and more as $N$ grows, giving near-linear speedup for realistic workloads - a fundamentally more optimistic picture for the common case of "we have more compute, so let's solve a bigger problem," not "we have more compute, let's solve the SAME problem faster."

Worked example

For deciding whether to parallelize a legacy single-threaded algorithm: first measure (via profiling) what fraction of its runtime is INHERENTLY serial (setup, I/O, dependent sequential steps) versus parallelizable. If profiling shows 30% serial, 70% parallelizable ($p=0.7$), Amdahl's law caps maximum speedup at $\frac{1}{0.3} \approx 3.33\times$ regardless of how many cores you add - meaning an investment in, say, 64-core parallelization would be substantially wasted effort if the GOAL is making this exact, fixed-size task run faster (you'd get nowhere near 64x, capping out around 3.33x, likely reached with far fewer cores). If instead the actual business goal is "process 64x more data in the same time" (a Gustafson-shaped goal), the same 30%-serial-overhead codebase could scale far more favorably, since the serial 30% might well be a roughly FIXED cost (e.g. one-time setup) that becomes a shrinking fraction as the per-processor workload grows.

Trade-offs & pitfalls

• The FIRST question before parallelizing anything should be "is this a fixed-size-speedup goal (Amdahl) or a bigger-problem-in-same-time goal (Gustafson)?" - conflating the two leads to either wasted parallelization investment (chasing Amdahl-capped speedup with too many cores) or an overly pessimistic assessment (applying Amdahl's ceiling to a genuinely Gustafson-shaped problem).
• Measuring the ACTUAL serial fraction via profiling (not guessing) is essential before either law can give a meaningful, actionable number - "70% parallelizable" needs to come from real measurement, not intuition.
• Neither law accounts for REAL parallelization overhead (communication, synchronization, load imbalance) beyond the idealized serial/parallel split - actual achieved speedup is typically somewhat worse than either formula predicts, due to these additional practical costs.

Situation: I led a launch that needed Product, Engineering, and Support to deliver the same outcome, which was reducing customer setup time.

Task: Each team had different priorities, so I needed one shared goal and a way to surface trade-offs early.

Action: I started with a single business metric, then broke it into team-level commitments. Product owned the user flow, Engineering owned reliability, and Support owned readiness. I held a weekly cross-functional checkpoint where each team shared risks, not just status. When conflicts came up, I made the trade-off explicit. For example, we chose to delay one nonessential feature so we could simplify onboarding and reduce support tickets.

Result: The teams stayed aligned, the launch shipped with fewer surprises, and the process made future collaboration easier because everyone knew how decisions would be made.

The key lesson was that shared outcomes work best when the goal is visible, disagreements are discussed early, and trade-offs are decided openly instead of being left to drift.

Best design for this problem

I would treat the customer table as a history table with effective start and effective end dates for each segment record. That makes the join logic much safer than trying to guess which duplicate row is current.

Table design

• CustomerID
• Segment
• StartDate
• EndDate

Each order then matches the row where the customer matches and the order date falls inside the effective date range.

Formula approach

=IFERROR(
  INDEX(CustomerHistory[Segment],
    MATCH(1,
      (CustomerHistory[CustomerID]=[@CustomerID]) *
      (CustomerHistory[StartDate]<=[@OrderDate]) *
      (CustomerHistory[EndDate]>[@OrderDate]),
    0)
  ),
"Unknown")

Why this works

• It handles customers who change segments over time.
• It avoids accidental matches to the wrong historical record.
• It gives a clean fallback when no record exists.

If I wanted a simpler model

• I would use Power Query to merge orders to the customer history table and filter by the date range.
• That is often easier to audit than a very long formula.

If there are duplicate history rows with the same date range, I would fix the source data first, because the lookup should not guess between conflicting records.

Direct answer

Frame non-inferiority as a one-sided test on the difference minus the margin: the null is that the new engine is worse than the current one by at least the margin $\delta$, and you reject that null (concluding non-inferiority) if the data rule it out. The Two One-Sided Tests (TOST) procedure runs this as two one-sided tests simultaneously; for a pure non-inferiority claim you only need the lower-bound side. Sample sizes for small margins are large, often far larger than a standard superiority test, because you're trying to rule out a small effect with precision rather than just detect any effect at all.

Structured elaboration

Hypotheses

With $p_{\text{new}}$ and $p_{\text{curr}}$ the true conversion rates and margin $\Delta = -0.005$ (a drop of more than 0.5 percentage points counts as "worse"):

Non-inferiority (one-sided):

Equivalence (two-sided, TOST), if the goal were "not meaningfully different in either direction":

TOST procedure

Run two one-sided tests against the two margin boundaries:

1. Test $p_{\text{new}} - p_{\text{curr}} > -\Delta$ (reject if the standardized statistic exceeds $z_{1-\alpha}$)
2. Test $p_{\text{new}} - p_{\text{curr}} < +\Delta$ (reject if the statistic is below $z_{\alpha}$)

If both one-sided tests reject, conclude equivalence within $\pm\Delta$. For pure non-inferiority (only care about the "not worse" direction), run only test 1, using a one-sided $\alpha$ (commonly 0.025, matching the one-sided error rate of a standard 0.05 two-sided test).

Test statistic (large-sample z-test for a difference of proportions):

Sample size

Choose one-sided $\alpha$, desired power $1-\beta$, a planning baseline rate $p_0$, and the true effect you want to be powered to detect ($\delta_{\text{target}}$, typically 0 - assuming the new engine performs identically to the old one and you want to reliably rule out that it's actually $\Delta$ worse):

Worked example

Planning inputs: baseline $p_0 = 0.10$, margin $\Delta = -0.005$, one-sided $\alpha = 0.025$, power $= 80\%$, assuming true rates are equal ($\delta_{\text{target}} = 0$, i.e. $p_1 = p_0 = 0.10$).

from scipy import stats
alpha, power = 0.025, 0.80
z_alpha, z_beta = stats.norm.ppf(1-alpha), stats.norm.ppf(power)
p0, delta_margin = 0.10, 0.005
var_term = 2 * p0*(1-p0)
n_per_group = (z_alpha+z_beta)**2 * var_term / delta_margin**2
# n_per_group = 56511.93... -> round up to 56512 per arm, ~113,024 total

Recommending this to stakeholders: "Ruling out a 0.5-point drop in conversion with 80% power requires roughly 56,500 users per arm, over 100,000 total, because we're trying to detect the absence of a small effect precisely, not just the presence of any effect." That single number is often the deciding factor in whether non-inferiority testing is practical for a given margin.

Trade-offs & pitfalls

• Margin choice drives everything. A tight margin (e.g., 0.5 points) demands a much larger sample than a loose one; the margin should be set from business impact (how much conversion loss is tolerable) or prior data, not picked for statistical convenience.
• One-sided $\alpha$ still needs justification. Using 0.025 one-sided matches a conventional two-sided 0.05 test's error budget, but confirm this convention with stakeholders since it directly trades off against required sample size.
• Analysis population matters more here than in superiority testing. Per-protocol analysis (excluding non-compliant users) is often preferred for non-inferiority because intention-to-treat analysis tends to bias results toward apparent non-inferiority (dilution of a real difference), which is the wrong direction to be conservative in.
• Asymmetric costs change the design. If a regression in conversion is much costlier than a similar-sized loss elsewhere, consider a stricter margin or higher power rather than treating this like a standard A/B test.

Direct answer
Refit the model on many bootstrap resamples of the training set, predict on a held-out test set from each refit, and use the AVERAGE prediction across resamples to estimate squared bias (versus the true function, if known, or a proxy) and the SPREAD across resamples to estimate variance.

Structured elaboration
For each of B bootstrap samples (drawn with replacement from the training set, same size as the original), refit the model and predict on a fixed held-out test set. Across all B sets of predictions at each test point: the mean prediction across resamples, compared to the true value, gives squared bias at that point; the variance of predictions across resamples at that point gives the variance term. Averaging both quantities across all test points gives an overall bias-squared and variance estimate for the model.

Worked example (executed)

import numpy as np

def bootstrap_bias_variance(model_fn, X_train, y_train, X_test, y_test_true_fn, n_bootstrap=50, seed=0):
    rng = np.random.RandomState(seed)
    n = len(X_train)
    preds = np.zeros((n_bootstrap, len(X_test)))
    for b in range(n_bootstrap):
        idx = rng.randint(0, n, size=n)
        model = model_fn().fit(X_train[idx], y_train[idx])
        preds[b] = model.predict(X_test)
    mean_pred = preds.mean(axis=0)
    y_true = y_test_true_fn(X_test)
    bias_sq = np.mean((mean_pred - y_true) ** 2)
    variance = np.mean(preds.var(axis=0))
    return bias_sq, variance

Run with a known ground-truth linear function (y = 3*x0 + 2*x1) and 60 bootstrap resamples each: a correctly-specified linear regression showed bias-squared 0.036 and variance 0.028 (both small, as expected for a model whose functional form matches the truth), while an unpruned decision tree on the same data showed bias-squared 1.374 and variance 3.379, dramatically higher variance in particular, exactly the expected signature of a high-capacity model overfitting to whichever specific bootstrap sample it happened to see.

Trade-offs & pitfalls
This estimate requires knowing (or having a good proxy for) the true function to compute bias against, which is available for synthetic verification but not for a real-world problem; on real data, you typically substitute the FULL-DATA model's prediction (fit once on all available data) as a stand-in for the "true" value when computing bias, an approximation, not an exact ground truth.