Apple Machine Learning Engineer (Staff Level) - Comprehensive Interview Preparation Guide

Direct answer
The underlying facts stay identical between an internal-executive version and an external audience version, but the internal version can include strategic context, risk framing, and unresolved uncertainty that the external version deliberately omits or simplifies, since the external audience doesn't share the same context or stake in the internal decision-making process.

Structured elaboration
What typically changes: the internal version can name specific trade-offs, competitor comparisons, or internal disagreements candidly, since the audience already has the organizational context to interpret them appropriately; the external version strips that context-dependent material and focuses on what the external party actually needs, without exposing internal deliberation that could be misread or used against you. What stays the same: the core facts and any commitment being made publicly should match exactly what's being represented internally, since a gap between the two versions, if discovered, is a credibility problem far worse than either version alone.
Preparation for the internal audience also typically comes first, since internal stakeholders need to be aligned and aware before an external version goes out, to avoid anyone being surprised by an external communication about something they didn't know was happening.

These three external audiences don't all get the same treatment, though. A customer version genuinely simplifies, stripping rationale and internal detail down to what they actually need to know. An auditor version usually needs the opposite: full traceability and often more granular detail than the internal version, since the auditor's specific job is to verify claims against evidence, and withholding detail because the audience is technically "external" would defeat that purpose. An investor version sits in between: the facts stay as precise and complete as the auditor version, since materiality and accuracy carry legal weight, but the framing adopts the cautious, forward-looking-statement style investors and regulators expect, rather than the confident simplicity that works for a customer.

Worked example
Communicating a pricing change: the internal executive version can discuss the specific competitive pressure driving the decision and the internal debate about timing; the external customer-facing version states the change plainly and simply, with a rationale framed around value delivered rather than competitive dynamics, and doesn't reference any internal disagreement about timing. The actual facts of the change (amount, effective date) are identical in both.

Trade-offs and pitfalls
The most common failure is letting the two versions diverge on the actual facts, not just the framing, which creates real risk if the discrepancy is ever discovered or compared. The second is failing to sequence internal-first, so an internal stakeholder learns about an external communication after the fact rather than before, which damages trust even if the content itself was accurate.

Direct answer

Slide a window over the string with two pointers, and keep a hash map from character to the index right after its most recent occurrence. When the character at the right pointer has been seen before at or after the current window's left edge, jump the left pointer forward to just past that previous occurrence, but never backward, and track the widest window seen along the way. Because the left pointer only ever advances and the right pointer sweeps the string once, the whole scan is $O(n)$ time using $O(\min(n, \text{alphabet size}))$ space for the map.

Approach

• Maintain last_seen, mapping each character to one past the index of its most recent occurrence (storing index + 1 lets an unseen character default cleanly to 0).
• Maintain left, the window's start, and best, the longest valid window length so far.
• For each right, if the current character was seen before, set left = max(left, last_seen[char]); the max is essential, not optional.
• Update best with the current window length right - left + 1, then record last_seen[char] = right + 1.

def length_of_longest_substring(s: str) -> int:
    """O(n) time, O(min(n, alphabet size)) space sliding window."""
    last_seen: dict[str, int] = {}
    left = 0
    best = 0
    for right, ch in enumerate(s):
        if ch in last_seen:
            left = max(left, last_seen[ch])
        best = max(best, right - left + 1)
        last_seen[ch] = right + 1
    return best


if __name__ == "__main__":
    print(length_of_longest_substring("abcabcbb"))  # 3
    print(length_of_longest_substring("bbbbb"))      # 1
    print(length_of_longest_substring("pwwkew"))     # 3
    print(length_of_longest_substring(""))           # 0

Running this prints 3, 1, 3, 0, matching the well-known cases for this problem.

Key points

• Why max, not a direct jump, is required: dropping the max and always jumping left = last_seen[ch] breaks correctness whenever a repeated character's earlier occurrence already fell outside the current window, because that would move left backward, re-including characters that had already been correctly excluded. Concretely, on "abba" a buggy version without max reports length 3, while the correct version reports 2 (the true answer, since "ab" and "ba" are the longest repeat-free substrings, both length 2). Verifying this directly:

def buggy_longest(s: str) -> int:
    last_seen: dict[str, int] = {}
    left = 0
    best = 0
    for right, ch in enumerate(s):
        if ch in last_seen:
            left = last_seen[ch]  # missing max(): can move left backward
        best = max(best, right - left + 1)
        last_seen[ch] = right + 1
    return best


def correct_longest(s: str) -> int:
    last_seen: dict[str, int] = {}
    left = 0
    best = 0
    for right, ch in enumerate(s):
        if ch in last_seen:
            left = max(left, last_seen[ch])
        best = max(best, right - left + 1)
        last_seen[ch] = right + 1
    return best


if __name__ == "__main__":
    print("abba ->", buggy_longest("abba"), "vs correct", correct_longest("abba"))

Running this prints abba -> 3 vs correct 2: the buggy version claims a length-3 window exists in "abba", but no 3-character substring of "abba" is actually repeat-free, so 3 is wrong.

• An alternative that uses a set instead of a map (shrinking left one character at a time until the repeat is gone, rather than jumping straight to the right position) is also correct and still $O(n)$ overall, since each character enters and leaves the set at most once, but it does more per-step work whenever the repeat is far behind the current position.

Complexity

Time: $O(n)$, since right advances once per character and left never moves backward, so together they make at most $O(n)$ total steps. Space: $O(\min(n, \text{alphabet size}))$ for the map.

Edge cases

• Empty string: returns 0.
• All identical characters (e.g., "bbbbb"): returns 1.
• All unique characters: returns the full length of the string.
• A single character: returns 1.

Objective: Deliver an analytics capability that gives execs timely, evidence-based acquisition decisions.
Datasets: Financials (revenue, margins), user metrics (MAU/DAU, retention cohorts), engagement (session length, feature use), market data (comps, TAM), tech health (scalability, debt), legal/IP metrics.
Modeling: LTV and unit-economics models, scenario-based valuation (DCF), growth propensity models (cohort retention forecasting), integration cost models, risk scoring (regulatory, tech fit).
Dashboards: Executive summary (KPIs, sensitivity analyses), deal-level drilldowns (user cohorts, revenue breakouts), red flags (privacy risks, churn), integration timelines.
Due diligence: Standardized checklist, data ingestion pipeline for target data, reproducible ETL, validation tests, backtests on historical acquisitions, playbooks for redlines.
Governance: Access controls, audit trails, one-source-of-truth datasets, and cross-functional review (legal, security, product, finance).
Milestones: 30-day MVP (core datasets + dashboards), 90-day validated models + playbooks, 6-month integration automation, ongoing quarterly reviews.

Direct answer

A human needs to be in the loop at the point where the assistant's output is both hard to fully verify automatically and has real-world consequence if wrong: specifically, before any output reaches a customer or triggers an action outside the system, unless an automated risk score is confidently low. The design decision isn't "always require a human" (that doesn't scale) or "never require one" (that's how a bad draft or a wrong action ships); it's a risk-scored router that sends the low-risk majority straight through and the flagged minority to a reviewer.

Structured elaboration

Architecture

flowchart TD
A[User or workflow trigger] --> B[Retrieval: grounding documents]
B --> C[LLM drafts content or action]
C --> D[Automated safety and policy classifiers]
D --> E{Risk score}
E -->|Low risk| F[Auto-publish or execute]
E -->|High risk| G[Human review queue]
G --> H{Reviewer decision}
H -->|Approve| F
H -->|Edit needed| C
H -->|Reject| I[Discard and log]
F --> J[Audit log: version, risk score, reviewer]

Key design decisions and why

1. Retrieval grounds the draft before the LLM ever generates. Pulling relevant, authoritative source documents into the prompt reduces hallucination and gives the output something to be checked against, both by the automated classifiers and by a human reviewer, rather than a reviewer having to fact-check from scratch.
2. A risk score, not a binary rule, decides whether a human sees it. The safety/policy classifier stage scores each draft or proposed action on dimensions like policy-sensitive content, factual claims that are hard to verify automatically, and irreversibility of the action (a drafted email is reversible before sending; a completed refund or a live-published page is not). This is the design decision that makes human review scale: it's not "review everything" or "review nothing," it's "review the fraction that's actually risky."
3. Irreversibility is the single strongest trigger for mandatory human review, independent of how confident the risk classifier is otherwise. An action that can't be undone once taken (sending a customer communication, executing a payment, publishing externally) should route to a human by default even when the content looks fine, because the cost of being wrong is asymmetric: a bad automated classification on a reversible draft costs an edit; on an irreversible action it costs an incident.
4. The reviewer's decision feeds back into the loop, not just forward. An "edit needed" outcome sends the draft back through generation (with the reviewer's correction as added context), not just a one-off manual fix; this is what keeps human review from becoming a dead-end patch that never improves the underlying system.
5. Every decision, automated or human, is logged with the model version and risk score. This is what makes "who approved this and why" answerable after the fact, which matters both for a customer complaint and for tuning the risk classifier's threshold over time based on which auto-approved outputs later turned out to be problems.

Worked example

Sizing the human-review queue is a straightforward capacity calculation once you know the flag rate and reviewer throughput. Suppose the assistant generates 100,000 drafts/actions per day, and the risk classifier flags the top 5% as high-risk:

If each reviewer can process 200 items/hour over an 8-hour shift:

Four reviewers clear the daily flagged queue with same-day turnaround. This number is what turns "human in the loop" from an abstract safety principle into an actual staffing and SLA (service-level agreement) commitment: if the flag rate crept up to 15% instead of 5%, the same math needs 10 reviewers, not 4, which is exactly the kind of capacity conversation this design should force before launch, not after the queue backs up and outputs start sitting un-reviewed for days.

Trade-offs & pitfalls

• Setting the risk threshold too conservative (flagging too much) doesn't make the system safer in practice, it makes the human queue the bottleneck, and a backed-up queue creates pressure to rubber-stamp reviews quickly, which defeats the purpose; the threshold has to be tuned against real reviewer capacity, not set to the most cautious number in isolation.
• Routing purely on content risk while ignoring reversibility is a common miscalibration: a mundane-sounding draft that triggers an irreversible action (an automated refund, a message to a large distribution list) deserves review even if the classifier sees nothing alarming in the text itself.
• Treating human review as a one-way gate rather than feeding "edit needed" corrections back into generation wastes the signal reviewers are generating; over time that signal should also inform retraining or prompt refinement, not just fix one instance.
• A retrieval layer reduces hallucination but doesn't eliminate it, and it's a mistake to treat "grounded in retrieved documents" as equivalent to "verified"; the retrieved documents themselves can be stale or wrong, so the safety pipeline still needs to check the draft's claims, not just confirm it cited something.
• Audit logging is easy to under-scope to just the final output; capturing the risk score, the model version, and the specific reviewer decision (not just "approved") is what makes it possible to later diagnose whether a bad outcome was a classifier miss, a reviewer miss, or a genuinely hard edge case, and those have different fixes.

Direct answer
A good 30-60-90 plan moves someone from learning the environment, to contributing under supervision, to owning outcomes independently, with the phase boundaries defined by demonstrated behavior (what they can do unsupervised) rather than by the calendar alone. Track it with a small number of concrete, visible outputs per phase so "on track" is something you can point to, not just a feeling.

The three phases, by what changes

• Days 1-30 (learn and observe): environment setup, codebase or domain orientation, shadowing, and one small real contribution rather than a toy task, so the first change is real but low-risk.
• Days 31-60 (contribute under guidance): own a medium-sized piece of work end to end with a mentor available for review and unblocking, not doing it alongside them line by line.
• Days 61-90 (own outcomes): lead something (a project, an on-call rotation, a smaller onboarding task for the next hire) with the mentor as a backstop, not a co-pilot.

How you know they're on track

• Define the signal per phase in advance, not retroactively: for phase 1, did they reproduce the environment and ship one small real change without major help; for phase 2, is their review feedback shrinking in volume and severity over successive changes; for phase 3, can they make a reasonable decision alone and only escalate the genuinely hard calls.
• Check in on cadence (weekly early on, less frequent later) rather than waiting for day 30, 60, or 90 to find out something drifted three weeks ago.

Adjusting the plan for real constraints

• Limited training resources: when there's no dedicated ramp-up bandwidth (no spare mentor hours, no formal training material), lean harder on asynchronous artifacts: written runbooks, recorded walkthroughs, a curated list of the most representative recent changes, and a lighter-touch weekly sync instead of daily pairing. The phases stay the same; what changes is how much is self-serve versus live.
• Cross-skill ramp: if someone hired primarily for one skill set is expected to also ship in an adjacent one by day 90 (for example, a backend-focused hire expected to ship frontend work), that adjacent skill needs its own explicit milestone inside the plan, not an assumption it'll happen by osmosis. Concretely: days 1-30 stays focused on their strong area to build early confidence and trust; days 31-60 introduces the adjacent skill on a small, well-scoped, low-risk piece with close review; days 61-90 has them own something end to end in the new area, even if smaller in scope than their core-skill ownership.

Worked example
For a new hire joining an established codebase with a small team and no dedicated onboarding budget (the limited-resources case), the 30-60-90 looked like: days 1-30, self-serve environment setup using a written runbook plus a single half-day pairing session, culminating in one small, real bug fix; days 31-60, ownership of one medium feature with async review as the main touchpoint, and a short weekly 15-minute sync instead of daily check-ins; days 61-90, the new hire wrote the onboarding runbook update for the next person, which served double duty as both a real deliverable and a check on whether they actually understood the system well enough to explain it. Being on track was tracked by a short checklist per phase (environment reproducible, first fix merged with normal review effort, feature shipped with review comments trending down) rather than a single blanket "how's it going" check-in.

Trade-offs and pitfalls

• Treating the day boundaries as fixed calendar dates rather than behavioral milestones creates false confidence; someone can hit day 60 without actually being ready for phase-3 ownership, and pushing them into it anyway sets them up to fail.
• Under-supporting the adjacent-skill ramp (assuming a backend engineer will "pick up" frontend without an explicit milestone) is a common way cross-skill onboarding quietly fails; it needs the same structure as the primary skill, just smaller in scope.
• Compressing the plan under limited training resources by cutting phase 1 short (rushing into real ownership before the environment and codebase are understood) trades a faster-looking ramp for more review overhead and rework later.

Direct answer

Match the pricing model to how predictable and interruptible the workload is: pay full price (on-demand) for unpredictable or short-lived work, commit to a discount (a reserved instance or savings plan) for the steady baseline you know you'll run for a year or more, and reach for the deep spot or preemptible discount only when the work can be killed and resumed without harm. Real workloads mix all three: a steady floor on committed pricing, a growth buffer on on-demand, and interruption-tolerant batch work on spot.

Structured elaboration

When each fits

Mixing for one workload

Split the workload into a steady floor, a variable middle, and interruption-tolerant batch, and price each piece with the model that fits it, rather than choosing one model for the whole workload.

Worked example (illustrative unit prices, not vendor-quoted rates)

Assume on-demand costs $0.10 per instance-hour, a committed baseline (reserved instance or savings plan) costs $0.06 per instance-hour (40% off), and spot averages $0.03 per instance-hour (70% off, an illustrative average across reclaim risk).

Baseline: 20 instances running 24/7 (730 hours/month):

Variable batch work: 10 additional instances running 8 hours/day on average (240 hours/month) on spot:

Blended monthly cost with the mix versus all on-demand:

Trade-offs & pitfalls

• Never put a single-master, stateful database on spot; there is no fallback for the primary if it is reclaimed mid-write.
• Overcommitting reserved or savings capacity beyond your actual steady floor turns a discount into waste; size the commitment to your measured baseline, not your hoped-for growth.
• Spot needs operational work to be safe: checkpointing, diversified instance types and zones so a single reclaim wave doesn't take out your whole batch fleet, and an automatic fallback to on-demand.
• Managing multiple pricing commitments has its own administrative overhead (tracking utilization, renewal timing); factor that into whether the discount is worth it at small scale.

The idea

Dynamic programming (DP) is a way to solve a problem by writing the answer for a big input in terms of answers for smaller inputs, and then computing each smaller answer only once and reusing it. Plain recursion often recomputes the same small answers over and over; DP stores them so the work is done once per distinct subproblem instead of once per path that reaches it.

The two properties

1. Optimal substructure. The best answer to the whole problem can be built from the best answers to smaller subproblems. Take coins of value 1, 3 and 4 and ask for the fewest coins that make 6. The best answer is one coin plus the best answer for a smaller amount: the best way to make 6 is a 3-coin plus the best way to make 3 (a single 3-coin), so 2 coins. Without this property, solving the pieces optimally does not give an optimal whole.
2. Overlapping subproblems. The same smaller problem is needed many times by different branches of the recursion. Without this property there is nothing to reuse, and storing results costs memory for no gain (merge sort has the first property and not this one: its subproblems are disjoint halves).

Both are needed: substructure makes the recurrence correct, and overlap makes the table worth building.

A string example: can the text be split into words?

Given a string and a set of words, can the string be written as a concatenation of the words? State: ok[i] is True when the first i characters can be split into words. Transition: ok[i] is True when some word w ends exactly at position i and ok[i - len(w)] is True. Base case: ok[0] is True (the empty prefix). Order: i ascending, since each cell reads smaller indices. Answer: ok[len(s)]. The run prints the filled array for leetcode with the words leet and code: ok[4] and ok[8] are True.

A numeric optimization example: 0/1 knapsack

Items have a weight and a value; each is taken at most once; the total weight must stay within a capacity; maximize the total value. State: dp[i][c] is the best value using only the first i items with capacity c. Transition: dp[i][c] = max(dp[i-1][c], dp[i-1][c - w_i] + v_i) (leave item i, or take it if it fits). Base: row 0 is all zeros (no items, no value). Order: rows ascending, so row i reads row i-1. Answer: dp[n][capacity]. For weights 1, 3, 4, 5, values 1, 4, 5, 7 and capacity 7 the printed table ends at 9 (items of weight 3 and 4).

One cell by hand: the row item 3 (the first three items, with weight 4 and value 5 for item 3) at capacity 7 is 9. Leaving item 3 keeps the cell above, dp[2][7] = 5. Taking it spends 4 of the 7 units, leaving capacity 3, whose best with the first two items is dp[2][3] = 4, and adds its value 5, giving 4 + 5 = 9. The larger of 5 and 9 is 9.

How to spot a DP problem in a new question

• The question has the right shape. It asks for a count, a maximum or minimum, or a yes or no, over many ways of making a sequence of choices (a split, a take-or-skip, a path).
• The future depends on a small summary of the past. After some choices are made, what matters about them fits in a few numbers (position i, capacity left). That summary is the state.
• The brute force repeats itself. Write the plain recursion and count how many calls there are against how many distinct argument values it sees. For the word split on a repeated n times followed by b, with words a and aa, the run prints 1 call at n = 0, 2 at n = 1, 4 at n = 2 and 28,656 calls at n = 20, against only 21 distinct positions. For the knapsack with n items of weight 1, calls are 2^(n+1) - 1 (1, 3, 7, 15 for n = 0 to 3 and 2,047 at n = 10) against 66 distinct (item, room) pairs at n = 10. A large gap between calls and distinct arguments is the overlap.
• A greedy rule fails on a small example. With coins 1, 3 and 4 and a target of 6, taking the biggest coin first uses 3 coins (4 + 1 + 1), while the best is 2 (3 + 3). When a locally best choice can be wrong, the problem usually needs DP.

The code

import re
import random
from itertools import combinations

# ---------- Sequence example: can a string be split into dictionary words? ----------
word_calls = 0
seen_positions = set()


def can_split_brute(s, words, i=0):
    """Plain recursion on 'can s[i:] be split?'. No caching."""
    global word_calls
    word_calls += 1
    seen_positions.add(i)
    if i == len(s):
        return True
    ok = False
    for w in words:                       # no early exit, so every branch is explored
        if s.startswith(w, i) and can_split_brute(s, words, i + len(w)):
            ok = True
    return ok


def can_split_table(s, words):
    """ok[i] is True when the first i characters of s can be split into words."""
    ok = [False] * (len(s) + 1)
    ok[0] = True
    for i in range(1, len(s) + 1):
        # w ends at position i when s[0:i] ends with w; then the prefix before it must also split
        ok[i] = any(ok[i - len(w)] for w in words if i >= len(w) and s.endswith(w, 0, i))
    return ok


# ---------- Numeric optimization example: 0/1 knapsack ----------
knap_calls = 0
knap_states = set()


def knap_brute(weights, values, i=0, room=0):
    """Best value using items i.. with `room` capacity left. Take it or leave it."""
    global knap_calls
    knap_calls += 1
    knap_states.add((i, room))
    if i == len(weights):
        return 0
    best = knap_brute(weights, values, i + 1, room)
    if weights[i] <= room:
        best = max(best, values[i] + knap_brute(weights, values, i + 1, room - weights[i]))
    return best


def knap_table(weights, values, capacity):
    """dp[i][c] = best value using only the first i items with capacity c."""
    dp = [[0] * (capacity + 1) for _ in range(len(weights) + 1)]
    for i in range(1, len(weights) + 1):
        for c in range(capacity + 1):
            dp[i][c] = dp[i - 1][c]
            if weights[i - 1] <= c:
                dp[i][c] = max(dp[i][c], dp[i - 1][c - weights[i - 1]] + values[i - 1])
    return dp


def knap_reference(weights, values, capacity):
    best = 0
    for k in range(len(weights) + 1):
        for pick in combinations(range(len(weights)), k):
            if sum(weights[i] for i in pick) <= capacity:
                best = max(best, sum(values[i] for i in pick))
    return best


# ---------- A greedy rule that looks right and is not ----------
def coins_greedy(coins, target):
    count = 0
    for c in sorted(coins, reverse=True):
        count += target // c
        target %= c
    return count if target == 0 else None


def coins_table(coins, target):
    inf = float("inf")
    best = [0] + [inf] * target
    for t in range(1, target + 1):
        for c in coins:
            if c <= t and best[t - c] + 1 < best[t]:
                best[t] = best[t - c] + 1
    return best[target]


# 1. Overlapping subproblems, measured: calls versus distinct arguments.
print("word split, s = 'a' * n + 'b', words {a, aa}")
print("n   calls  distinct positions")
for n in [0, 1, 2, 10, 20]:
    word_calls = 0
    seen_positions.clear()
    can_split_brute("a" * n + "b", ["a", "aa"])
    print(f"{n:<3} {word_calls:<6} {len(seen_positions)}")

print("\nknapsack, n items of weight 1 and value 1, capacity 100, so every subset fits")
print("n   calls  bound 2^(n+1)-1  distinct (item, room)")
for n in [0, 1, 2, 3, 10]:
    knap_calls = 0
    knap_states.clear()
    knap_brute([1] * n, [1] * n, 0, 100)
    assert knap_calls <= 2 ** (n + 1) - 1
    print(f"{n:<3} {knap_calls:<6} {2 ** (n + 1) - 1:<17} {len(knap_states)}")

# 2. Filled tables for small inputs.
s, words = "leetcode", ["leet", "code"]
ok = can_split_table(s, words)
print("\nsplit 'leetcode' with {leet, code}")
for i, flag in enumerate(ok):
    print(f"  ok[{i}] ({s[:i]!r}): {flag}")

w, v, cap = [1, 3, 4, 5], [1, 4, 5, 7], 7
dp = knap_table(w, v, cap)
print("\nknapsack weights", w, "values", v, "capacity", cap)
print("capacity: " + " ".join(str(c) for c in range(cap + 1)))
for i, row in enumerate(dp):
    label = "no items" if i == 0 else f"item {i}  "
    print(f"{label:<9} " + " ".join(str(x) for x in row))
print("best value:", dp[-1][-1])

# 3. Both DPs agree with an independent reference on seeded random inputs.
rng = random.Random(4)
for _ in range(300):
    words = rng.sample(["a", "ab", "b", "ba", "aab"], rng.randint(1, 4))
    text = "".join(rng.choice("ab") for _ in range(rng.randint(0, 9)))
    pattern = "(?:" + "|".join(words) + ")*"   # regex "zero or more of these words", an independent reference
    assert can_split_table(text, words)[-1] == bool(re.fullmatch(pattern, text))
    word_calls = 0
    assert can_split_brute(text, words) == can_split_table(text, words)[-1]
    n = rng.randint(0, 7)
    ws = [rng.randint(1, 6) for _ in range(n)]
    vs = [rng.randint(0, 9) for _ in range(n)]
    c = rng.randint(0, 12)
    assert knap_table(ws, vs, c)[-1][-1] == knap_reference(ws, vs, c)
print("\n300 random word-split and 300 random knapsack cases match their references")

# 4. Greedy versus DP on coins.
print("\ncoins {1, 3, 4}, target 6: greedy", coins_greedy([1, 3, 4], 6),
      "coins, DP", coins_table([1, 3, 4], 6), "coins")

Output

word split, s = 'a' * n + 'b', words {a, aa}
n   calls  distinct positions
0   1      1
1   2      2
2   4      3
10  232    11
20  28656  21

knapsack, n items of weight 1 and value 1, capacity 100, so every subset fits
n   calls  bound 2^(n+1)-1  distinct (item, room)
0   1      1                 1
1   3      3                 3
2   7      7                 6
3   15     15                10
10  2047   2047              66

split 'leetcode' with {leet, code}
  ok[0] (''): True
  ok[1] ('l'): False
  ok[2] ('le'): False
  ok[3] ('lee'): False
  ok[4] ('leet'): True
  ok[5] ('leetc'): False
  ok[6] ('leetco'): False
  ok[7] ('leetcod'): False
  ok[8] ('leetcode'): True

knapsack weights [1, 3, 4, 5] values [1, 4, 5, 7] capacity 7
capacity: 0 1 2 3 4 5 6 7
no items  0 0 0 0 0 0 0 0
item 1    0 1 1 1 1 1 1 1
item 2    0 1 1 4 5 5 5 5
item 3    0 1 1 4 5 6 6 9
item 4    0 1 1 4 5 7 8 9
best value: 9

300 random word-split and 300 random knapsack cases match their references

coins {1, 3, 4}, target 6: greedy 3 coins, DP 2 coins

Complexity

States times work per state. The word split has n + 1 states and each tries every word, so O(n * W * L) for W words of length up to L (the endswith comparison costs up to L). The knapsack has (n + 1)(capacity + 1) states at O(1) each, so O(n * capacity) time and the same space (reducible to O(capacity) by keeping a single row). That is pseudo-polynomial, meaning polynomial in the numeric value of the capacity rather than in the number of bits needed to write it.

Common mistakes

• Choosing a state that leaves out something the future needs (for the knapsack, forgetting the remaining capacity), so two different situations share one cell.
• Forgetting the base case for the empty prefix: nothing can then start a split, because every cell reads from an earlier cell that is itself unset.
• Reaching for DP when the problem has substructure but no overlap (divide and conquer), or when a proven greedy rule exists.

Running the code

Save the listing as dp_intro.py and run python dp_intro.py (Python 3.12, standard library only).

Direct answer

Prefer Union-Find when connectivity needs to be queried repeatedly against a growing (insert-only) graph, or when the graph arrives as a stream of edges that must never all be held in memory at once as an adjacency structure. Prefer DFS/BFS when you need the graph traversed only once (a single connected-components pass), when you need the actual member list or shape of each component (not just "are these two connected"), or when the graph already needs to be materialized in memory for other reasons anyway. Both give the same $O(V+E)$-class total cost for a one-shot components computation; the real difference is in incremental behavior and memory footprint, not raw speed for a single full pass.

Structured elaboration

Practical performance on large sparse graphs. For a single, one-time "compute all components now" pass, DFS/BFS and Union-Find are asymptotically equivalent: $O(V+E)$ for the traversal versus $O(E \cdot \alpha(n)) \approx O(E)$ for Union-Find (since $\alpha(n) \le 4$ in practice). On a sparse graph ($E = O(V)$, the common real-world case for social graphs, service topologies, and dependency graphs), both are effectively linear in $V$. The practical performance difference shows up in constant factors and access pattern: DFS/BFS needs random access into an adjacency structure (cache-friendly if built as compressed sparse row, less so as a hash-map-backed adjacency list), while Union-Find only ever touches two flat arrays indexed directly by node ID, which tends to have excellent cache locality regardless of how the edges happen to be ordered.

Memory.

Union-Find's memory advantage is specifically that it never needs random access back into the edge set once an edge has been processed; DFS/BFS inherently needs to revisit a node's neighbor list every time that node is reached, so the adjacency structure must stay resident for the traversal's duration.

Incremental update implications. This is the sharpest practical difference. Adding a new edge to an already-computed Union-Find result is cheap: one union call, $O(\alpha(n))$ amortized, no need to touch anything else. Adding a new edge after a DFS/BFS-computed component list, by contrast, requires either a fresh traversal from one of the edge's endpoints to merge two existing components (cheap if the components were tracked as explicit sets you can merge, expensive if you would naively recompute everything), or accepting that the previously-computed component list is now stale. Neither structure handles edge REMOVAL cheaply: DFS/BFS needs a full re-traversal of the affected component to determine if it split, and Union-Find has no "un-union" operation at all, so both degrade to "rebuild from scratch" (or a much more complex dedicated dynamic-connectivity structure) under deletions.

Worked example

Consider a service-dependency graph with 50,000 services (nodes) and 120,000 dependency edges (sparse, average out-degree 2.4), where the graph is rebuilt from a fresh edge list once per minute by a monitoring job, and additionally single new edges arrive in near-real-time as services register new dependencies between rebuilds.

• The once-per-minute full rebuild is a one-shot computation over the complete edge list either way: a DFS/BFS pass costs $O(V+E) = O(50{,}000 + 120{,}000) = O(170{,}000)$ node-and-edge touches; a Union-Find pass over the same 120,000 edges costs $O(E \cdot \alpha(n)) \approx O(120{,}000 \times 4) = O(480{,}000)$ amortized array operations, which is a larger constant-factor count of individual operations but each operation is a simple array read/write rather than a hash-map or adjacency-list lookup, so in practice the two approaches land in a similar wall-clock ballpark for this size; the choice would come down to which representation the rest of the monitoring job already needs (if it needs an adjacency structure for other purposes anyway, DFS/BFS reuses it for free).
• The near-real-time single-edge updates between rebuilds are where the two approaches diverge sharply: applying one new edge to an already-built Union-Find structure is a single $O(\alpha(n))$ union call, whereas correctly updating a DFS/BFS-derived component list for one new edge (without a full re-traversal) requires the monitoring job to have retained enough bookkeeping to merge two component records directly, which is realistically only clean to do if that bookkeeping itself amounts to reimplementing a Union-Find-like merge step on top of the traversal result.

Trade-offs and pitfalls

• Common mistake: treating this as purely an asymptotic-complexity question. For a single full pass, both are effectively $O(V+E)$-class; the decision in practice is almost always about incremental-update patterns and what auxiliary information (member lists vs a fast connectivity check) the downstream consumer actually needs, not about which is "faster."
• Common mistake: assuming Union-Find handles deletions just because it handles insertions cheaply. The asymmetry (cheap merge, no cheap split) is the single most important practical trade-off and the one most likely to bite a design that assumed both operations were equally supported.
• If the downstream need is "list every member of every component," DFS/BFS gives that as a direct byproduct of the traversal; Union-Find gives it only after an additional $O(V)$ grouping pass over all nodes by final root, a cost that is easy to forget when comparing the two approaches only on their headline complexity.
• On a graph that is already required to be memory-resident as an adjacency structure for other parts of the system (routing, querying neighbors, rendering the topology), DFS/BFS effectively costs nothing extra in memory, since it reuses that existing structure; Union-Find would then be introducing a second, redundant representation purely to get a connectivity answer.

Direct answer
Two practical cost-reduction moves: use fewer, larger outer folds (or a single held-out outer split, sacrificing some estimate stability for a big compute saving) combined with a cheaper inner search strategy (random or Bayesian search over a small budget instead of an exhaustive grid), and reuse a fast proxy model or a smaller data subsample for the inner search before confirming the winning configuration on the full data.

Structured elaboration
Nested CV's cost scales roughly as (outer folds) x (inner folds) x (configurations tried), so any lever that shrinks one of those three factors directly reduces total cost. Reducing outer folds from 10 to 5 (or even using a single outer train/test split rather than K-fold) cuts outer-loop cost proportionally, at the cost of a less stable outer estimate (fewer independent test-fold measurements to average). Reducing configurations tried, by switching the inner search from exhaustive grid to a bounded random or Bayesian search, is often the bigger lever since grid size grows combinatorially with the number of hyperparameters, while random/Bayesian search lets you fix a compute budget (a trial count) independent of how many hyperparameters you're tuning.
A complementary trick: run the inner search on a representative SUBSAMPLE of the outer-training data (say 20-30%) to find a promising region of the hyperparameter space cheaply, then do a final, smaller confirming search near that region using the full data, rather than running the expensive full-data inner search from scratch at every outer fold.

Worked example
Going from 10 outer folds x 5 inner folds x a 100-point grid (5,000 total fits) to 5 outer folds x 4 inner folds x a 20-trial random search (400 total fits) is roughly a 12x compute reduction, while still preserving genuine outer-loop separation between tuning and evaluation, the core property that makes nested CV unbiased.

Trade-offs & pitfalls
Cutting outer folds to just one train/test split removes the ability to report a confidence interval around your generalization estimate from fold-to-fold variance; if you need that uncertainty quantification for a business decision, don't cut this lever below 3-5 outer folds even under compute pressure.

Direct answer

Terms used below. Autograd is PyTorch's automatic differentiation: during the forward pass it records the operations and keeps ("saves") the tensors it will need to compute gradients in the backward pass. Activation checkpointing (recomputation) means not keeping some of them and recomputing them during the backward pass. fp16 is a 16-bit floating point number, 2 bytes per element, and mixed precision means storing most tensors in 16 bits and a few in 32 bits. A GiB is 2^30 bytes. A KV cache holds the keys and values of earlier tokens while a model generates text one token at a time. A tiled, fused kernel (the idea behind FlashAttention-style implementations) is a GPU routine that computes attention in small blocks that fit in fast on-chip memory, so it never writes the full N by N matrix to main memory.

Per layer, with N tokens, width d, H heads and feed-forward width 4d (batch size 1):

• FLOPs (floating-point operations, counting a multiply-add as 2): 24 N d^2 + 4 N^2 d. The first term is the dense projections and feed-forward network, linear in N. The second is attention's score computation and weighted sum, quadratic in N. Multiply by L for the stack.
• Activation memory kept for the backward pass, in elements: N(16d + 4) + H N^2. The first term is linear in N; the second is the attention score matrices, one N by N matrix per head.
• Attention score matrices: H N^2 elements per layer, so L H N^2 for the model.
• Where attention overtakes the feed-forward network: in compute, at N = 4d against the feed-forward network alone and N = 6d against all the dense matrix multiplications. In memory, at N = 16d/H, which is earlier than 4d whenever H is more than 4 (as in the 16-head example below), because every head owns its own N by N matrix.

Derivation of the FLOPs

A matrix multiplication of an (a by b) matrix with a (b by c) matrix costs 2abc FLOPs. Per layer, counting only matrix multiplications (layer norm, softmax and the activation are elementwise and add terms linear in N or in N squared with tiny constants):

Note that H cancels: splitting d into H heads of size d/H leaves the FLOPs unchanged. For a general d_ff the total is 2N(4d^2 + 2 d d_ff) + 4N^2 d, the form the program uses.

Where attention overtakes. The quadratic term 4N^2 d equals the feed-forward term 16 N d^2 when N = 4d, and equals all dense terms 24 N d^2 when N = 6d. For d = 1024 that is N = 4096 and N = 6144. The three crossovers compare different things: N = 4d compares the attention score work with the feed-forward network alone, N = 6d compares it with all the other matrix multiplications together, and the memory crossover below compares storage, not operations. The backward pass costs about twice the forward pass because each forward matrix multiplication needs two in the backward pass of the same size, one for the gradient with respect to its input and one for the gradient with respect to its weights (the program counts 2.0 on the test block), so training costs about 3 times the forward FLOPs.

A tiny case to trace by hand: N = 4, d = 8, H = 2, feed-forward width 32. The projections cost 8 x 4 x 64 = 2,048, the scores 2 x 16 x 8 = 256, the weighted sum 256 and the feed-forward network 4 x 4 x 8 x 32 = 4,096, for 6,656, which equals 24 x 4 x 64 + 4 x 16 x 8 = 6,144 + 512. The first lines of the printed output show these numbers.

Derivation of the activation memory

"Activation memory" here means the tensors the framework must keep from the forward pass to compute gradients (not counting parameters, their gradients or optimizer states, which are separate and left out). For one pre-norm layer with attention written as plain matrix multiplications:

• 8 tensors of size N by d (each is kept because some backward step needs it: a layer norm's backward needs its input; a linear layer's weight gradient needs the input that fed it; the score product's backward needs q and k, and the weighted-sum product's backward needs v): the input to the first layer norm, its output (which feeds the Q, K, V projections), q, k, v, the attention output before the output projection, the residual stream after attention (input to the second layer norm), and the second layer norm's output (input to the feed-forward network).
• 2 tensors of size N by d_ff: the first feed-forward linear's output (needed by the activation's backward) and the activation's output (needed by the second linear). With d_ff = 4d this is 8Nd.
• 4N numbers: the mean and inverse standard deviation saved by each of the two layer norms.
• H N^2 numbers: the softmax output, the attention probabilities. The softmax backward and the product with v both need it. The pre-softmax logits are not kept, because the softmax backward is computed from its output.

Setting the score term equal to the linear term gives the memory crossover: H N^2 = N(16d + 4), which for large d is about 16 N d, so N = 16d/H (for d = 1024 and H = 16, N = 1024).

Total per layer: N(8d + 2 d_ff + 4) + H N^2, which is N(16d + 4) + H N^2 for d_ff = 4d. In plain terms, the program works like this: saved_tensors_hooks calls a function each time autograd saves a tensor for the backward pass; the function records its memory address (data_ptr) and element count, skips the model's own weights, and counts a tensor once even if several saved items are views of the same memory ("distinct storage"). FlopCounterMode counts the floating-point operations of each matrix multiplication as the forward pass runs. Measured equals formula because the formulas list exactly the tensors the backward pass needs. The program uses a manual attention so the N by N probabilities are real tensors.

import torch, torch.nn as nn
from torch.utils.flop_counter import FlopCounterMode

torch.set_num_threads(1)

class Block(nn.Module):
    """Pre-norm block, manual attention so the N x N score tensors are real tensors."""
    def __init__(s, d, H, ff):
        super().__init__()
        s.H = H
        s.ln1, s.ln2 = nn.LayerNorm(d), nn.LayerNorm(d)
        s.q, s.k, s.v, s.o = (nn.Linear(d, d) for _ in range(4))
        s.f1, s.f2 = nn.Linear(d, ff), nn.Linear(ff, d)
    def forward(s, x):
        B, N, d = x.shape
        h = s.ln1(x)
        sp = lambda t: t.view(B, N, s.H, d // s.H).transpose(1, 2)       # (B, H, N, d/H)
        q, k, v = sp(s.q(h)), sp(s.k(h)), sp(s.v(h))
        p = torch.softmax(q @ k.transpose(-1, -2) / (d // s.H) ** 0.5, dim=-1)   # (B, H, N, N)
        x = x + s.o((p @ v).transpose(1, 2).reshape(B, N, d))
        return x + s.f2(torch.nn.functional.gelu(s.f1(s.ln2(x))))

def measure(N, d=64, H=4, ff=256):
    blk = Block(d, H, ff)
    x = torch.randn(1, N, d)
    params = {p.data_ptr() for p in blk.parameters()}
    seen = {}
    def pack(t):
        if t.data_ptr() not in params: seen[t.data_ptr()] = t.numel()
        return t
    with torch.autograd.graph.saved_tensors_hooks(pack, lambda t: t):
        with FlopCounterMode(display=False) as fc:
            y = blk(x)
    return sum(seen.values()), fc.get_total_flops()

# tiny case, small enough to trace by hand: N = 4 tokens, d = 8, H = 2 heads, feed-forward width 32
n0, d0, h0, f0 = 4, 8, 2, 32
parts = {"Q,K,V,O projections": 8 * n0 * d0 * d0, "scores QK^T": 2 * n0 * n0 * d0,
         "weighted sum with V": 2 * n0 * n0 * d0, "feed-forward": 4 * n0 * d0 * f0}
for k, v in parts.items():
    print(f"{k:<22}{v:>6} FLOPs")
print("sum of parts:", sum(parts.values()), " formula 24 N d^2 + 4 N^2 d:", 24 * n0 * d0 * d0 + 4 * n0 * n0 * d0)
lin, att = n0 * (8 * d0 + 2 * f0 + 4), h0 * n0 * n0
print("saved elements by formula: linear part", lin, "+ score part", att, "=", lin + att)
m0, fl0 = measure(n0, d0, h0, f0)
print("measured on a real block: saved elements", m0, " forward FLOPs", fl0)
print()

d, H, ff = 64, 4, 256
print("N | saved elements: measured, formula | forward FLOPs: measured, formula")
for N in (32, 128, 512):
    m, f = measure(N)
    mem = N * (8 * d + 2 * ff + 4) + H * N * N          # per layer, one sequence
    fl = 2 * N * (4 * d * d + 2 * d * ff) + 4 * N * N * d
    print(N, "|", m, mem, "|", f, fl)

# forward + backward FLOPs of the same block, counted
blk, x = Block(d, H, ff), torch.randn(1, 128, d)
with FlopCounterMode(display=False) as fwd:
    y = blk(x)
with FlopCounterMode(display=False) as both:
    blk(x).sum().backward()
print("backward / forward FLOPs:", round((both.get_total_flops() - fwd.get_total_flops()) / fwd.get_total_flops(), 3))

# scale up with the formulas: L=24, d=1024, H=16, ff=4d, fp16 (2 bytes)
L, D, HH, F, bytes_ = 24, 1024, 16, 4096, 2
for N in (1024, 4096, 16384):
    attn = L * HH * N * N * bytes_ / 2**30
    rest = L * N * (8 * D + 2 * F + 4) * bytes_ / 2**30
    sc = 4 * N * N * D; ffn = 4 * N * D * F
    print(f"N={N:6d} score matrices {attn:8.2f} GiB | everything else {rest:6.2f} GiB | score FLOPs / FFN FLOPs {sc / ffn:.2f}")
print("FLOP crossover vs FFN alone N = 4d =", 4 * D, "; vs all dense matmuls N = 6d =", 6 * D)
print("memory crossover N = 16 d / H =", 16 * D // HH)

Output:

Q,K,V,O projections     2048 FLOPs
scores QK^T              256 FLOPs
weighted sum with V      256 FLOPs
feed-forward            4096 FLOPs
sum of parts: 6656  formula 24 N d^2 + 4 N^2 d: 6656
saved elements by formula: linear part 528 + score part 32 = 560
measured on a real block: saved elements 560  forward FLOPs 6656

N | saved elements: measured, formula | forward FLOPs: measured, formula
32 | 36992 36992 | 3407872 3407872
128 | 197120 197120 | 16777216 16777216
512 | 1574912 1574912 | 117440512 117440512
backward / forward FLOPs: 2.0
N=  1024 score matrices     0.75 GiB | everything else   0.75 GiB | score FLOPs / FFN FLOPs 0.25
N=  4096 score matrices    12.00 GiB | everything else   3.00 GiB | score FLOPs / FFN FLOPs 1.00
N= 16384 score matrices   192.00 GiB | everything else  12.00 GiB | score FLOPs / FFN FLOPs 4.00
FLOP crossover vs FFN alone N = 4d = 4096 ; vs all dense matmuls N = 6d = 6144
memory crossover N = 16 d / H = 1024

The measured and formula columns agree exactly at all three lengths, for both quantities. The program uses a small block (d = 64, 4 heads, d_ff = 256) in float32 with no dropout; the formulas then give the 24-layer, d = 1024, 16-head, fp16 estimates in the last block of lines.

The attention score matrices in forward and backward

In the forward pass of a plain implementation, the logits S = QK^T / sqrt(d/H) (H matrices of N by N) and the probabilities P = softmax(S) both exist, though only P is kept. In the backward pass the gradient with respect to P, written dP (the derivative of the loss with respect to each probability; it equals the gradient of the attention output times v transposed), and the gradient with respect to S, written dS (from the softmax backward), are each another H N^2 numbers. All four tensors, S, P, dP and dS, have shape (batch, H, N, N); for N = 4 and H = 2 each has 32 elements, for N = 4096 and H = 16 each has about 268 million. So the peak for one layer's attention can reach several score-sized tensors at once (S, P, dP, dS) depending on how early buffers are freed, while the part kept across layers is only P. Optimizer states and parameter gradients are outside this accounting.

The printed table shows the scale for L = 24, d = 1024, H = 16, fp16 (2 bytes per number). At N = 4096 the saved score matrices take 12.00 GiB against 3.00 GiB for everything else; at N = 16384 they take 192.00 GiB against 12.00 GiB. At N = 1024 the two are equal at 0.75 GiB, matching the memory crossover N = 16d/H = 1024. This table assumes the score matrices are stored. With a tiled, fused kernel they are not stored, so a practitioner using one sees memory dominated by the "everything else" column and growing linearly in N. In FLOPs the score and weighted-sum work equals the feed-forward work at N = 4096 (ratio 1.00) and is four times it at N = 16384.

What this means and where it fails

• With many heads (H above 4, as in the 16-head example), memory hits the quadratic wall before compute does, because of the factor H. That is why kernels that compute attention in tiles and never write the N by N matrix to memory (the tiled, fused approach used by FlashAttention-style kernels) cut the stored term to linear in N at the price of recomputation in the backward pass; the forward FLOPs are unchanged and the backward pass does somewhat more work because it recomputes the scores.
• The formulas cover matrix multiplications only. Elementwise operations, embeddings, the output layer (vocabulary projection) and the KV cache during generation are separate line items.
• Activation recomputation (checkpointing) trades the linear term for extra forward FLOPs; mixed precision changes bytes per element but not element counts.

Common mistakes

• Counting a multiply-add as 1 FLOP in one place and 2 in another.
• Writing the score cost as N^2 d / H (forgetting the H heads) or as N^2 only (forgetting the contraction over d/H).
• Using one head's N by N matrix as the attention memory instead of H of them.
• Saying "attention is O(N^2), so it dominates": at short N the feed-forward and projections dominate, and the crossover depends on d.

Running the code

Install: pip install torch. Save the program to a file and run it with python. It uses random inputs, so the counts do not depend on a seed.