Amazon AI Engineer Interview Preparation Guide - Staff Level

Direct answer

Flag a row as an outlier when its value falls below $Q1 - k\cdot IQR$ or above $Q3 + k\cdot IQR$, computing Q1, Q3, and IQR from the column while gracefully skipping any missing values rather than letting them break the calculation.

Structured elaboration

import numpy as np
import pandas as pd

def detect_outliers_iqr(df, column, k=1.5):
    """Return the index labels of rows considered outliers via the IQR method."""
    values = pd.to_numeric(df[column], errors="coerce")   # tolerate numeric-looking strings, NaN otherwise
    valid = values.dropna()
    q1, q3 = valid.quantile(0.25), valid.quantile(0.75)
    iqr = q3 - q1
    lower, upper = q1 - k * iqr, q3 + k * iqr
    is_outlier = (values < lower) | (values > upper)   # NaN comparisons are False, so NaNs are never flagged
    return df.index[is_outlier.fillna(False)]

df = pd.DataFrame({"amount": [10, 12, 11, 13, 9, 500, None, 14]})
outlier_idx = detect_outliers_iqr(df, "amount", k=1.5)
print(outlier_idx.tolist())

Handling NaNs gracefully: computing the quantiles on valid = values.dropna() means missing values never distort Q1/Q3/IQR, and the final boolean mask's comparisons against NaN naturally evaluate to False (never flagged as an outlier), with an explicit .fillna(False) as a defensive safeguard in case a comparison chain produces something other than a clean boolean.

Worked example

Running the code above on [10, 12, 11, 13, 9, 500, None, 14], Q1 and Q3 are computed from the 7 valid values, giving a fence that easily flags 500 while leaving None untouched (not flagged, correctly excluded rather than erroring); the function returns the index position of the 500 row only.

Trade-offs and pitfalls

The IQR method's known limitation on a skewed distribution: because Q1 and Q3 are computed from the RAW (unlogged, untransformed) values, a heavily right-skewed feature can have its upper fence pulled disproportionately high by the skew itself, causing the method to under-flag genuinely unusual values on the long tail; for such features, applying a skew-reducing transform (log, Box-Cox) before running the IQR check, or using a method less sensitive to skew, often catches outliers this straightforward version misses.

Direct answer: A low-latency, fault-tolerant serving architecture for reward models and RLHF-trained policy servers needs tiered GPU pools split by latency versus throughput needs that scale independently based on real-time load, dynamic request batching with a latency-aware cap, a semantic cache for repeated or near-identical prompts, and a graceful-degradation path to a simpler fallback model when the primary pool is overloaded or unhealthy.

flowchart LR
    Req[Request] --> Router[Orchestrator/Router]
    Router --> Cache{Cache hit?}
    Cache -->|yes| Resp[Response]
    Cache -->|no| Batch[Dynamic Batcher]
    Batch --> Primary[Primary GPU pool: reward model + policy]
    Primary -->|healthy| Resp
    Primary -->|overloaded/unhealthy| Fallback[Fallback: instruction-tuned CPU pool]
    Fallback --> Resp

Structured elaboration:

• Request flow and pooling: an orchestrator in front of the model pools routes requests, enforces latency SLOs, and assembles batches; separate small-batch, low-latency GPU pods from large-batch, high-throughput GPU workers, since a single pool tuned for one goal tends to underperform on the other, and keep CPU-only fallback replicas of a simpler instruction-tuned model ready for degraded-mode serving.
• Autoscaling: since RLHF-trained policy and reward-model traffic is bursty, each tier's GPU pool should scale horizontally on its own signal, primarily queue depth and per-tier tail latency rather than raw request count alone, since request volume spikes unevenly across the low-latency and high-throughput lanes; use a cooldown-guarded scaling policy so a brief spike does not trigger a scale-up whose new replicas finish warming only after the spike has already passed, keep a floor of always-warm replicas per tier so autoscaling never scales a tier fully to zero (which would reintroduce a cold start on the very next burst), and scale the CPU-backed fallback pool independently from the primary GPU pools, since a primary-pool incident is often exactly when fallback demand spikes hardest.
• Batching: dynamic batching with a latency-aware cap (for example a maximum wait time in the tens of milliseconds, or a batch-size cap, whichever triggers first) balances GPU utilization against per-request latency; a heterogeneous batching strategy that fills throughput-oriented GPU micro-batches while still draining the low-latency lane immediately avoids one traffic pattern starving the other.
• Caching: an LRU or TTL cache keyed on a deterministic prompt hash, supplemented with an approximate-nearest-neighbor layer over embeddings to also serve near-identical (not just byte-identical) prompts from cache, cuts real compute for repeated queries, but the cache key design needs care, a cache hit on a semantically-similar-but-meaningfully-different prompt would silently serve a stale or incorrect response.
• Graceful degradation and cold start: if the primary GPU pool is overloaded or unhealthy, route to the simpler instruction-tuned fallback (adjusting generation parameters as needed) rather than failing the request outright, and clearly flag when a response came from the fallback path rather than the primary reward-and-policy pipeline; for rollouts of a new model version, pre-warm standby GPUs with the new checkpoint and run synthetic traffic to build warm kernels and populate the cache before shifting any real traffic, and validate correctness and latency in shadow mode before gradually shifting live traffic.
• Fault tolerance and observability: stateless frontends and replicated orchestrators so no single node is a hard dependency, health checks and circuit breakers between every tier, and end-to-end tracing (per-model latency, GPU utilization, batch sizes, queue depth) so a latency regression can be localized to a specific tier rather than investigated as a single opaque number.

Worked example: A concrete degradation sequence under load: the primary reward-model and policy GPU pool's queue depth crosses a threshold; the autoscaler first attempts to add replicas within its floor-and-ceiling bounds, and if new capacity cannot warm up fast enough to relieve the queue, the orchestrator's health check marks the pool degraded and begins routing new requests to the CPU-backed instruction-tuned fallback instead of queuing indefinitely behind the overloaded primary pool; once the primary pool's queue depth and latency recover below the threshold for a sustained window, routing shifts back automatically and the autoscaler scales the temporary extra fallback capacity back down; throughout this, responses served via the fallback are tagged so downstream analytics and any user-facing quality monitoring can account for the temporary quality difference rather than conflating it with the primary pipeline's normal behavior.

Trade-offs and pitfalls: Smaller micro-batches favor low latency but reduce achievable GPU throughput, which is why a tiered pooling strategy (rather than one batch-size setting for everything) is the practical resolution rather than picking a single compromise batch size for all traffic. Caching reduces compute cost meaningfully but only if the cache key design correctly distinguishes prompts that are similar in embedding space but different in a way that matters, an overly aggressive near-neighbor cache match is a subtle correctness bug, not just a performance tuning knob. Autoscaling on raw request count alone is a common mistake, since a small number of very long-running requests can saturate a GPU pool's queue and latency while request count looks unremarkable, scaling decisions need to be driven by queue depth and tail latency, the actual symptoms of overload, not a proxy metric that can look fine while the pool is genuinely struggling.

Direct answer

A feature store is a shared system that computes, versions, and serves the transformed inputs ("features") a model consumes, so the training pipeline and the live serving path read the exact same feature definitions instead of each team reimplementing them separately. Teams build one because ad hoc computation means the same transformation logic gets written twice (a batch job for training, application code for serving), and those two implementations drift apart over time with nobody noticing. It keeps training and serving consistent by having one canonical, registered transformation materialize into two stores, an offline store for historical training joins and an online store for low-latency point lookups at request time, instead of two independently maintained code paths.

Structured elaboration

Why not ad hoc:

• Duplication risk: a feature like "days since last purchase" gets implemented once in a training notebook and once in a serving service; different rounding, null handling, or time windows creep in between the two.
• No shared registry: two teams may build the same feature independently, slightly differently, with nobody able to tell the definitions have diverged.
• No lineage: without a central store, nobody can answer "which exact feature version did the model currently in production train on."

Offline vs online planes:

How consistency is achieved:

• A single registered feature transformation is executed once and fanned out to both stores, not maintained as two separate codebases.
• Point-in-time correctness: the offline store must answer "what was this feature's value at the time the training label was observed," not "what is its value now"; otherwise information from after the label leaks into training.
• Shared materialization job: one streaming or batch job writes to the online row and the offline partition at the same time, so the two stores never disagree on freshness semantics.
• Contract tests: sample a serving key and compare its online value against what the offline store would have produced at that same event time.

Worked example

Consider a feature "purchases in the last 7 days" for a user with purchases on day 1, day 3, day 9, and day 12. The training label for this user was observed on day 10. A correct, point-in-time join asks "what was this feature's value as of day 10," which counts purchases in the window [day 3, day 10): that is day 3 and day 9, a count of 2. A naive join that instead counts all of the customer's purchases up to whenever the pipeline happens to run (say day 20) would include the day-12 purchase as well, a count of 3, silently leaking a purchase that had not happened yet at label time into the training feature. This single miscounted purchase is training-serving skew in miniature: the model trains on a feature value ("3 recent purchases") that could never have existed at serving time for that label, and a feature store's point-in-time join discipline is precisely what prevents it.

Trade-offs and pitfalls

Running two stores (offline and online) is genuine operational cost, so a small team with only a handful of features and one consuming model may be better off with a well-tested shared library than a full feature store; the store earns its cost once several teams are reusing the same features and skew has already bitten someone. The most common pitfall is skipping the point-in-time join because it is harder to implement than "join on the most recent value," which quietly reintroduces skew even after a feature store is in place. A second pitfall is treating the online store's eventual consistency (a brief lag between an event happening and its feature updating) as if it were instantaneous, which matters most for features that are supposed to reflect very recent behavior.

Feature leakage: when a feature includes information that wouldn’t be available at prediction time or directly encodes the target.
Three examples:

• Data collection: timestamps include future labels — e.g., including 'cancellation_time' when predicting churn at signup.
• Feature engineering: aggregations computed using the full history including the target window — e.g., mean spend computed including post-label period.
• Labeling: label derived from features used as inputs — e.g., using a heuristic that uses a form field that’s also an input.
  Testing strategy to detect/prevent leaks:

1. Code review checklist: require reviewers to verify temporal validity and availability of each feature at scoring time.
2. Offline tests: simulate production scoring pipeline and assert all features are computable using only allowed data; run feature-timestamp validations.
3. Holdout experiments: train model with and without suspect features; if performance drops dramatically when removing them, investigate for leakage.
4. Cross-validation with time-based folds and forward-chaining CV for temporal data.
5. Automated unit tests in CI that enforce feature provenance metadata (source, last-updated time, allowed lag) and block deployment if provenance missing.

Direct answer

A p-value answers one narrow question: if there were truly no effect, how surprising would data like ours be? A small p-value, commonly under 0.05, means the pattern we saw would be unusual if nothing were really going on, so it counts as evidence worth acting on rather than random noise. Caution in one sentence: a p-value does not tell you the probability the effect is real, or how big or important it is, only how surprising the data would look under a "nothing is happening" assumption.

Structured elaboration

Explaining a statistics term to a non-technical stakeholder is less about simplifying the definition and more about choosing what to omit and what to guard against:

1. Pick one mental model and don't mix in a second. "How surprising the data would be if nothing were going on" is enough; don't also try to explain sampling distributions or the formal null hypothesis by name in the same breath.
2. State the caution as its own sentence, not a footnote. The most common misread is treating the p-value as "the probability we're right." Say plainly what it is not.
3. Separate statistical significance from size. A p-value can be tiny on an effect too small to matter, or the reverse: with a small sample, a real and sizeable effect can fail to look significant. That is where a confidence interval (the range the true effect probably falls in, not just whether it beat a cutoff) and margin of error earn their keep: they answer "how big, and how sure," a different question from "was this surprising."
4. Check understanding by asking the stakeholder to restate it in their own words. If they say "so it proves we're right," the caution hasn't landed yet.

The same "how surprising is this" frame extends past hypothesis tests. If a stakeholder asks how confident a model's predicted probabilities are, model calibration answers a related but different question: not "was this one result surprising" but "when the model says 70%, does that outcome actually happen about 70% of the time." And when a CFO worries a result is a fluke of a small sample, the honest reassurance isn't the p-value at all, it's showing the sample size and how much the estimate would tighten with more data, since a small or biased sample can produce a small p-value that still doesn't generalize.

Worked example

An A/B test on a new checkout flow comes back with a borderline p-value of 0.04. In the room:
"We ran the new checkout against the old one with about 20,000 customers on each side. The lift we saw would only happen by chance about 4 times out of 100 if the new flow actually made no difference, so this is likely real, but it's close to our usual cutoff, not a landslide. Before we roll it out everywhere, I'd want to see the confidence interval, the range of lift the data actually supports, because at this sample size a real-but-small effect and a borderline-noise effect can look similar on the p-value alone. If that range includes 'basically no change,' I'd want one more week of data before we call it, rather than lock in a decision off a single borderline number."
Caution restated: a p-value close to the cutoff isn't something you can round away in either direction, it's a signal to look at the interval and the trend, not settle the question on the spot.

Trade-offs and pitfalls

Dropping the caution sentence to keep the pitch upbeat is the most common mistake, and it's exactly what produces "the data proves it" overconfidence later. Overcorrecting the other way, hedging so hard that a genuinely strong result reads as shaky, costs credibility too. Borderline p-values, roughly 0.03 to 0.07, deserve more nuance than either "significant, ship it" or "not significant, ignore it": show the interval and the trend, not just the single number against the 0.05 line.

Direct answer

Use synchronous scoring only when the action on the score must happen before the transaction completes, such as blocking a fraudulent charge. If the action can wait (nightly review, retraining labels), batch scoring buys a heavier, more accurate model at a fraction of the cost. The deciding question: what does a delayed decision cost you, versus an always-on low-latency fleet?

Structured elaboration

Real-time: one hop in a hard end-to-end latency budget, needs near-zero-staleness features (a hot feature store: a system that serves the same precomputed input values, like a rolling transaction count, to the model at scoring time; "hot" means it's updated in near real time rather than nightly), and an always-on fleet sized for peak plus a fallback if the call times out.

Batch: runs on a schedule, so it can use a bigger, slower model on cheaper bursty compute, but adds a detection lag equal to the batch interval, and a partial failure is a silent under-score, not an outage.

Worked example

A 250ms authorize/decline budget, with auth, ledger, and notification already at 180ms:

$\text{fraud model budget} = 250\text{ms} - 180\text{ms} = 70\text{ms}$

Subtract 20ms network/serialization overhead:

$\text{compute budget} = 70\text{ms} - 20\text{ms} = 50\text{ms}$

That rules out heavy ensembles (combining predictions from several models, which multiplies the per-request compute cost) needing multiple feature joins (a feature store lookup that assembles several separate precomputed values into one input record, each join adding its own latency): illustratively, a 3-model ensemble with 2 extra feature joins might cost 90-120ms on its own, already over the 50ms budget, which is what "rules out" means here concretely. For batch: 10 million transactions overnight, 500 records per inference batch, 50ms per batch on one GPU:

$\text{batches} = \frac{10{,}000{,}000}{500} = 20{,}000, \quad \text{GPU time} = 20{,}000 \times 50\text{ms} \approx 16.7\text{ min}$

Across 4 GPUs, about 4 minutes wall time, no idle fleet cost during the day.

Trade-offs and pitfalls

Real-time buys speed but pays for peak capacity around the clock and widens the request's failure surface. Batch is cheaper but the fraud can complete before you act. A common mistake is defaulting to real-time without pricing the fleet against the actual cost of delay. The answer flips when either side of that equation moves: if fraud losses from a delayed decision start to dwarf the cost of an always-on fleet, or a cheaper real-time model becomes accurate enough to fit the leftover latency budget, you're pushed toward real-time; if the model needs more compute than the authorization path can spare, or false declines against legitimate customers become the bigger cost, you're pushed back toward batch.

What the interviewer probes next

Hybrid streaming or micro-batch designs, detecting a batch job that fails partway through, and how the answer shifts if a missed fraud case gets an order of magnitude costlier.

Direct answer

SageMaker, Vertex AI, and Azure ML all cover the same broad capability set: managed training, serving, and a registry/pipeline layer, but differ in ecosystem integration, portability, and how opinionated their pipeline abstractions are, which is usually the deciding factor over raw feature parity.

Structured elaboration

1. Managed training and hyperparameter tuning: all three offer managed training jobs with built-in hyperparameter-tuning services (SageMaker's Automatic Model Tuning, Vertex AI's Hyperparameter Tuning, Azure ML's HyperDrive): broadly comparable capability, differing mainly in configuration syntax and integration depth with each platform's other services.
2. Inference serving options: each supports serverless/on-demand inference, persistently-hosted real-time endpoints, and batch inference: again broadly comparable at a feature-checklist level, with real differences showing up in cold-start latency characteristics and autoscaling behavior under load, which matter more in practice than the checklist suggests.
3. Model registry and pipeline offerings: each has a native registry and a pipeline-orchestration capability (SageMaker Pipelines, Vertex AI Pipelines built on Kubeflow, Azure ML Pipelines): Vertex AI's being Kubeflow-based gives it a genuine portability edge if you might want to run similar pipeline definitions outside GCP later; SageMaker and Azure ML's pipeline systems are more tightly coupled to their respective ecosystems.
4. Key limitations pushing toward self-hosted: portability (all three managed offerings create real platform lock-in at the pipeline and tooling level, even though the underlying models themselves are usually portable); custom networking (highly specific VPC/network requirements can be easier to satisfy with self-hosted infrastructure you fully control); very custom GPU configurations (a managed platform's supported instance types may lag behind the newest hardware, or not offer the exact multi-GPU topology a specialized training job needs); and compliance (certain regulated environments have specific infrastructure requirements a managed platform's shared responsibility model may not cleanly satisfy without significant additional configuration).

Worked example

A concrete decision case: a team already deeply invested in GCP's data ecosystem (BigQuery, Dataflow) gets outsized integration value from Vertex AI specifically because of how tightly it connects to those existing services, an advantage that wouldn't transfer if the team evaluated SageMaker or Azure ML in isolation on feature checklists alone: the existing ecosystem investment is often the deciding factor in practice, more than any single platform capability being objectively superior.

Trade-offs & pitfalls

The trap in this kind of comparison is evaluating platforms on a feature checklist alone, since all three genuinely do cover the same broad capabilities at a surface level: the real differentiators are ecosystem fit (what else does your organization already run on this cloud), portability needs (how much does vendor lock-in actually matter for your situation), and operational specifics (autoscaling behavior, cold-start latency) that only show up under real production load, not in a documentation comparison.

Direct answer

I would write the assumption as a claim data could contradict (falsifiable: able to be shown wrong), set the stop bar before looking at anything, and use evidence already in hand (logs, tickets, past experiments) plus one light behavioral probe instead of calls. Two weeks is enough: days 1 to 2 for the claim and the bar, days 3 to 7 for existing data, days 8 to 12 for a probe if the data is not decisive, and days 13 to 14 to decide.

Elaboration

1. Make it falsifiable. "Users want better results" cannot fail. "Users who rewrite a query within 60 seconds do so because the first results were poor" can.
2. Set the bar from the size of the prize. Ask how large the affected group must be to justify the work, and write it down first. Otherwise any result gets argued into a yes.
3. Look at behavior you already have: logs, tickets, earlier experiments.
4. Probe if needed (a behavioral probe: a cheap test that watches what users do, not what they say): a fake-door button (a button for a feature that does not exist yet, to count clicks) or a small in-product prompt.
5. Decide in three zones: clearly above the bar, go; clearly below it, stop or redesign; between, run the probe.

Worked example (illustrative; retraining a search ranker)

Assumption: many users reformulate their query (retype it with different words) because the ranker (the model that orders search results) put the useful result too low, and retraining it (re-running its learning on fresh data) will fix it.

Bar: the pattern must appear in at least 10% of sessions. Reasoning: retraining will not rescue every case, because some reformulations come from vague wording or missing content that no ranking change fixes. Taking an optimistic guess that it fixes at best half, 10% of sessions becomes about 5% of sessions improved, the smallest gain the team judged worth about two months of work.

Test: sample 2,000 sessions from the logs. A session matches if the user reformulated within 60 seconds and then clicked a result that the first list had placed lower. Suppose 130 match.

How to read the margin: 130 matches in 2,000 sessions is one sample, and a different random 2,000 would give a slightly different rate. The margin says how far the true rate is plausibly from 6.5%. The 95% means that if you repeated this sampling many times, about 95 of 100 intervals built this way would contain the true rate. The 1.96 is the standard multiplier for 95%, and the square-root part is the typical wobble of a rate measured on 2,000 sessions (the normal approximation, a textbook shortcut that works well when there are many matches and many non-matches).

The whole interval is under 10%, so the stop holds even allowing for sampling noise, assuming the sample is random. I would stop, and also read 20 raw sessions to see what is going on, because logs show what happened, not why.

Pitfalls

• Choosing the bar after seeing the number.
• A narrow sample window, such as one holiday week, that hides the usual behavior.
• Treating "no calls" as "no evidence". Behavioral data is evidence, just thin on motive.

Direct answer

Developing independent judgment, not just correct outputs, requires repeated exposure to the same class of decision with you gradually receding from it, and requires the person to narrate their reasoning, not just report their choice. You know it's internalized, not just imitated, when their reasoning transfers to a situation you never coached them on directly, ideally one you weren't even present for.

How judgment gets built and verified

Coach the decision class, not the individual decision. A one-off answer to "should we do X" teaches them what to do this time. Judgment comes from recognizing the same underlying trade-off recurring in different clothes, which means you have to name the pattern explicitly rather than just resolving each instance.

Recede deliberately in stages. Start by explaining your own reasoning out loud when a decision comes up. Then ask them to predict what you'd decide, and why, before you weigh in. Then let them make the call and explain their reasoning to you after the fact. Then stop reviewing it at all. Each stage removes a layer of your safety net.

Make them narrate the criteria, not just the outcome. If someone can only say "I did X because I figured that's what you'd want," they've pattern-matched to you specifically, not internalized the underlying principle. You're listening for whether their stated reasoning would still hold up in a case where the "obvious" answer is actually wrong.

Verify with a novel or unobserved case. The strongest signal is watching them apply the same reasoning to a situation they haven't seen before, particularly one where you weren't in the loop and only heard about the decision afterward.

Worked example

Someone you're mentoring kept bringing you a specific recurring trade-off as if it were a one-off question each time: whether to fix a flaky, intermittently-failing test or ship a feature that was ready and waiting on it. Each time, you could have just answered the immediate question. Instead you treated it as a judgment gap and built a repeatable heuristic with them: is the flake masking a real intermittent bug or is it environment noise, what's the actual blast radius of shipping with it unresolved, and is there a way to quarantine the test that unblocks delivery without hiding the underlying risk.

Weeks later, a similar trade-off came up and they handled it without asking you first, only mentioning the decision afterward along with their reasoning. Their stated criteria matched the heuristic you'd built together, but in their own words, applied to a case with a different shape than the original one. That, not their confidence in the moment, was the signal it had actually internalized rather than just been remembered.

Trade-offs and pitfalls

Asking someone "do you understand?" tells you almost nothing; people say yes regardless of whether it's true. The only real test is watching the reasoning survive a situation you didn't script.

A subtle failure mode: rewarding a decision because it matches what you personally would have done, rather than evaluating whether the reasoning behind it was sound. If the original case was genuinely a coin toss, insisting they land on your exact answer trains obedience, not judgment.

The deeper trade-off is time and tolerance for being wrong. Actually receding means letting them face real stakes without a safety net, which means tolerating some decisions that turn out wrong in hindsight. That's not a bug in the process; it's the cost of judgment actually being tested rather than simulated.

A mentor who never truly recedes, who keeps reviewing every instance of the decision "just to be safe," never actually finds out whether the judgment transferred, because it's never been tested without the net.

Direct answer: Full-batch gradient descent computes the gradient over the ENTIRE dataset of size N before each update - O(N) work per step, but very few steps needed since each step's gradient is exact. Pure SGD (batch size 1) computes a gradient from a single example - O(1) work per step, but needs many more steps and each step's gradient is a noisy, high-variance estimate. Mini-batch SGD (batch size b) is the practical middle ground, O(b) work per step with gradient variance that decreases as $1/\sqrt{b}$, and - critically for modern hardware - the per-step work is highly PARALLELIZABLE across the batch dimension on a GPU, which is why throughput doesn't simply plateau immediately as batch size grows from 1.

Structured elaboration

• Per-step cost: proportional to batch size b (O(b) forward+backward work), true for all three variants (full-batch is simply b=N, pure SGD is b=1).
• Gradient variance: averaging over b independent samples reduces the VARIANCE of the gradient estimate by a factor of b (standard error shrinks as $1/\sqrt{b}$) relative to a single-sample gradient - this is why larger batches give a more accurate (lower-variance) estimate of the true full-dataset gradient direction, at the cost of more compute per step.
• GPU throughput and batch size: a GPU has a large but FIXED amount of parallel compute capacity; for small batch sizes, the GPU is under-utilized (idle compute units, since there isn't enough parallel work to fill them), so increasing batch size increases THROUGHPUT (examples processed per second) roughly linearly at first. Past some batch size, the GPU's compute units are fully saturated, and further batch-size increases no longer improve throughput per unit time (you're now compute-bound, not parallelism-starved) - this is why throughput plateaus, not because the underlying O(b) per-step cost changes, but because the WALL-CLOCK cost of processing b examples stops decreasing per-example once hardware parallelism is maxed out.

Worked example

Consider a GPU with enough parallel compute to fully utilize itself at batch size 256 for a given model. At batch size 32, the GPU processes each step in roughly the SAME wall-clock time it would take at batch size 256 (both are "cheap enough" to fit within one wave of parallel execution, dominated by fixed per-step overhead like kernel launch latency, not by the actual per-example compute) - so throughput (examples/second) at batch 32 is roughly 8x lower than at batch 256, purely from underutilizing available parallelism. Increasing batch size from 256 to 1024 (4x), once the GPU is already saturated at 256, roughly QUADRUPLES the wall-clock time per step (now truly compute-bound, scaling with actual work) while processing 4x the examples - so throughput stays roughly flat past this point, confirming the plateau.

Trade-offs & pitfalls

• Very large batch sizes, beyond the GPU-saturation throughput benefit, introduce their own OPTIMIZATION-QUALITY trade-off separate from the raw throughput question: overly large batches can generalize worse (a well-documented empirical phenomenon, sometimes called the "generalization gap"), requiring learning-rate scaling and other adjustments to compensate.
• Memory, not just compute, constrains batch size in practice - larger batches need proportionally more memory for activations (as discussed in the dense-layer-complexity survivor), and can hit a hard memory ceiling well before hitting a compute-bound throughput plateau.
• The variance-reduction benefit of larger batches has DIMINISHING returns ($1/\sqrt{b}$, not $1/b$) - doubling batch size from 1024 to 2048 reduces gradient noise by only about 29% (since $1/\sqrt{2} \approx 0.71$), a much smaller relative improvement than the first doubling from 1 to 2, which is part of why very large batch sizes give diminishing optimization benefit even before hitting hardware throughput limits.