DoorDash Staff-Level AI Engineer Interview Preparation Guide

Direct answer: A 12-month roadmap to build production RLHF capability should move through discovery and infrastructure design, a preference-collection and reward-model pilot, a small-scale RLHF deploy with safety validation, and finally scaled, automated, governed production rollout, with explicit team roles and measurable success criteria gating every stage.

Structured elaboration:

• Months 0-3 (discovery and foundations): define the success metrics up front, audit data, privacy, and safety constraints, establish a baseline evaluation of the current model, and produce an infrastructure design covering preference collection, reward-model training, and the RLHF training pipeline; this stage needs a staff-level lead, ML engineers for training infrastructure, data engineers for labeling pipelines, a researcher focused on reward-model design, a safety lead, and UX research input; success criteria are concrete (baseline helpfulness and harm-rate metrics collected, infrastructure design formally approved, sampling plan covering the large majority of core use cases).
• Months 4-6 (pilot: preference collection and first reward model): run a closed pilot collecting tens of thousands of ranked preference pairs, train the first reward model, and integrate the annotation UI and logging; success criteria include a reward-model quality bar (a held-out AUC comfortably above chance, for example above 0.85), a sustained annotation throughput target, and an inter-annotator agreement bar (a Krippendorff's alpha or similar statistic clearing a substantial threshold, for example above 0.6).
• Months 7-9 (RLHF training and small-scale canary deploy): run RLHF on a medium-scale model, validate offline, and deploy to a small canary cohort with safety filters and red-teaming in place; success criteria include a measurable helpfulness improvement over baseline (for example a double-digit percentage gain on internal evaluation), a measurable reduction in safety incidents on the canary cohort, and staying within latency SLOs.
• Months 10-12 (scale, automation, and governance): scale RLHF to the full production model line, automate the preference-collection pipeline so it does not depend on manual one-off pilots, and formalize governance (audit logs, rollout playbooks, retrospective review); success criteria include production rollout reaching a large majority of traffic with a further measurable helpfulness gain and safety incidents at or below baseline, an automated retraining cadence (for example every 8 weeks) running reliably, and documented governance formally signed off.
• Cross-functional governance, established starting in months 0-3 and running throughout: bring product, legal, security, and safety together to agree an explicit policy on what content can be annotated, data retention and consent rules for preference data, an incident-response process for a safety regression, clearly assigned decision authority for who can approve a production rollout, and audit logging covering every training and deployment decision, this governance layer is what keeps the later stages accountable, not a document produced once and shelved.
• Cross-cutting metrics tracked continuously, not just at stage gates: normalized human-preference win rate or task-success rate, policy-violation rate per fixed query volume, reward-model calibration and AUC, annotation throughput and agreement, and operational metrics (training cost per model, deployment latency, rollback frequency).

Worked example: A concrete stage-gate decision: if by month 6 the reward model's held-out AUC is below the 0.85 bar, or inter-annotator agreement is below the 0.6 threshold, the roadmap should NOT proceed to the month 7-9 small-scale RLHF deploy on schedule, since training RLHF against an unreliable reward model is exactly the setup that produces reward hacking and wasted compute; the correct response is extending the pilot stage to fix the reward model or annotation process first, even if that pushes the overall timeline, rather than proceeding on schedule with a known-weak reward signal.

Trade-offs and pitfalls: The most common risk to this kind of roadmap is treating the month-by-month milestones as fixed calendar commitments rather than as gates contingent on the PRIOR stage's success criteria actually being met, proceeding to RLHF training on a reward model that failed its own quality bar, purely to hit a roadmap date, reliably produces exactly the reward-hacking and safety-regression problems this roadmap's later stages are meant to prevent. Annotator bias and reward hacking are named as recurring risks for a reason, they are not one-time gates passed at month 6, they need periodic re-auditing and red-teaming built into every later stage, not just the initial pilot.

Situation: At my last company I led deployment of a customer-support generative QA model integrated into our chat widget. Two days after go‑live, users reported that the bot returned confidently wrong and sometimes unsafe instructions for refund/chargeback processes, causing customer frustration and a spike in support tickets.

Task: I had to stop the user harm quickly, diagnose the root cause, and put safeguards in place so we wouldn't repeat it.

Action:

• Immediate remediation:
  • I temporarily rolled back to the previous deterministic FAQ system within 30 minutes to stop further bad responses.
  • I disabled the model’s ability to make policy-changing recommendations (removed “advisory” turn) and added a banner clarifying responses were suggestions, not legal advice.
  • I created an incident channel and coordinated with support to prioritize and respond to the highest-impact tickets.
• Diagnosis:
  • I reviewed request/response logs, prompt templates, and recent fine‑tune changes. I discovered a recent prompt-engineering update removed a validation step and we had fine‑tuned on a noisy dataset containing outdated policy examples; combined with a temperature bump in production, this increased hallucination.
  • Reproduced failure cases locally by replaying production inputs and confirmed the faulty prompt + dataset combo produced unsafe output.
• Long-term fixes:
  • Restored the validation guard in the prompt and reverted the fine‑tune. Instituted an approval workflow: any prompt or fine‑tune change must pass a small “pre-release” suite (100 representative queries, automatic heuristics for hallucination and toxicity, and human review).
  • Added real‑time monitoring: soft‑confidence scoring, perplexity thresholds, and an automated rollback trigger when error rate or ticket volume crosses thresholds.
  • Implemented content filters and a “safe fallback” that returns verified FAQ snippets when confidence is low.
  • Updated runbooks and ran a blameless postmortem; we trained the team on the new release process and improved dataset labeling guidelines.

Result: After rollback and fixes we reduced erroneous responses to near zero within 24 hours and ticket volume returned to baseline in 48 hours. The new pre‑release tests caught two subsequent risky changes before deployment. The incident taught me to treat prompt and fine‑tune changes with the same rigor as production code: small changes, automated checks, and clear rollback paths.

High-level summary: ONNX Runtime (ORT), TensorRT, and OpenVINO are production-focused inference runtimes with different strengths: ORT for portability and broad backend support, TensorRT for maximum NVIDIA GPU throughput/low-latency, and OpenVINO for Intel CPUs, iGPUs, and VPUs with strong CPU/edge optimization.

ONNX Runtime

• Performance advantages: Good out-of-the-box performance across CPUs and GPUs; vendor-optimized execution providers (CUDA, ROCm, DirectML, OpenVINO) and graph optimizations (fusions, constant folding).
• Supported hardware: CPUs (x86/ARM), NVIDIA/AMD GPUs via providers, Windows ML/DirectML, specialized accelerators via plugins.
• Conversion pain points: ONNX export fidelity varies by framework/operator version; opset mismatches and custom ops may require rewriting or custom kernels.
• When to choose: Multi-platform deployments where portability, fast iteration, and easy integration with ML infra matter (cloud + edge hybrid).

TensorRT

• Performance advantages: Best throughput/latency on NVIDIA GPUs via aggressive layer fusion, kernel autotuning, FP16/INT8 quantization, and optimized memory planning.
• Supported hardware: NVIDIA GPUs / Jetson.
• Conversion pain points: Requires ONNX or framework export compatible with TRT; unsupported ops need plugins; dynamic-shape handling and calibration for INT8 add engineering overhead.
• When to choose: High-throughput GPU inference on NVIDIA hardware (server GPUs or Jetson) when maximizing performance and minimizing latency is critical.

OpenVINO

• Performance advantages: Optimized for Intel CPUs, integrated CPU vectorization, low-latency on iGPUs and Movidius VPUs; good CPU inference performance without a GPU.
• Supported hardware: Intel x86 CPUs, integrated GPUs, Intel VPUs (Neural Compute Stick), Windows/Linux.
• Conversion pain points: Requires conversion to IR or OpenVINO format; some ops and newer frameworks need workarounds; quantization/tooling quirks.
• When to choose: Edge or on-prem CPU/iGPU/Intel VPU deployments where GPU is unavailable and power/latency constraints exist.

Trade-offs / practical guidance:

• Start with ORT for cross-platform proof-of-concept; profile.
• If deploying to NVIDIA and need peak perf, convert to TensorRT and invest in plugin/INT8 calibration.
• For CPU/Intel-edge targets, convert to OpenVINO for best CPU/generic-edge latency.
• Keep an automated CI conversion & validation pipeline and unit tests to catch numerical/regression issues after conversion.

Also covers (folded from merged near-duplicates): 794ec20b/a4074587/fdfae4f6 fold the fusion/precision-lowering/memory-pooling mechanics of how these runtimes get their speedups.

Start by clarifying requirements and constraints: quantify the acceptable false positive rate, the minimum recall to preserve, latency and throughput limits, and the costs of FPs vs FNs.

Which lever to pull first: rank the candidate levers by expected precision gain divided by (cost x time x reversibility), and go cheapest-and-most-reversible first:

1. Recalibration and per-segment threshold search on the EXISTING model (hours, fully reversible, no retraining): often recovers real precision for free if the model has drifted since it was last tuned.
2. Deterministic post-processing / business rules for known FP patterns (days, cheap, easy to roll back): removes the 'easy' false positives without touching the model at all.
3. A second-stage verifier or cascade layered on top of the existing high-recall model (one to two weeks, moderate cost, and crucially never touches first-stage recall since the first stage is left untouched).
4. Loss-function changes or full retraining (weeks, expensive, and the only lever that risks recall directly since it changes the underlying model).
   Concretely: run 1 and 2 first and measure the precision delta on last week's traffic via offline replay (no user risk); only invest in 3 if the residual FP rate is still unacceptable, and only invest in 4 if the operating point 1-3 land on genuinely can't clear the bar without more discriminating signal.

Architecture changes

• Add a two-stage pipeline: a high-sensitivity first-stage model to keep recall, followed by a more precise second-stage verifier (a heavier model or rule-based checks).
• Separate feature stores for fast online features vs. richer batch features used by the verifier.
• Add an explainability / feature-importance layer to enable targeted post-processing.

Loss-function strategies

• Use asymmetric loss to penalize false positives more: weighted cross-entropy or focal loss with higher weight on negative examples misclassified as positive.
• Consider custom cost-sensitive losses reflecting the business cost matrix, or optimize differentiable proxies for precision@k.
• If recall must be preserved, add a recall-constraint via Lagrangian optimization (constrained learning).

Thresholding and calibration

• Move from a default 0.5 to an operating-point selection on the validation ROC/PR curve: pick the threshold maximizing F1, or precision at the required recall.
• Calibrate probabilities (Platt scaling, isotonic regression) separately per segment so thresholds behave consistently.
• Use per-segment thresholds (customer, country, confidence bucket) if operating characteristics vary.

Ensembling / cascaded models

• Ensemble diverse models (tree-based plus neural) using stacking, where the meta-model focuses on reducing FPs.
• Cascade: a fast model filters candidates, a verifier (a larger network, gradient-boosted model, or rules plus human-in-loop) reduces FPs. Use disagreement signals as features for the verifier.
• Use model uncertainty (Monte Carlo dropout, deep ensembles) to route low-confidence positives to manual review.

Post-processing and business rules

• Implement deterministic rules that eliminate known false-positive patterns (transaction thresholds, blacklists).
• Use time-window aggregation (require repeated signals) to reduce transient FPs.
• Add a feedback loop: log and store FP examples for periodic retraining.

Evaluation steps

• Evaluate on a stratified holdout and realistic production-like data; report precision at fixed recall (e.g., precision at 95% recall), PR curves, and cost-weighted loss.
• Run shadow deployments / A/B tests comparing baseline vs. the cascaded pipeline; measure business KPIs (cost of FPs, latency).
• Monitor calibration drift and population shift; alert on metric degradation.
• Iterate: retrain with newly labeled FPs, adjust thresholds per segment, and measure impact.

This approach preserves high recall by keeping the sensitive first stage untouched, while improving precision through the cheapest levers first (recalibration, rules), then targeted loss design, calibrated thresholds, cascaded verification, ensembling diversity, and robust evaluation and monitoring.

Direct answer

When a set of small improvements each show diminishing returns, the decision to keep iterating versus switch to exploratory research comes down to estimating the remaining headroom on the current path against the expected value of a genuinely different approach, adjusted for how much uncertainty you can tolerate right now.

Structured elaboration

1. Estimate remaining headroom on the current path. Plot the trend of gains from recent incremental improvements; a clearly flattening curve suggests the current approach is running out of room, while a still-meaningfully-positive slope suggests more is available even if each individual improvement looks small.
2. Estimate the research bet's expected value and its uncertainty. A promising new direction (a different model family, a fundamentally different feature source) might have a much higher ceiling, but with real uncertainty about whether it pays off at all within a reasonable timeframe.
3. Factor in your tolerance for uncertainty right now. If the business needs predictable, incremental progress this quarter, continuing the diminishing-returns path might be the safer near-term choice even with a lower ceiling; if there's room for a bet with a longer payoff horizon, the exploratory path may be worth the risk.
4. Consider a split allocation rather than an all-or-nothing switch. Continue harvesting the remaining small gains on the known path with a minority of the team's time, while dedicating the rest to exploring the higher-ceiling but riskier direction, rather than abandoning one entirely for the other.

Worked example

If the last three incremental improvements averaged a shrinking gain each time (a clear diminishing pattern) and a promising alternative feature source has shown early signal in a small pilot, splitting the team's time (a smaller group continuing quick wins on the known path, a smaller group prototyping the new direction) often captures more expected value than betting everything on either path alone.

Trade-offs and pitfalls

The common mistake is continuing down a diminishing-returns path purely out of inertia, since each individual increment still looks like a small win even as the cumulative trend flattens. The opposite mistake is abandoning a reliable, if unglamorous, source of steady gains for a speculative bet without first checking whether the bet has any early validating signal at all.

1. Dataset prep

• Annotate 500 images in COCO/COCO-JSON format; ensure polygons per instance and correct class IDs.
• Split stratified by scene/class: train 70% (350), val 20% (100), test 10% (50). For small data consider 5-fold cross-validation if resources allow.
• Inspect masks for quality; normalize input sizes (resize shorter side to 800 px, max 1333) but keep aspect ratio.

2. Baseline and data loaders

• Load Mask R-CNN pretrained on COCO (backbone ResNet-50/101 + FPN).
• Replace classification/mask heads to output 5 classes (+background). Initialize new heads randomly (Xavier).

3. Freezing strategy (progressive)

• Stage A (heads only): freeze backbone and FPN; train RPN + box & mask heads for 5–10 epochs with batch size 2–8 (GPU-limited).
• Stage B (partial unfreeze): unfreeze last ResNet block (C4/C5) + FPN for 10–20 epochs.
• Stage C (fine-tune all): unfreeze entire backbone for 5–10 epochs with lower LR only if val loss still improving.

4. Learning rate schedule

• Use SGD with momentum 0.9, weight decay 1e-4.
• Stage A LR: 1e-3 (if batch small, scale with linear rule); Stage B LR: 5e-4; Stage C LR: 1e-4.
• Warmup for first 500–1000 iterations (LR ramp from 0.1× to target).
• Use step decay (reduce LR by 0.1 on plateau) or cosine annealing; combine with early stopping based on val mAP.

5. Augmentations (critical for 500 images)

• Geometric: horizontal flip (50%), random scale (0.8–1.2), small rotation ±10°, random crop when object still present.
• Photometric: brightness ±20%, contrast ±20%, hue/saturation jitter, Gaussian blur (rare).
• CutMix/Mosaic generally not for instance masks; instead use copy-paste augmentation (paste objects into other images with blending) to increase instance variety.
• Ensure masks transformed consistently.

6. Validation & metrics

• Evaluate every epoch on val set: bbox mAP@0.5:0.95 and mask mAP, per-class AP, precision/recall, and mean IoU.
• Use test-time augmentation (flips, multi-scale) on final model for robust estimates.
• If using cross-val, average mAPs and monitor variance.

7. Avoiding overfitting

• Aggressive augmentations and copy-paste to increase diversity.
• Strong regularization: weight decay 1e-4, dropout in heads if available.
• Early stopping when val mAP stops improving for 5–10 epochs.
• Model checkpointing: keep best by val mAP and last N checkpoints.
• Monitor per-class performance for overfitting to dominant classes; use class-balanced sampling or loss weighting if imbalance present.

8. Practical tips

• Use mixed precision (FP16) to increase batch size.
• Log experiments (wandb/MLflow), save seeds for reproducibility.
• If convergence slow, try pretrained backbone from similar domain or perform self-supervised pretraining on unlabeled domain images.

Expected outcome: heads converge quickly; careful progressive unfreezing + augmentations and early stopping typically yield good mask AP on small datasets without catastrophic overfitting.

Direct answer

Quicksort is in-place with average time $O(n \log n)$ but a worst case of $O(n^2)$ on an unlucky pivot sequence; merge sort and heap sort both guarantee $O(n \log n)$ in every case. Merge sort needs $O(n)$ extra space and is stable; heap sort needs only $O(1)$ extra space but is not stable; quicksort's extra space is $O(\log n)$ for the recursion stack on average, but can grow to $O(n)$ in the worst case. For nearly-sorted data, pick an adaptive sort such as TimSort (the hybrid merge/insertion sort behind Python's and Java's built-in sort); when a guaranteed worst case matters more than average speed, pick heap sort or merge sort, never plain quicksort.

Structured elaboration

Nearly-sorted input

Plain quicksort and plain top-down merge sort are not adaptive: both do the same $O(n \log n)$ work regardless of how ordered the input already is. TimSort is: it scans for existing ascending or descending runs, extends and merges them, and degrades toward close to linear work as the input approaches already-sorted. For nearly-sorted data, reach for TimSort (or, if you must hand-roll something, a natural merge sort) rather than a textbook quicksort or merge sort.

Worst-case guarantees matter more than average speed

Both heap sort and merge sort guarantee $O(n \log n)$ in every case; quicksort does not, no matter how the pivot is chosen, because an adversary (or, unintentionally, already-sorted or already-reverse-sorted input under a naive pivot rule) can always construct a sequence that degrades a fixed pivot strategy to $O(n^2)$. Choose heap sort when the extra $O(n)$ memory merge sort needs is unavailable and stability is not required; choose merge sort when stability is required alongside the worst-case guarantee and the memory budget allows it.

Two side notes worth naming explicitly

• Parallelization on resource-constrained devices: merge sort's divide phase maps cleanly onto independent worker threads or cores (each half sorts independently before a merge step), which is attractive on a multi-core mobile device; the cost is the extra O(n) buffer merge sort needs, which is a real constraint on memory-limited hardware. Quicksort's partitions can also be sorted concurrently, but partition sizes are unpredictable (a skewed pivot gives one thread almost all the work), so load balancing is harder to reason about.
• Cross-language floating-point sort determinism: when the same data is sorted by comparator across different languages or platforms, an unstable sort's tie-breaking for equal keys is unspecified and can differ, and NaN comparisons under IEEE 754 floating point are neither less-than nor greater-than any value, which breaks the total-order assumption most sort implementations rely on. If reproducible ordering across systems matters (for example, deterministic test fixtures or replaying a pipeline), use a stable sort and either exclude or explicitly place NaNs, rather than relying on the default comparator.

Worked example

A concrete way to see the worst case: implement a plain quicksort that always pivots on the last element, and run it on an already-sorted array.

def quicksort_last_pivot_count(a: list[int]) -> int:
    """
    Naive quicksort that always pivots on the last element.
    Returns the number of comparisons performed (element-to-pivot checks).
    """
    comparisons = 0

    def sort(lo: int, hi: int) -> None:
        nonlocal comparisons
        if lo >= hi:
            return
        pivot = a[hi]
        store = lo
        for i in range(lo, hi):
            comparisons += 1
            if a[i] < pivot:
                a[i], a[store] = a[store], a[i]
                store += 1
        a[store], a[hi] = a[hi], a[store]
        sort(lo, store - 1)
        sort(store + 1, hi)

    sort(0, len(a) - 1)
    return comparisons


if __name__ == "__main__":
    for n in [6, 10, 20]:
        already_sorted = list(range(n))
        c = quicksort_last_pivot_count(already_sorted)
        expected = n * (n - 1) // 2
        print(f"n={n}: comparisons={c}, n(n-1)/2={expected}")

Running this prints:

n=6: comparisons=15, n(n-1)/2=15
n=10: comparisons=45, n(n-1)/2=45
n=20: comparisons=190, n(n-1)/2=190

Every partition step on already-sorted input with a last-element pivot puts everything on one side, so the recursion depth is n and the total comparisons are exactly $n(n-1)/2 = \Theta(n^2)$, confirmed by the counts matching the closed-form prediction at every size tested. A randomized or median-of-three pivot choice avoids this specific failure mode but does not eliminate the worst case in general, only make it exponentially unlikely to hit by chance.

Trade-offs & pitfalls

The most common wrong turn is treating quicksort as unconditionally the fastest choice: on already-sorted or reverse-sorted input under a naive pivot rule, it is the slowest of the three by an order of magnitude, as the worked example shows directly. A second common gap is forgetting that merge sort's memory cost is real: at large enough n, the O(n) auxiliary buffer competes with other memory pressure, which is exactly why external (disk-based) sorting is built on multi-way merge rather than quicksort, since merge sort's sequential access pattern suits disk or network I/O far better than quicksort's more random access pattern. A third trap is ignoring stability when it silently matters: if you sort by a secondary key after already sorting by a primary key, only a stable sort preserves the primary ordering among equal secondary keys; using an unstable sort there produces a result that looks correct on small examples but is wrong in general.

Direct answer

At this scale a single model cannot score 10 million items per request in time, so the system splits into a fast, recall-oriented candidate-generation stage that narrows 10 million items down to a few hundred plausibly relevant ones, followed by a slower, precision-oriented ranking stage that only has to score that few hundred. Freshness is handled by separating what must update within seconds, a user's most recent actions, from what can lag by minutes or hours, item popularity and heavier aggregates, and recomputing each on a schedule that matches how quickly that particular signal actually goes stale.

Structured elaboration

Two-stage funnel: candidate generation (for example, embedding-based approximate nearest-neighbor, ANN, retrieval plus cheap popularity or co-occurrence heuristics, run over the full item catalog) feeds ranking (a heavier model that only scores the few hundred candidates the first stage returned). Doing this in a single stage would mean running an expensive model 10 million times per request; splitting it means the cheap stage runs once at full scale and the expensive stage runs only a few hundred times.

Feature freshness tiers:

Serving path: request arrives, the user's real-time and cached features are fetched, candidate generation runs an ANN lookup against item embeddings, a feature join enriches the returned candidates, the ranking model scores them, business rules and diversity filtering are applied, and the response returns. Only the batch recompute happens off this path entirely.

Train/serve consistency: the same feature definitions and item and user embeddings used to train the ranking model must be the ones materialized into the online store the serving path reads, the same feature-store discipline used to avoid training-serving skew, applied here at scale.

flowchart LR
  U[Request: user_id] --> RT[Real-time feature fetch]
  RT --> CG[Candidate generation: ANN plus heuristics]
  CG --> FJ[Feature join for candidates]
  FJ --> RK[Ranking model]
  RK --> BR[Business rules and diversity filter]
  BR --> RESP[Response]
  BATCH[Batch feature and embedding jobs] -.refresh.-> CG
  BATCH -.refresh.-> RK

Worked example

Latency budget for a 200 ms SLA (service level agreement), allocated across the serving path:

Item embedding storage: 10 million items, each a 128-dimension embedding stored as 4-byte (32-bit) floats.

The full embedding table fits comfortably in memory on a single high-memory host, which is why an in-memory ANN index is realistic at this item count; a tenfold growth in item count or embedding dimension would scale this to roughly 47.7 GiB, the point at which a sharded ANN index becomes the right call instead.

QPS (queries per second): 100 million users, 5 percent daily active, 4 recommendation-triggering requests per active user per day.

At a conservative 5x peak-to-average ratio:

Trade-offs and pitfalls

A common wrong turn is making every feature real-time "to be safe," which multiplies write load on the low-latency store for signals that do not actually change fast enough to matter, spending a budget that should go to genuinely time-sensitive signals. Tuning candidate generation purely for precision silently shrinks recall and starves the ranking stage of good candidates it never gets a chance to score. Sharding the ANN index for scale trades a small amount of recall and added merge latency for memory headroom, so it should be adopted deliberately at a measured threshold, as the tenfold-growth arithmetic above illustrates, not by default from the start.

Direct answer

GPU memory capacity limits how large a batch and model you can fit at all, while memory bandwidth (how fast data moves between GPU memory and compute units) limits how fast you can actually process that data once it fits, so a training configuration can be capacity-bound, bandwidth-bound, or compute-bound depending on which resource is the tightest constraint for a given model and batch size.

Structured elaboration

• Capacity constraint: parameters, gradients, optimizer state, and activations must all fit within the GPU's total memory (e.g. 80GB on an H100); exceeding this causes an out-of-memory error regardless of how fast the GPU could otherwise compute, forcing a smaller batch size, model, or the memory-saving techniques (mixed precision, checkpointing, sharding) discussed elsewhere in this topic.
• Bandwidth constraint: even when everything fits, operations that move a lot of data relative to the compute they perform (elementwise operations, some normalization layers, small matrix multiplies) can be bottlenecked by how fast data streams between HBM and the GPU's compute units rather than by the compute units' raw throughput; this shows up as low GPU utilization percentage even though the workload is running continuously.
• Batch size trade-off: larger batch sizes generally improve compute efficiency (better utilization of the GPU's parallel compute, amortizing fixed per-kernel-launch overhead) up to the point where capacity is exhausted; very small batch sizes tend to be more bandwidth-bound (relatively more data movement per unit of useful compute) since fixed overheads and memory-bound elementwise operations don't scale down favorably.
• Model size trade-off: a larger model directly increases capacity pressure (more parameters, more activation memory per sample) which forces either a smaller batch size or additional memory-saving techniques, in turn affecting how compute-efficient a given step actually is.

Worked example

Training a model where activation memory dominates: at batch size 8, activations plus weights plus optimizer state fit within an 80GB GPU with headroom, and the GPU runs near its compute-bound peak throughput; increasing batch size to 32 might exceed the 80GB budget entirely (capacity-bound failure), while decreasing to batch size 1 for debugging purposes leaves so much of each kernel launch as fixed overhead relative to the tiny amount of actual compute that the GPU spends much of its time bandwidth- or launch-overhead-bound rather than compute-bound.

Trade-offs & pitfalls

A common mistake is assuming more GPU memory alone fixes a slow training run; if the workload is bandwidth-bound rather than capacity-bound, adding memory (a bigger GPU) doesn't address the actual bottleneck, only profiling (checking whether the GPU is near its compute-throughput ceiling or spending time waiting on memory transfers) reveals which constraint is actually binding.

Direct answer

Handle Unicode normalization first (so the same character isn't represented multiple different byte-level ways), detect the language or script per document (or per segment, for code-switched text), route to a per-language tokenizer where meaningful differences exist, and build a shared vocabulary that lets related languages share subword tokens rather than maintaining entirely separate, non-overlapping vocabularies.

Structured elaboration

Unicode normalization: apply a consistent normalization form (NFC or NFKC) so visually-identical characters that differ at the byte level (accented characters composed differently, full-width versus half-width variants common in East Asian text) get treated as the same token rather than silently fragmenting your vocabulary with near-duplicate entries.

Script and language detection: detect the script (Latin, Cyrillic, Thai, CJK) and, within a script, the specific language, since tokenization needs genuinely differ (whitespace-based word boundaries work for many Latin-script languages but not for Thai or Chinese); for a genuinely multilingual document collection, this detection needs to run per-document or even per-segment if individual documents mix languages.

Per-language tokenization: languages needing fundamentally different tokenization strategies (character-based for Chinese, morphology-aware for agglutinative languages like Turkish or Finnish) should get purpose-appropriate tokenizers rather than forcing one whitespace-based scheme onto all of them.

Shared vocabulary creation: for a multilingual MODEL specifically (not just per-language preprocessing), a jointly-trained subword vocabulary across all target languages lets related languages share tokens for cognates and loanwords, improving cross-lingual transfer, at the cost of needing careful attention to vocabulary allocation so no single high-resource language dominates the shared vocabulary at the expense of lower-resource ones.

Worked example

A dataset mixing English, Chinese, and code-switched Singlish (English-Chinese mixed within single sentences, common in Singapore/Malaysia) needs per-SEGMENT language detection, not just per-document, since a single sentence might legitimately need both a whitespace-aware English tokenizer for part of it and a character-based Chinese tokenizer for another part, a genuinely harder preprocessing problem than either monolingual case alone.

Trade-offs and pitfalls

The shared-vocabulary allocation problem is a real, often underestimated trade-off: if the corpus is 90% English and 10% a mix of 10 other languages, a naively-trained shared subword vocabulary will allocate the vast majority of its capacity to English-optimized tokens, leaving the lower-resource languages with excessive subword fragmentation (very short, uninformative tokens) that hurts their downstream model quality specifically, which is why deliberate vocabulary-allocation strategies (oversampling low-resource languages during vocabulary training, or a temperature-based sampling scheme) are worth the added complexity for a corpus with genuinely imbalanced language representation.