DoorDash AI Engineer Interview Preparation Guide - Entry Level

Situation: My MT model shows falling validation loss but BLEU and human ratings don't improve. That mismatch suggests evaluation or inference problems rather than learning progress. I would run a targeted debug plan covering data leakage, tokenization, metric correctness, decoding, and smoothing.

Step-by-step plan:

1. Verify no data leakage

• Recompute hash/signature of validation source/target files used during training and during evaluation; compare.
• Check training pipeline for accidental eval-target copies (grep for file paths in training logs).
• Train a tiny model on train=val subset; if val loss collapses and BLEU is perfect, leakage likely.

2. Tokenization / detokenization mismatch

• Confirm same tokenizer config (vocab, BPE/ SentencePiece model, normalizers, lowercasing) used for training, validation loss computation, and test-time decoding.
• Round-trip test: take a few validation source sentences -> tokenizer -> detokenizer -> compare to original.
• Ensure metric and human eval use detokenized, detruecased outputs matching training text normalization.

3. Metric implementation correctness

• Validate BLEU implementation: compute sacreBLEU on sample predictions and known references; compare with sacrebleu CLI.
• Sanity check: if pred==ref, metric must return max score. Feed a tiny set where you know expected BLEU/n-gram counts.
• Ensure consistent tokenization used by BLEU (sacreBLEU's internal tokenizer vs raw).

4. Decoding / hyperparameters

• Compare training-time teacher-forcing vs inference: test greedy, beam search (various beam sizes), length penalty, sampling temperature, and repetition penalty.
• Sweep a grid: beam={1,4,8}, length_penalty={0.6,1.0,1.4}, min_len/max_len. Log likelihoods vs BLEU.
• If higher validation log-prob but greedy output is poor, consider that model assigns mass to many plausible outputs (mode averaging).

5. Label-smoothing / over-smoothing checks

• Check training config for label_smoothing value; retrain a short run with label_smoothing=0 to see effect on BLEU.
• Inspect output distribution entropy: compute average softmax entropy per token on validation set. Excessively high entropy suggests over-smoothing.
• Visual check of top-k probabilities; if top-1 not dominant, model may be assigning probability mass to synonyms causing low BLEU but acceptable human quality.

6. Additional diagnostics

• Human vs BLEU gap: compute chrF and METEOR to see if other metrics capture improvements.
• Analyze examples: sample cases where loss decreased but BLEU dropped—inspect references, multiple valid translations, and alignment attention maps.
• Check training/eval mode flags: ensure dropout, batchnorm are disabled during inference.

Expected outcomes and next steps:

• If leakage found: remove overlap and retrain.
• If tokenization mismatch: unify tokenizer and re-evaluate.
• If metric bug: fix metric and re-score.
• If decoding hyperparameters help: update default decoding or calibrate score with length penalty.
• If label smoothing causes high-entropy outputs: reduce smoothing or use sequence-level fine-tuning (minimum risk training, RL-based BLEU fine-tuning).

This systematic approach isolates whether the issue is data, preprocessing, metric, inference, or training regularization and guides corrective actions.

Situation: Our product team wanted a prototype that could generate branded marketing images from short prompts. We had only a generic Stable Diffusion baseline and needed higher-fidelity, brand-consistent outputs quickly.

Task: I had to learn latent diffusion fine-tuning techniques (DreamBooth / LoRA) and GPU profiling to optimize training/inference so we could deliver a working prototype within three weeks.

Action:

• Week 1: Rapid self-study — read the original DDPM paper, the Latent Diffusion paper, and DreamBooth/LoRA posts; followed Hugging Face fine-tuning guides and reviewed top GitHub repos (CompVis, diffusers). Supplemented with two video tutorials and a Colab notebook to follow along.
• Week 2: Hands-on experiments — set up reproducible training on a single A100 instance, implemented LoRA to fine-tune on 50–100 branded images. I used Hugging Face Diffusers + Accelerate and wrote scripts to log checkpoints and metrics.
• Week 3: Validation & optimization — validated with quantitative (FID dropped from 38 to 22 vs baseline on a 200-image holdout) and qualitative A/B tests with product stakeholders. Used NVIDIA Nsight and torch.profiler to find a memory bottleneck; switching to mixed precision + gradient checkpointing reduced peak GPU memory by ~35% and training time per step by ~25%. Also ran ablation comparing full fine-tune vs LoRA to show cost/quality trade-offs.

Result: Delivered a prototype within three weeks that matched brand style with significantly better fidelity. Using LoRA + mixed precision cut training cost by ~40% versus full fine-tune and made iterative fine-tuning feasible for the product team. The prototype enabled a successful stakeholder demo and a follow-up ticket to productionize the pipeline. I documented the process and created a templated notebook so other engineers can reproduce results within a day.

This taught me to combine targeted paper reading with hands-on small experiments and profiling — that mix gets you from zero to production-ready fast while controlling cost and risk.

Direct answer
A blameless postmortem is a structured written review of an incident that treats the failure as evidence of a gap in the system rather than as evidence of a person's incompetence. It assumes everyone involved acted reasonably given the information and pressure they had at the time, and it asks 'what about the system made this possible' instead of 'who made this mistake.' A good postmortem document has a small, consistent set of sections: an incident summary and severity, a timestamped timeline, quantified impact, the root cause and any contributing factors, immediate mitigations already taken, and a list of owned, dated action items.

Structured elaboration
Each section earns its place by answering a different question a reader will actually ask:

• Summary and severity. One or two sentences so a reader who will never open the full document still knows what happened and how bad it was.
• Timeline. An objective, timestamped sequence of what happened, detected, and was done. This is the shared factual spine the rest of the document hangs off; without it, discussion drifts into competing memories.
• Impact. Quantified: how many users, how much revenue, how long, which SLOs were breached. Impact is what makes prioritization of the resulting action items defensible later.
• Root cause and contributing factors. The root cause is the condition that, if changed, would have prevented the incident; contributing factors made it more likely or worse but would not alone have caused it. Separating the two stops the document from over-claiming a single tidy cause when the real story is usually several factors lining up.
• Immediate mitigation. What was done to stop the bleeding, kept separate from the long-term fix, since these often have very different owners and timelines.
• Action items with owners and dates. Concrete, individually verifiable, and never phrased as 'be more careful.' A postmortem that ends with vague advice instead of an owned commitment produces no durable change.

The wording throughout matters as much as the structure. 'The on-call engineer missed a step in the runbook' names a person; 'the runbook did not make the required step hard to skip' names a system gap that is actually fixable. This isn't softening the facts, it's redirecting the analysis toward the thing you can change.

Worked example
An API returns errors for 45 minutes after a deploy. A blame-oriented writeup might say: "the engineer pushed a bad config and didn't test it." A blameless version says: "a config change with an invalid timeout value was deployed to production without automated validation or a staged rollout; the on-call engineer restored service in 12 minutes by rolling back. Root cause: the deploy pipeline allows unvalidated config to reach 100% of traffic in one step. Contributing factor: the config schema has no automated check for out-of-range timeout values. Action items: (1) add schema validation to the deploy pipeline, owner platform-team, due in two weeks; (2) require staged rollout for config-only changes above a defined blast-radius threshold, owner SRE lead, due in one month." Same incident, same facts, but the second version is auditable, points at fixable system gaps, and produces action items an unrelated engineer could pick up and execute.

Trade-offs and pitfalls
The most common failure is stopping the investigation at 'human error' as though that were itself the root cause. If a person did something reasonable given what they knew and the system still let it cause an outage, the real root cause is upstream: missing validation, an unclear runbook, a dangerous default. A second common failure is a postmortem so long and hedged nobody reads it. Sections should be short and factual; depth belongs in linked artifacts (logs, dashboards), not in the narrative itself.

Operator fusion merges multiple adjacent operators into a single kernel so they run as one fused computation instead of separate stages. That reduces intermediate memory reads/writes, kernel launch overhead, and allows combined kernel-level optimizations (e.g., better register reuse, vectorization).

Example - Conv → BatchNorm → ReLU:

• Without fusion: Conv writes output to DRAM, BatchNorm reads it, writes normalized output, ReLU reads/writes again. Each step incurs memory bandwidth and separate kernel launch overhead.
• With fusion: A single kernel computes convolution, applies BN scaling/shift, and applies ReLU on-the-fly before writing final outputs. Savings: fewer DRAM transfers (lower memory bandwidth), fewer kernel launches (lower latency), and better cache/register utilization.

Two pitfalls across runtimes:

1. Non-uniform fusion support: Different runtimes (TensorRT, ONNX Runtime, XLA, TVM) fuse different patterns or require graph/transformation passes; a model optimized on one runtime may not see the same gains on another. Relying on fusion without verifying runtime behavior can cause unpredictable performance.
2. Numerical and debugging issues: Fusing changes evaluation order and can slightly change floating-point results (reduced rounding, different accumulation). It also makes profiling and debugging harder because intermediate tensors are eliminated - complicating quantization calibration, gradient checking (in training), and root-cause analysis.

Best practice: profile end-to-end on target runtime/hardware, validate numerical equivalence within tolerance, and include fallback/controls for fusion passes.

Direct answer: A policy that collapses to short, generic replies while still scoring highly with the reward model is a textbook length-based reward-hacking failure, and the highest-leverage fix is almost always correcting the reward model itself, since the policy is behaving exactly as the (miscalibrated) reward signal instructs.

Structured elaboration:

• Likely causes, ranked by how directly they explain the symptom: the reward model itself has learned a spurious correlation between brevity or genericness and high reward (often inherited from annotation patterns where raters happened to prefer shorter responses on average, or from limited diversity in what scored highly during reward-model training); insufficiently diverse or too-short preference training data for the reward model; PPO dynamics that let the policy drift too far toward exploiting that reward-model bias (a KL penalty against the reference policy that is too weak, or a learning rate or update size that is too aggressive); and, more subtly, an under-trained value-function baseline that produces noisy advantage estimates and can itself destabilize training toward degenerate policies.
• Highest-ROI fix, correcting the reward model: audit the reward-model training data specifically for annotation shortcuts (raters systematically rewarding brevity), add contrastive training examples where a longer, genuinely more helpful response is explicitly labeled as preferred over a short-but-unhelpful one, and consider an ensemble of reward models, using their disagreement or a consensus score reduces the odds the policy can exploit a single model's specific blind spot.
• Second-priority fix, improving preference data collection: deliberately collect additional human preferences that emphasize completeness and helpfulness, including negative examples that are explicitly short-but-inadequate, and use red-team-style adversarial prompts to surface any other reward-model exploits before they show up in a full training run.
• Third-priority fix, constraining the PPO update itself: strengthen the KL penalty against the reference (supervised-fine-tuned) policy, reduce the clip range or learning rate, and add an explicit early-stopping rule that halts training if the measured reward keeps rising while an independent held-out human-eval sample shows quality dropping, exactly the divergence signature that indicates hacking is underway.
• Lower-priority but useful auxiliary fixes: combine the reward model's score with an auxiliary, context-aware length or completeness signal (not a blanket length penalty, since some prompts genuinely call for brevity), and periodically retrain the reward model on newly-collected labels that specifically include the policy's own hacking failures once discovered.

Worked example: If, after 500 PPO update steps, average response length has collapsed from roughly 90 tokens to 15 while the reward model's average score has risen 12%, but a small held-out human-eval sample rates the newer, shorter responses as measurably less helpful, that divergence between the automated reward metric and the independent human check is direct evidence of reward hacking specifically through brevity, and the fix sequence above (starting with the reward-model audit) is the ranked response, not simply reducing the learning rate and hoping the policy self-corrects.

Trade-offs and pitfalls: Reaching immediately for a blanket length penalty in the reward function is a tempting quick fix, but it risks over-penalizing legitimately brief responses to prompts that call for brevity, a context-aware or conditional length signal (informed by whether the prompt itself implies a need for detail) avoids this, at the cost of more engineering complexity than a blanket penalty. A second pitfall is treating this purely as an algorithmic PPO problem (tightening KL and clip) without also fixing the reward model itself, tightening PPO constraints slows down how fast the policy can exploit a bad reward model, but does not remove the underlying incentive to do so if given enough training steps.

Direct answer: Migrating hundreds of ad-hoc feature jobs into a centralized platform needs a staged, low-risk cutover (discovery, dependency mapping, shadow validation, gradual cutover, and only then decommissioning the old path), and the safety-critical part specifically for features shared between training and serving is making sure the migration itself doesn't introduce new training-serving skew during the transition.

Structured elaboration:
The general migration sequence: discovery (inventory every existing ad-hoc job and what it produces, since you can't safely migrate what you haven't found); dependency analysis (which models actually consume each legacy feature, so you know the blast radius of touching it); shadow validation (compute the new, centralized version alongside the legacy one without switching any model over, and diff the outputs); gradual cutover (migrate one low-risk consumer at a time, not a big-bang switch); and decommissioning the legacy pipeline only once nothing depends on it anymore, confirmed, not assumed.

For features used in both training and serving specifically, the migration needs its own safety net beyond the general plan: version both the old and new computation paths explicitly, run the new path in canary (a small percentage of live traffic, or shadow mode with no live effect) alongside the old, with automated validation comparing the two before fully cutting over, and a fast rollback path if the new implementation's outputs diverge from the old one in a way that would introduce skew a model wasn't trained to handle.

Worked example: Migrating a legacy "user activity score" feature computed by an old, undocumented batch job into the new centralized platform: shadow-running the new implementation for two weeks reveals a small but consistent numerical discrepancy versus the legacy version, traced to a subtle rounding difference; catching this in shadow mode, before any model actually depends on the new path, avoids introducing a fresh skew bug in the very act of trying to consolidate and simplify the feature infrastructure.

Trade-offs and pitfalls: The temptation to do a big-bang cutover once the new platform "looks ready" is exactly what risks silently reintroducing the training-serving-skew problem this whole migration was meant to fix; the shadow-and-canary discipline costs calendar time but is what actually protects against that specific, easy-to-overlook risk.

Approach: For each incoming order, greedily select the best available driver within max_pickup_time_seconds who has spare capacity and whose acceptance probability ≥ threshold. Score drivers by a weighted function (ETA, remaining capacity, acceptance probability). Assign immediately (online). This is simple, low-latency, and practical for dispatch.

# Pseudocode / Python-like
def greedy_dispatch(order_stream, drivers, max_pickup_time_seconds, driver_capacity, accept_prob_threshold):
    # drivers: dict driver_id -> {location, capacity_used, accept_prob, available:bool}
    # order_stream: yields orders with {order_id, location, time_received}
    for order in order_stream:
        candidates = []
        for d_id, d in drivers.items():
            if not d['available']:
                continue
            remaining = driver_capacity - d['capacity_used']
            if remaining <= 0:
                continue
            eta = estimate_travel_time(d['location'], order['location'])
            if eta > max_pickup_time_seconds:
                continue
            if d['accept_prob'] < accept_prob_threshold:
                continue
            # score: lower is better
            score = alpha*eta - beta*remaining - gamma*d['accept_prob']
            candidates.append((score, d_id, eta))
        if not candidates:
            handle_unassigned(order)  # e.g., queue, expand radius, or notify customer
            continue
        # pick best (min score)
        candidates.sort()
        best_score, best_driver_id, best_eta = candidates[0]
        assign_order_to_driver(order['order_id'], best_driver_id)
        drivers[best_driver_id]['capacity_used'] += 1
        if drivers[best_driver_id]['capacity_used'] >= driver_capacity:
            drivers[best_driver_id]['available'] = False

Key concepts:

• Online decision: immediate assignment without future knowledge.
• Scoring balances ETA, remaining capacity, and acceptance probability.
• Parameters alpha/beta/gamma tune policy.

Time/Space complexity:

• Per order: O(D log D) if sorting candidates, or O(D) with single-pass min; D = #drivers. Space O(D) for candidate list.

Edge cases:

• No candidate within threshold: implement fallback (wait queue, increase radius, guaranteed dispatch with surge).
• Rapid driver state changes (concurrency): require atomic updates / optimistic locking.
• Acceptance probabilities can be stale—update with live signals.

When greedy fails / better algorithms:

• High contention: greedy can block better future matchings (e.g., using capacity for low-value order). Use batched matching (periodic windowed optimization) solved via max-weight bipartite matching or min-cost flow to global-optimize assignments.
• Stochastic arrivals / uncertainty: use online stochastic optimization or prophet/secretary algorithms with learned thresholds.
• Strategic objectives (latency + fairness + revenue): use integer programming / constrained MDP, or RL-based dispatch policies trained in simulation to trade off long-term metrics.
• Scalability with complex constraints (multi-stop routes, time windows): use route-aware assignment (VRP heuristics) rather than per-order greedy.

Recommendation: Start with greedy for low latency; monitor regret metrics (missed high-value matches). If regret grows or KPIs suffer, migrate to hybrid: short batching windows + fast solver or learned policy (RL/heuristic) for near-optimal online matching.

Direct answer

The model improves without raw data ever leaving the device by having each device compute a small, local update (not a copy of its data) and sending only that update, clipped and noised for privacy, into a secure aggregation protocol that combines many devices' updates before the server sees anything. Validation without inspecting individual training data shifts entirely onto proxy signals: aggregate, privacy-preserving metrics across cohorts, canary devices that opt into more visible QA, and a staged rollout with instant rollback, since you're validating the update's effect rather than the data that produced it.

Structured elaboration

Architecture

flowchart TD
A[On-device usage signals] --> B[Local adapter fine-tune: LoRA-style]
B --> C[Clip and add DP noise to update]
C --> D[Secure aggregation across devices]
D --> E[Server: aggregate and validate new global adapter]
E --> F{Offline and canary-cohort checks pass}
F -->|No| G[Discard update]
F -->|Yes| H[Sign and stage rollout]
H --> I[Canary devices]
I --> J[Fleet-wide rollout]
J --> A
H --> K[Revert to last-known-good on regression]

Key design decisions and why

1. Personalization happens through a small adapter, not the full model. Rather than fine-tuning an entire model on-device (expensive in compute and storage), a lightweight adapter (a small set of additional parameters using a parameter-efficient method like LoRA, which trains a low-rank update instead of the full weight matrix) captures the user-specific signal. This keeps on-device training fast enough for phone-class hardware and keeps the update small enough to transmit cheaply.
2. Every local update is clipped and noised before it leaves the device. Clipping bounds how much any single device's update can influence the aggregate (protecting against one device dominating or an outlier skewing the result); adding calibrated noise gives a differential-privacy guarantee (a formal bound on how much any single user's data could be inferred from the published aggregate).
3. Secure aggregation means the server only ever sees a sum, never an individual update. Using a secure aggregation protocol, devices' encrypted updates combine such that the server can decrypt the aggregate total but not any individual device's contribution, so even a compromised or curious server can't reconstruct one user's local adapter from what it receives.
4. Validation moves from "inspect the data" to "measure the effect." Since you can't look at what a specific user's device learned from, you validate using: offline replay against a held-out benchmark, aggregate metrics computed the same privacy-preserving way as training (so no individual is exposed), and a canary cohort of devices that have explicitly opted into contributing more detailed, still-aggregated QA telemetry.
5. Every rollout is staged and instantly reversible. Because you can't manually inspect why an update might be misbehaving for a given user (no access to their data), the operational answer is to bound the blast radius (canary first, small percentage ramp) and make rollback automatic on a metric regression, rather than trying to debug the specific cause before reverting.

Worked example

Differential privacy's guarantee weakens (the privacy "budget" $\varepsilon$ grows) every time a device contributes another round of updates, and the arithmetic is what makes "how many rounds can this run before privacy is meaningfully spent" concrete. Under basic composition (the simplest, worst-case-additive way of combining the privacy cost across repeated releases), the total privacy loss after $T$ rounds, each with per-round budget $\varepsilon_{\text{round}}$, is:

If a device participates in one aggregation round per day for a quarter (about 30 rounds is a conservative stand-in for how often meaningful personalization updates are needed) at a per-round budget of $\varepsilon_{\text{round}} = 0.1$, basic composition puts the cumulative budget at $\varepsilon_{\text{total}} = 3.0$. An $\varepsilon$ around 1 is generally considered a reasonably strong privacy guarantee, and by 3.0 the guarantee has weakened meaningfully; this is the concrete reason the system needs either a hard cap on how many rounds a device participates in per period, a tighter per-round $\varepsilon_{\text{round}}$, or an advanced composition method (which gives a tighter, sub-linear bound than simple multiplication, at the cost of a more complex accounting mechanism) rather than assuming a device can keep contributing updates indefinitely at a fixed per-round cost.

Trade-offs & pitfalls

• The privacy/utility trade-off is not free: clipping and noising the update to get a strong DP guarantee reduces how much signal makes it into the aggregate, so a stronger privacy guarantee directly costs some personalization quality; the right operating point depends on how sensitive the specific use case is.
• Not being able to inspect training data also means you can't directly debug "why did this specific update go wrong" the way you could with centrally logged data; the mitigation (canary cohorts, staged rollout, instant rollback) is an operational answer, not a technical fix for the inspection gap, and it's a real limitation worth naming rather than glossing over.
• Secure aggregation and DP noise both add real latency and complexity to each update round; a common mistake is under-provisioning for this and having personalization rounds take so long that they stop reflecting recent user behavior by the time they land.
• A rollout that's staged only by percentage of devices, without also stratifying by device or usage segment, can miss a regression that only affects one segment (a particular hardware generation, a particular locale) until it's already at a meaningful percentage of the fleet; canary cohorts should be chosen to represent that diversity, not just be "the first 1%."
• On-device incremental learning without a hard cap on update frequency or magnitude risks a device's local adapter drifting in a way that's hard to detect centrally, since the whole point of the design is that the server never sees the individual adapter directly; this is why clipping and bounded learning rates on-device matter as much as the server-side aggregation safeguards.

Direct answer
Bias is the error from a model's assumptions being too simple to capture the true relationship in the data; variance is the error from a model being too sensitive to the specific training sample it happened to see. Expected generalization error decomposes into (roughly) bias squared plus variance plus irreducible noise, so both matter and pulling one down often pushes the other up.

Structured elaboration

• High bias, low variance: a linear model fit to a clearly non-linear relationship (predicting house price from square footage alone with a straight line), or a shallow decision tree on a task with complex interactions. It underfits: training and validation error are both mediocre and close together.
• Low bias, high variance: a deep, unpruned decision tree, a k-nearest-neighbors model with k=1, or a large neural network trained too long without regularization on a small dataset. It overfits: training error is very low, validation error is noticeably worse.
• Practical indicators: the gap between training and validation error is the single most useful signal. A small gap with both errors high points to bias; a large gap with low training error points to variance. Learning curves (error versus training-set size) sharpen this: a bias-limited model's curves converge to a high error floor even with more data; a variance-limited model's validation curve keeps improving as you add data.
• Mitigation by regime: for high bias, increase model capacity (deeper trees, more layers, richer features, less regularization) or reduce built-in assumptions. For high variance, add regularization (L1/L2, dropout), collect more data, simplify the model, or average multiple models via ensembling (bagging in particular targets variance directly).

Worked example
Fit polynomial regression of degree 1 (a line) versus degree 15 to a dataset generated from a mildly curved true function plus noise. The degree-1 fit will show high, roughly-equal train and validation error (high bias). The degree-15 fit will show near-zero training error but validation error that swings wildly with which points happened to land in the training set (high variance): refitting on a different random subsample of the same data would give a visibly different curve, which is the concrete meaning of "sensitive to the training sample."

Trade-offs & pitfalls
The gap-based diagnostic can mislead you if your validation set is small or unrepresentative: normal sampling noise can look like a variance problem. Always sanity-check with k-fold cross-validation (multiple splits) before concluding you have a real bias or variance issue rather than a one-off unlucky split. Also, adding regularization to fix variance always costs a little bias; the goal is minimizing total error, not chasing zero variance.

Direct answer

The DAG runs data validation, then three parallel feature-extraction tasks, then training, then evaluation, and conditionally triggers deployment only if the new metric beats the stored baseline: the conditional branch is what makes this DAG genuinely ML-specific rather than a generic four-step pipeline.

Structured elaboration

from airflow.decorators import dag, task
from datetime import datetime

@dag(schedule="@daily", start_date=datetime(2026, 1, 1), catchup=False)
def retrain_and_conditionally_deploy():

    @task
    def validate_data():
        # raise on failure -- this halts the whole DAG downstream
        return {"validated_at": datetime.utcnow().isoformat()}

    @task
    def extract_features(source: str):
        # runs once per source, in parallel with the other two calls below
        return {"source": source, "row_count": 100_000}

    @task
    def train_model(feature_batches: list[dict]):
        return {"model_uri": "s3://models/candidate-v42", "trained_on": feature_batches}

    @task
    def evaluate_model(model_info: dict):
        candidate_metric = 0.87  # from real evaluation against a held-out set
        return {"model_uri": model_info["model_uri"], "metric": candidate_metric}

    @task.branch
    def decide_deployment(eval_result: dict):
        import json
        # baseline metric stored in object storage from the last promoted model
        baseline_metric = json.loads(_read_baseline_from_object_storage())["metric"]
        return "deploy_model" if eval_result["metric"] > baseline_metric else "skip_deployment"

    @task
    def deploy_model(eval_result: dict):
        pass  # promote to registry, trigger canary rollout

    @task
    def skip_deployment(eval_result: dict):
        pass  # log the skip, no-op otherwise

    def _read_baseline_from_object_storage():
        return '{"metric": 0.85}'

    validated = validate_data()
    feature_batches = [extract_features(src) for src in ["source_a", "source_b", "source_c"]]
    for fb in feature_batches:
        validated >> fb  # each extraction task depends on validation succeeding
    model_info = train_model(feature_batches)
    eval_result = evaluate_model(model_info)
    branch = decide_deployment(eval_result)
    branch >> [deploy_model(eval_result), skip_deployment(eval_result)]

retrain_and_conditionally_deploy()

Conditional branching: the @task.branch decorator makes decide_deployment return the TASK ID to run next rather than a value, which is how Airflow's TaskFlow API expresses "run exactly one of these two downstream paths": deploy_model runs only if the candidate beats the baseline; skip_deployment runs (and the other is marked skipped, not failed) otherwise, so the DAG run still shows a clean success either way.

Worked example

The three parallel feature-extraction tasks each depend on validate_data succeeding first (fan-out from one upstream task to three parallel downstream tasks), and train_model depends on ALL THREE completing (fan-in) before it can run: Airflow's dependency graph handles this fan-out/fan-in pattern natively via the >> operator applied to a list.

Trade-offs & pitfalls

A real production version needs to handle a genuinely delicate edge case this sketch glosses over: what happens if the baseline metric itself is unavailable (the object-storage read fails)? Silently treating that as "always deploy" or "never deploy" are both wrong; the correct behavior is failing the branch task loudly rather than defaulting either way, since a silent wrong default here is exactly the kind of bug that ships a bad model with nobody noticing until much later.