End-to-End ML System Design Questions

Direct answer

Versioning and lineage matter because a production model is a claim about correctness that can only be re-examined, debugged, or audited if you can point to the exact data behind it. Reconstructing that six months later requires an immutable, addressable chain, captured automatically at training time, of the raw data snapshot, the feature-transformation code version, and the exact training configuration, tied together under one identifier in a model registry.

Structured elaboration

Three concrete reasons this matters:

1. Debugging: when a production model misbehaves, the fastest diagnostic is comparing current serving data to the exact data it trained on, which only works if that training data is still identifiable.
2. Compliance and audit: regulated domains (credit, hiring, healthcare) may require showing exactly what data trained a decision-making model, sometimes long after the fact.
3. Safe iteration: retraining or rolling back needs a known-good prior state to diff against or revert to; without lineage, "roll back" means guessing.

What needs to be captured at training time, not reconstructed after the fact:

• A content-addressed or timestamped snapshot of the raw training data, not a live pointer to a table that keeps changing underneath you.
• The exact version of the feature-transformation code (a source-control commit hash or equivalent).
• The training configuration: hyperparameters and random seed.
• The resulting model artifact, all tied together under one version identifier in a model registry entry.

Reconstruction, six months later, concretely: given a model version identifier, the registry entry resolves to the training data snapshot's storage location (which must not have been deleted or overwritten under retention policy) and the code commit. Checking out that commit and rerunning the deterministic feature pipeline against that exact snapshot should regenerate the same training set. If the underlying source data is mutable (a source table updated in place), lineage additionally has to capture when the snapshot was taken relative to that source, since "the table" is not a stable reference on its own, only a point-in-time copy of it is.

Worked example

Suppose a model registry entry for a fraud model records: a data-snapshot path, a code commit hash, a random seed of 42, and a feature-definition version. Reconstructing it six months later, on today's date, looks like this:

1. Confirm the snapshot still exists under the retention policy. If retention is set to 400 days and the snapshot is roughly 6 months (about 180 days) old, it is safely inside the window and has not been purged.
2. Check out the recorded code commit.
3. Re-run the feature pipeline at the recorded feature-definition version against that snapshot with the recorded seed.
4. Compare the output's row count and a stored checksum against what the registry recorded at training time.

If the checksum matches, the reconstruction is exact, not merely plausible. If it doesn't match, that mismatch is itself the useful signal: something in the chain (an unpinned dependency, a transformation that wasn't actually deterministic) broke reproducibility, which is precisely the failure this design exists to catch.

Trade-offs & pitfalls

• A common wrong turn is versioning the model artifact but not the data: that lets you reload old weights, but not explain or regenerate why they are what they are.
• Storing a pointer to "the current production table" instead of an immutable snapshot silently breaks reconstruction the moment that table is updated in place; this is the single most common lineage failure in practice.
• Keeping every historical snapshot indefinitely at full resolution has a real, ongoing storage cost; the usual resolution is a retention window sized to the longest realistic audit or debugging need, with older snapshots summarized (schema and statistics) rather than kept byte for byte.
• Lineage that is captured retroactively, guessed from logs after the fact, is much weaker than lineage captured automatically at training time; if capture requires a manual step, it will eventually be skipped under deadline pressure.

Direct answer

The three main ways to split a training job are data parallelism (each worker holds a full copy of the model and a different slice of the data, and workers exchange gradients to stay in sync), model or tensor parallelism (the model itself, or a single large layer, is split across workers, which exchange intermediate activations mid-computation), and pipeline parallelism (the model's layers are grouped into sequential stages across workers, which exchange only the activations at each stage boundary). Which one you reach for depends on whether the bottleneck is dataset size, model size, or both.

Structured elaboration

In practice, training very large models typically combines all three, sometimes called 3D parallelism: tensor parallelism within a machine over a fast local interconnect, pipeline parallelism across machines within a group, and data parallelism across groups, because each strategy's communication pattern fits a different tier of network speed: the fastest links carry the most frequent, tensor-parallel traffic, and the slowest cross-group links carry the least frequent, data-parallel gradient synchronization.

Worked example

Consider an 8-billion-parameter model stored at 16-bit precision (2 bytes per parameter): weights alone need roughly $8\times10^9 \times 2 = 16 \text{ GB}$. A common optimizer keeps additional per-parameter state that can add another 2 to 3x on top of the weights; at 3x, total memory needed is $16 \times 3 = 48$ GB. If each device has 24 GB of memory, this does not fit on one device at all, and no amount of extra data-parallel workers changes that, since every data-parallel worker would still need to hold the full 48 GB. Splitting the model across 2 devices (model or tensor parallelism) brings the per-device share to $48/2 = 24$ GB, right at the edge of fitting; that arithmetic, not preference, is what forces model parallelism here.

Separately, consider the communication cost that makes pure data parallelism expensive at this scale: an all-reduce of an 8-billion-parameter gradient tensor at 2 bytes each moves 16 GB of gradient data across workers every step. At a hypothetical 100 gigabit-per-second (12.5 GB/s) interconnect, transferring that volume alone takes on the order of:

$16 / 12.5 \approx 1.28 \text{ seconds of network time per step, in the naive case}$

Efficient ring all-reduce implementations move closer to $2\times(N-1)/N$ of that volume per node rather than the full amount, but the underlying point holds: gradient-synchronization cost scales with model size, not batch size, which is why pure data parallelism becomes network-bound long before it becomes compute-bound on very large models, and why the other two strategies exist at all.

Trade-offs & pitfalls

• Reaching for model parallelism when the real bottleneck is dataset throughput and the model fits comfortably in memory adds mid-layer communication overhead for no benefit, when data parallelism's simpler gradient-only synchronization would have been enough.
• Pipeline parallelism's bubble problem, workers idling while waiting on earlier or later stages, shrinks but never disappears with more micro-batches; more micro-batches reduce the idle fraction but increase per-batch overhead, so the choice is a tuning trade-off, not a solved problem.
• Data parallelism's communication cost scales with model size, so simply adding more data-parallel workers to speed up training on a very large model eventually stops helping once the job becomes network-bound.
• Combining all three strategies is powerful but adds real engineering and debugging complexity around exactly which workers communicate with which, in what order, at what precision; reaching for that combination before confirming plain data parallelism is actually insufficient is a common overengineering trap.

Direct answer

Label skew is when the distribution of labels in a training set doesn't match the real population the model will be scored against, for example a fraud model trained on data artificially balanced to 10% positive when real traffic is 0.5% positive. Feature skew is when an input feature's distribution at training time differs from what the model actually receives at serving time, for example a feature computed from batch logs during training but computed live, with a different lookback window, during serving. Both quietly bias a model because it learns decision boundaries calibrated to a distribution it will never actually see in production, and nothing about training loss or offline accuracy flags this on its own.

Structured elaboration

Sources of label skew: sampling procedures that oversample positives for training convenience; selection bias in which examples get a label at all (only labeling transactions a human analyst chose to review, which already correlates with what made them look suspicious); and label lag, where positive labels for recent examples haven't arrived yet, making recent data look artificially clean.

Sources of feature skew: two different code paths computing "the same" feature for training versus serving (this specific case is called training-serving skew); the feature's real-world distribution genuinely shifting over time between when training data was collected and when the model serves; and default or fallback values behaving differently online, where serving substitutes a placeholder for a missing value that training data never actually contained.

What to actually do about it:

1. Compare training-set label prevalence against a recent, unbiased sample of the true population prevalence, not just the labeled set itself, and correct for known sampling procedures explicitly (reweight examples inversely to their sampling probability, or recalibrate the model's output probabilities against the true base rate) rather than trusting the training set's balance at face value.
2. Compute per-feature distribution statistics on both the training set and a live production sample, using the exact same feature-computation code for both, and diff them on a recurring schedule, so a divergence reflects a real shift rather than a code-path bug.
3. Audit where labels come from before trusting them, specifically checking whether the labeling process was itself conditioned on an earlier automated or human decision. This is the surest way label skew hides, since it never shows up as a data-quality problem, only as a silently wrong modeling assumption.

Worked example

Consider a fraud model with a true production positive rate of 0.5%. Suppose the training set keeps all positives and downsamples negatives to reach a 10% positive rate for training convenience. In a random 100,000-transaction sample, the true rate implies about $100{,}000 \times 0.005 = 500$ positives and 99,500 negatives. To reach a 10% training positive rate while keeping all 500 positives, negatives must be reduced to $n$ such that:

$\frac{500}{500+n} = 0.10 \;\Rightarrow\; 500 = 0.10 \times (500+n) \;\Rightarrow\; 5000 = 500 + n \;\Rightarrow\; n = 4500$

So negatives go from 99,500 down to 4,500, a keep-rate of $4500/99500 \approx 4.52\%$. If the model's raw output is used directly against a fixed business threshold, its probabilities will be systematically too high, since they're calibrated to a 10% base-rate world, not the true 0.5% one. Since positives were fully retained (keep-rate 1.0) and negatives were kept at rate $s \approx 0.0452$, the standard correction for this kind of downsampling rescales the raw score back to the true base rate:

$p_{\text{corrected}} = \frac{s \cdot p_{\text{model}}}{s \cdot p_{\text{model}} + (1-p_{\text{model}})}$

If a transaction scores $p_{\text{model}}=0.5$ under the training-calibrated model, the corrected probability is:

$p_{\text{corrected}} = \frac{0.0452 \times 0.5}{0.0452 \times 0.5 + 0.5} = \frac{0.0226}{0.5226} \approx 0.0432$

So a transaction that looks like a coin flip under the training distribution is really only about 4.3% likely to be fraud in the true population, which is the concrete reason applying a naive 50% cutoff directly to raw model output, without this correction, over-flags a large share of legitimate transactions.

Trade-offs & pitfalls

• Validating against a holdout drawn from the same skewed training process doesn't catch label skew at all; the validation set needs to reflect the true population, or be explicitly reweighted to it, not just be a random split of the same biased sample.
• The correction above assumes negatives were downsampled uniformly at random; if the downsampling was non-random (keeping whichever negatives were easiest to log), the correction's assumption breaks and the fix silently fails.
• Feature skew caused by two independent code paths is not fixed by any statistical correction; it needs a single shared feature-computation path, not a calibration formula.
• Chasing an exact match between training and production prevalence is itself a trap in genuinely rare-event problems: forcing training data down to the true base rate can leave too few positive examples for the model to learn from at all, so downsampling combined with an explicit, documented correction is usually the better choice than pretending the imbalance doesn't exist.

Direct answer

Point-in-time correctness means that when a feature value is joined to a label for training, only the feature value that was actually known at or before the label's timestamp is used, never a value computed or updated afterward. Get it wrong and you leak information from the future into training, which inflates offline metrics in a way that does not hold once the model is scored on data it genuinely could not have seen in advance.

Structured elaboration

The join rule: for each entity and label timestamp, look up the most recent feature value with a feature timestamp less than or equal to the label timestamp, never the feature's later or current value.

Why naive joins get this wrong: a common shortcut joins on the entity alone against "the current feature table," which silently uses whatever the feature happens to be as of whenever the join runs, not as of when the label was actually generated. This is one of the most common sources of target leakage (information about the outcome, or about a later point in time, leaking into the inputs used to predict it) in tabular pipelines, and it is invisible in the code; it only shows up if you inspect timestamps directly.

What actually goes wrong downstream: the model looks excellent offline, because it is effectively being given a preview of information from after the decision point, and then underperforms once served, because production inference can only ever see features known up to that moment, never future information. Teams sometimes chase this gap for weeks assuming it's a modeling or infrastructure problem when it's really a training-data construction problem.

Applied variants of the same requirement: late-arriving events make "the latest known value" itself ambiguous. A mobile event that occurred on one day may not land in the feature store until up to 48 hours later, so a strict point-in-time join has to use the value that was actually materialized and available by the label timestamp, not merely the value whose real-world event happened by then; otherwise the same leakage reappears in a subtler form. Incremental daily feature jobs are the concrete mechanism this shows up in: if a daily batch job computes "features as of end of day," a label generated partway through the next day must join against the prior day's completed batch, not a still-in-progress one, even though the in-progress one will eventually contain a fresher, more complete value.

How to enforce it structurally: compute point-in-time snapshots from an append-only, timestamped feature log rather than reading a mutable "current state" table, and build the training join against those snapshots, so the same construction procedure works whether the feature refresh cadence is real-time, hourly, or once daily.

Worked example

Consider a churn label generated for a user at label time = day 10, midnight. The user's "days since last login" feature is computed by a daily batch job that runs at the end of each day, but due to the 48-hour late-arrival window for mobile events, a given day's count only finalizes two days later. A naive join using "whatever value is currently in the feature table" at label time would pull an incomplete day-9 value, since some day-9 mobile events haven't arrived yet as of day 10 midnight, while a later re-materialization on day 11 would produce a more complete day-9 value after those events settle. Point-in-time correctness means the training join must use the day-9 value as it was computable at day 10 midnight, using only events that had already landed by then, not the more complete day-11 version, even though the day-11 version is, in an ordinary sense, "more correct" data. Using the day-11 version at training time is leakage: production inference at day 10 could never have seen those late-arriving events either.

As a hypothetical to make the mechanism concrete: suppose 12.5% of a day's mobile events arrive in the 24 to 48 hour late window. Then ignoring that window would overstate feature completeness for about one in eight users at label time ($12.5\% \approx 1/8$), meaning the offline evaluation would be quietly cheating on roughly that same fraction of rows.

Trade-offs & pitfalls

• Validating with a random train/test split does not surface this kind of leakage at all, since both splits are drawn from the same "current state" table; only a strictly time-based holdout, joined the same point-in-time way as training, will surface it.
• Being maximally strict, only treating a feature as usable once every late event has fully settled, trades leakage safety for freshness: it can mean training on features that are, in production, always somewhat staler than what the strict join implies, so the join rule needs to match what serving can actually see, not an idealized fully-settled value.
• Incremental daily jobs make this easy to get subtly wrong at day boundaries and across time zones; a job that materializes "as of end of day" needs an explicit, unambiguous cutoff convention, or the join silently mixes settlement windows in ways that are hard to debug later.

What I would inspect first
I would start with per-step timing on the slow worker versus the rest of the cluster. If the code, model, and data are unchanged, a single lagging worker is usually a host or systems issue, not an ML issue.

Checks in order

• GPU utilization and memory bandwidth on the slow node
• Data loader wait time and local disk throughput
• CPU steal, thermal throttling, and noisy neighbors
• Network errors, packet drops, and collective communication logs
• Kernel and container logs for retries or hardware faults

Mitigation
If the worker is clearly abnormal, I would cordon it, move the job to a fresh node, and keep the training moving. If the slowdown comes from input starvation, I would reduce preprocessing on that host, increase local caching, or lower dataloader contention.

Worked example
If most workers take 180 ms per step but one takes 420 ms and spends 250 ms waiting on input, the bottleneck is the input path, not the model.

The goal is to isolate the bad actor quickly and avoid letting one slow node stall the whole synchronous job.