End-to-End ML System Design Questions

First split the problem into two buckets

1. True optimization issues: the model is mathematically harder to train at larger scale.
2. Distributed systems issues: the parallel run is changing the effective training behavior.

Things I would investigate

• Effective batch size and learning rate scaling. A bigger world size often means a larger batch, which can require LR tuning.
• Data sharding. I would verify that each sample is seen once, not duplicated or skipped.
• Gradient synchronization. I would check that all ranks contribute the same gradients and that there are no stale or dropped updates.
• Precision and numerics. Mixed precision can introduce overflow, underflow, or different rounding behavior at scale.
• Stateful layers like batch normalization, which can behave differently across replicas.

How I would separate them
I would compare small-scale and large-scale runs with the same global batch and fixed seeds, then inspect gradient norms, loss curves, and sample hashes.

Worked example
If 8 workers train with a batch of 256 each, the global batch becomes 2048. If loss gets noisier only after that change, it may be an optimization tuning problem. If the run only degrades when messages cross nodes, it is more likely a communication or synchronization problem.

That distinction tells me whether to tune the optimizer or fix the distributed stack.

Redesign the input path
I would separate expensive preprocessing from the training loop and make the input stream sharded, cached, and deterministic.

Main changes

• Compact many small object-store files into larger shards so reads are sequential instead of chatty.
• Precompute expensive transforms once, then store the transformed output or an intermediate cache.
• Use a manifest that records shard order, sample IDs, and versioned transform code.
• Prefetch data to local disk or memory so GPUs do not wait on network reads.
• Keep lightweight, deterministic transforms in the training worker so results are reproducible.

Worked example
If the dataset is 10,000 files of 2 MB each, that is 20 GB spread across many requests. Repacking into 200 shards of 100 MB keeps the same 20 GB, but cuts file open and listing overhead dramatically.

Why this helps
The GPUs stay busy because decoding and network latency happen ahead of time. Reproducibility comes from versioned manifests and fixed seeds. Debuggability comes from keeping raw sample IDs so a bad example can be replayed exactly.

This is usually the highest leverage change when preprocessing time dominates training time.

Direct answer

Point-in-time correctness means that when a feature value is joined to a label for training, only the feature value that was actually known at or before the label's timestamp is used, never a value computed or updated afterward. Get it wrong and you leak information from the future into training, which inflates offline metrics in a way that does not hold once the model is scored on data it genuinely could not have seen in advance.

Structured elaboration

The join rule: for each entity and label timestamp, look up the most recent feature value with a feature timestamp less than or equal to the label timestamp, never the feature's later or current value.

Why naive joins get this wrong: a common shortcut joins on the entity alone against "the current feature table," which silently uses whatever the feature happens to be as of whenever the join runs, not as of when the label was actually generated. This is one of the most common sources of target leakage (information about the outcome, or about a later point in time, leaking into the inputs used to predict it) in tabular pipelines, and it is invisible in the code; it only shows up if you inspect timestamps directly.

What actually goes wrong downstream: the model looks excellent offline, because it is effectively being given a preview of information from after the decision point, and then underperforms once served, because production inference can only ever see features known up to that moment, never future information. Teams sometimes chase this gap for weeks assuming it's a modeling or infrastructure problem when it's really a training-data construction problem.

Applied variants of the same requirement: late-arriving events make "the latest known value" itself ambiguous. A mobile event that occurred on one day may not land in the feature store until up to 48 hours later, so a strict point-in-time join has to use the value that was actually materialized and available by the label timestamp, not merely the value whose real-world event happened by then; otherwise the same leakage reappears in a subtler form. Incremental daily feature jobs are the concrete mechanism this shows up in: if a daily batch job computes "features as of end of day," a label generated partway through the next day must join against the prior day's completed batch, not a still-in-progress one, even though the in-progress one will eventually contain a fresher, more complete value.

How to enforce it structurally: compute point-in-time snapshots from an append-only, timestamped feature log rather than reading a mutable "current state" table, and build the training join against those snapshots, so the same construction procedure works whether the feature refresh cadence is real-time, hourly, or once daily.

Worked example

Consider a churn label generated for a user at label time = day 10, midnight. The user's "days since last login" feature is computed by a daily batch job that runs at the end of each day, but due to the 48-hour late-arrival window for mobile events, a given day's count only finalizes two days later. A naive join using "whatever value is currently in the feature table" at label time would pull an incomplete day-9 value, since some day-9 mobile events haven't arrived yet as of day 10 midnight, while a later re-materialization on day 11 would produce a more complete day-9 value after those events settle. Point-in-time correctness means the training join must use the day-9 value as it was computable at day 10 midnight, using only events that had already landed by then, not the more complete day-11 version, even though the day-11 version is, in an ordinary sense, "more correct" data. Using the day-11 version at training time is leakage: production inference at day 10 could never have seen those late-arriving events either.

As a hypothetical to make the mechanism concrete: suppose 12.5% of a day's mobile events arrive in the 24 to 48 hour late window. Then ignoring that window would overstate feature completeness for about one in eight users at label time ($12.5\% \approx 1/8$), meaning the offline evaluation would be quietly cheating on roughly that same fraction of rows.

Trade-offs & pitfalls

• Validating with a random train/test split does not surface this kind of leakage at all, since both splits are drawn from the same "current state" table; only a strictly time-based holdout, joined the same point-in-time way as training, will surface it.
• Being maximally strict, only treating a feature as usable once every late event has fully settled, trades leakage safety for freshness: it can mean training on features that are, in production, always somewhat staler than what the strict join implies, so the join rule needs to match what serving can actually see, not an idealized fully-settled value.
• Incremental daily jobs make this easy to get subtly wrong at day boundaries and across time zones; a job that materializes "as of end of day" needs an explicit, unambiguous cutoff convention, or the join silently mixes settlement windows in ways that are hard to debug later.

Direct answer

Label skew is when the distribution of labels in a training set doesn't match the real population the model will be scored against, for example a fraud model trained on data artificially balanced to 10% positive when real traffic is 0.5% positive. Feature skew is when an input feature's distribution at training time differs from what the model actually receives at serving time, for example a feature computed from batch logs during training but computed live, with a different lookback window, during serving. Both quietly bias a model because it learns decision boundaries calibrated to a distribution it will never actually see in production, and nothing about training loss or offline accuracy flags this on its own.

Structured elaboration

Sources of label skew: sampling procedures that oversample positives for training convenience; selection bias in which examples get a label at all (only labeling transactions a human analyst chose to review, which already correlates with what made them look suspicious); and label lag, where positive labels for recent examples haven't arrived yet, making recent data look artificially clean.

Sources of feature skew: two different code paths computing "the same" feature for training versus serving (this specific case is called training-serving skew); the feature's real-world distribution genuinely shifting over time between when training data was collected and when the model serves; and default or fallback values behaving differently online, where serving substitutes a placeholder for a missing value that training data never actually contained.

What to actually do about it:

1. Compare training-set label prevalence against a recent, unbiased sample of the true population prevalence, not just the labeled set itself, and correct for known sampling procedures explicitly (reweight examples inversely to their sampling probability, or recalibrate the model's output probabilities against the true base rate) rather than trusting the training set's balance at face value.
2. Compute per-feature distribution statistics on both the training set and a live production sample, using the exact same feature-computation code for both, and diff them on a recurring schedule, so a divergence reflects a real shift rather than a code-path bug.
3. Audit where labels come from before trusting them, specifically checking whether the labeling process was itself conditioned on an earlier automated or human decision. This is the surest way label skew hides, since it never shows up as a data-quality problem, only as a silently wrong modeling assumption.

Worked example

Consider a fraud model with a true production positive rate of 0.5%. Suppose the training set keeps all positives and downsamples negatives to reach a 10% positive rate for training convenience. In a random 100,000-transaction sample, the true rate implies about $100{,}000 \times 0.005 = 500$ positives and 99,500 negatives. To reach a 10% training positive rate while keeping all 500 positives, negatives must be reduced to $n$ such that:

$\frac{500}{500+n} = 0.10 \;\Rightarrow\; 500 = 0.10 \times (500+n) \;\Rightarrow\; 5000 = 500 + n \;\Rightarrow\; n = 4500$

So negatives go from 99,500 down to 4,500, a keep-rate of $4500/99500 \approx 4.52\%$. If the model's raw output is used directly against a fixed business threshold, its probabilities will be systematically too high, since they're calibrated to a 10% base-rate world, not the true 0.5% one. Since positives were fully retained (keep-rate 1.0) and negatives were kept at rate $s \approx 0.0452$, the standard correction for this kind of downsampling rescales the raw score back to the true base rate:

$p_{\text{corrected}} = \frac{s \cdot p_{\text{model}}}{s \cdot p_{\text{model}} + (1-p_{\text{model}})}$

If a transaction scores $p_{\text{model}}=0.5$ under the training-calibrated model, the corrected probability is:

$p_{\text{corrected}} = \frac{0.0452 \times 0.5}{0.0452 \times 0.5 + 0.5} = \frac{0.0226}{0.5226} \approx 0.0432$

So a transaction that looks like a coin flip under the training distribution is really only about 4.3% likely to be fraud in the true population, which is the concrete reason applying a naive 50% cutoff directly to raw model output, without this correction, over-flags a large share of legitimate transactions.

Trade-offs & pitfalls

• Validating against a holdout drawn from the same skewed training process doesn't catch label skew at all; the validation set needs to reflect the true population, or be explicitly reweighted to it, not just be a random split of the same biased sample.
• The correction above assumes negatives were downsampled uniformly at random; if the downsampling was non-random (keeping whichever negatives were easiest to log), the correction's assumption breaks and the fix silently fails.
• Feature skew caused by two independent code paths is not fixed by any statistical correction; it needs a single shared feature-computation path, not a calibration formula.
• Chasing an exact match between training and production prevalence is itself a trap in genuinely rare-event problems: forcing training data down to the true base rate can leave too few positive examples for the model to learn from at all, so downsampling combined with an explicit, documented correction is usually the better choice than pretending the imbalance doesn't exist.

Approach
I would make checkpointing a consistent snapshot of everything needed to resume training, not just model weights. That means saving model parameters, optimizer state, learning-rate scheduler state, random number generator seeds, current epoch and batch cursor, and any data sampler state.

Design

• Write checkpoints atomically: first write to temporary storage, validate checksums, then publish a manifest pointer.
• Keep checkpoints incremental when possible, but always make the latest one self-contained for fast restart.
• Trigger an immediate checkpoint on spot interruption notice, then resume from the last good manifest.
• Store code version, config, and data version so the run is reproducible.

Worked example
If I checkpoint every 10 minutes and a preemption happens at minute 37, the restart only redoes at most 7 minutes of work, not the whole job.

Why this works
The manifest guarantees consistency, the RNG and sampler state keep replay deterministic, and the atomic publish prevents half-written checkpoints from being used.