Model Training Infrastructure and Distributed Training Questions

Direct answer

Object storage (S3/GCS) and a shared filesystem (NFS, Lustre) trade off differently on throughput, consistency, metadata-operation performance, and operational simplicity for training data and checkpoints: object storage offers near-unlimited scale and simple operations at the cost of higher per-request latency, weaker filesystem semantics, and notably slow metadata operations, while a shared filesystem gives POSIX-like semantics, fast metadata operations, and lower per-file latency at the cost of needing dedicated, harder-to-scale infrastructure.

Structured elaboration

• Throughput characteristics: object storage scales aggregate throughput very well across many parallel workers and large sequential reads, but individual small-object requests carry meaningfully higher latency; Lustre/NFS-class shared filesystems offer lower per-file latency and better small-file/random-access performance.
• Consistency: object storage now largely offers strong read-after-write consistency on major providers (worth verifying for the specific provider/API); shared filesystems typically offer POSIX consistency semantics, which some checkpoint/restore code implicitly relies on.
• Metadata operations: object storage has no true directory hierarchy: a "list objects with this prefix" call is a linear scan over a flat key namespace and gets slower (and more expensive, since it's billed per request/page) as the number of objects under a prefix grows into the millions, which is a real problem for a sharded-checkpoint format or a dataset with many small per-sample files, where enumerating "what's here" is itself a bottleneck; there is also no cheap stat-equivalent atomic rename, or file-existence check as fast as a POSIX call. A shared filesystem supports fast, O(1)-ish stat, ls, and rename operations on individual files/directories, because it maintains a real filesystem metadata tree, which matters heavily for workloads (like sharded checkpoints with thousands of small files, or datasets with millions of small sample files) that do frequent listing, existence-checking, or renaming rather than pure bulk sequential read/write.
• Operational complexity: object storage is fully managed, effectively infinite capacity, pay-per-use; a shared filesystem like Lustre requires provisioning and operating dedicated storage server infrastructure.
• Cost model: object storage typically has lower baseline cost per GB stored plus request/egress charges (which directly punishes metadata-heavy access patterns, since every list/head call is a billed request); shared filesystems often cost more per GB provisioned but avoid per-request charges.

Worked example

A training pipeline reading large (multi-GB) sharded dataset files sequentially is a good fit for object storage: few, large requests, no heavy metadata traffic. A dataset stored as millions of individual small per-sample files (a common ImageNet-style layout) is a poor fit for object storage specifically BECAUSE of metadata operations: listing or checking existence across millions of keys is slow and expensive per-request, whereas the same layout on Lustre/NFS resolves listing and existence checks via fast filesystem metadata lookups; the standard mitigation is packing many small samples into a few large shard files (e.g. WebDataset/TFRecord-style) specifically to avoid the metadata-operation cost object storage imposes on many-small-file layouts.

Trade-offs & pitfalls

A common design mistake is choosing storage based on familiarity rather than access pattern: forcing a workload dominated by many small random reads/writes, or heavy listing/existence-checking, onto object storage (paying per-request latency, cost, and slow flat-namespace listing repeatedly) or provisioning an expensive shared filesystem for a workload that's really just large sequential reads.

Direct answer

Cutting a distributed training pipeline's cloud cost by 50% while preserving model quality is best approached as a prioritized menu of levers ordered by expected cost-savings-to-quality-risk ratio: start with the cheapest, lowest-risk wins (spot/preemptible instances, mixed precision, right-sizing instance types) before touching anything that could meaningfully change model quality (reducing dataset size, more aggressive compression).

Structured elaboration

• Spot/preemptible instances: often 60-90% cheaper than on-demand for the same GPU type, with the cost of needing robust checkpointing and fault tolerance to handle preemption; for a training job that already checkpoints reasonably often, this is usually the single largest, lowest-quality-risk lever available.
• Mixed precision and larger effective batch via gradient accumulation: reduces wall-clock training time (fewer GPU-hours billed) with negligible to no quality impact if implemented correctly (loss scaling, appropriate LR scaling), a near-free win if not already in place.
• Right-sizing instance/GPU type: matching the GPU generation and count to the model's actual compute/memory profile rather than defaulting to the largest available instance; over-provisioned memory or compute that the workload never uses is pure waste.
• Reducing redundant/wasted compute: eliminating quota-padding in the training pipeline itself (unnecessary re-runs, overly frequent validation passes, excessive checkpoint overhead) before touching the model or data.
• Higher-risk-to-quality levers, use last and validate carefully: reducing dataset size or training steps (risks under-training), more aggressive quantization/compression during training (risks numerical instability), and reducing model size (directly risks quality) should be considered only after the lower-risk levers are exhausted, and always validated against the actual quality metric, not assumed safe.

Worked example

A training pipeline costing $100k/month on on-demand GPU instances: moving to spot instances at a typical 70% discount for the bulk of training (assuming acceptable preemption handling) alone could reduce cost to roughly $30k for that portion; combined with mixed precision cutting wall-clock time by, say, 30-40% (fewer billed GPU-hours for the same work), the compounded reduction plausibly reaches the 50% target without touching dataset size or model architecture at all, which is exactly the order of operations that keeps quality risk lowest.

LARS/LAMB for larger batch sizes, and more efficient kernels

• Larger batch sizes with LARS/LAMB: a bigger batch reduces the number of optimizer steps (and thus wall-clock time and GPU-hours) for the same total data seen, but plain SGD/Adam destabilizes at very large batch sizes unless the learning rate is scaled up proportionally, which itself causes divergence beyond a point; layer-wise adaptive rate scaling optimizers (LARS for SGD-style training, LAMB for Adam-style, both commonly used for large-batch BERT/ResNet-scale training) normalize the update per layer by that layer's weight norm, which lets the effective learning rate scale further with batch size before training destabilizes, making a bigger batch (and the throughput win it buys) actually usable rather than just theoretically available. This is a moderate-risk lever: it changes optimizer dynamics, so it needs the same before/after validation-metric check as any other quality-sensitive change.
• More efficient kernels: replacing generic framework ops with fused, hardware-tuned kernels (e.g. FlashAttention for the attention block, fused LayerNorm/optimizer kernels, or a framework's torch.compile/XLA graph compilation) cuts wall-clock time per step with no change to the actual numerical computation being performed (same math, fewer memory round-trips and kernel-launch overheads), making it one of the lowest-quality-risk levers on the list, similar in risk profile to mixed precision.

Trade-offs & pitfalls

The critical discipline here is validating that "quality is preserved" empirically (tracking the actual validation metric before and after each change) rather than assuming a change is safe because it is theoretically lossless; mixed precision done carelessly (no loss scaling, no LR adjustment) or aggressive spot-instance usage without adequate checkpointing (losing significant progress on preemption) can each silently degrade either quality or effective cost savings if not implemented and monitored carefully.

Direct answer

Synchronous SGD has every worker compute a gradient against the same, current parameter values and waits for all workers before applying a single combined update, giving convergence behavior equivalent to (or very close to) single-machine SGD at a larger effective batch size; asynchronous SGD lets each worker push its gradient and pull fresh parameters independently, without waiting for others, trading some workers computing gradients against slightly outdated ("stale") parameters for higher hardware utilization.

Structured elaboration

• Synchronous: every worker's gradient this step is computed against identical parameter values (the state after the previous step's update); once all gradients arrive, they're averaged and applied as one update, after which every worker again has identical, up-to-date parameters. Convergence guarantees closely mirror standard SGD's, since the process is mathematically equivalent to computing a gradient over a larger effective batch (the concatenation of every worker's mini-batch).
• Asynchronous: a worker pulls current parameters, computes a gradient, and pushes it back independently of other workers' progress; by the time its push arrives, the server's parameters may have already been updated by other workers' pushes in the meantime, meaning the pushed gradient was computed against parameters that are now "stale" (out of date) relative to the current server state.
• Staleness and its effect: the degree of staleness (how many other updates happened between a worker's pull and its push) tends to grow with more workers and with heterogeneous worker speeds (a slow worker's gradient becomes more stale the longer it takes to compute); staleness biases the effective update direction, since it's technically a gradient of an earlier point on the loss surface being applied to a later point, which can slow or, in extreme cases, destabilize convergence if unbounded.
• When each is chosen: synchronous is the default for most modern large-scale training (predictable convergence behavior, well-supported by AllReduce-based collectives) provided stragglers are managed; asynchronous is chosen specifically when worker heterogeneity or unreliability is severe enough that waiting for the slowest worker every step would be prohibitively wasteful, accepting some convergence-quality cost in exchange for higher aggregate hardware utilization.

Worked example

With 8 workers, one of which is consistently 3x slower than the others (a straggler): synchronous training's every-step wall-clock time is bounded by that slowest worker, wasting the other 7 workers' idle time waiting each step; asynchronous training lets the 7 faster workers keep contributing updates continuously without waiting, at the cost of the slow worker's occasional contributions being noticeably stale (computed against parameters several updates out of date) by the time they arrive.

Trade-offs & pitfalls

Bounded-staleness schemes (allowing async updates but capping how stale any single contribution is allowed to be before it's rejected or down-weighted) are a common middle ground, retaining most of asynchronous training's utilization benefit while limiting the worst-case convergence-bias risk that fully unbounded asynchrony carries.

Direct answer

Deciding between buying a larger GPU cluster to finish training faster versus training longer on cheaper hardware is a cost-per-unit-of-training-progress comparison: model each option's total cost (hardware/rental cost times duration) against the value of finishing sooner (if there's a genuine time-value, e.g. faster iteration enabling more experiments, or a hard deadline), and choose whichever option minimizes total cost for the actual constraint that matters (fixed budget, fixed deadline, or a genuine trade-off between the two).

Structured elaboration

• Cost model inputs: for the larger/faster cluster option, hourly cost per GPU (higher-end or more numerous GPUs typically cost more per hour) times the (shorter) training duration; for the smaller/cheaper option, lower hourly cost per GPU times the (longer) training duration; both need to be normalized to the SAME unit (total dollar cost for the SAME amount of total training progress, e.g. the same number of tokens/epochs). A fourth input, time-to-convergence curves (loss or eval-metric versus wall-clock time for each candidate configuration, not just raw throughput), matters because scaling to more GPUs often forces a larger global batch size, which can shift how many optimizer steps are needed to reach the same loss target; comparing options purely on throughput/GPU-hours without checking their actual convergence curves risks concluding the larger cluster is cheaper or faster when, after accounting for it needing more total steps to converge at the larger batch size, it may not be.
• When faster wins on pure cost: if the larger cluster achieves more than proportionally faster completion for its proportionally higher cost (e.g. due to per-GPU efficiency gains from better interconnect at larger scale, or economy-of-scale pricing), total dollar cost can actually be LOWER for the faster option, not just faster; this isn't guaranteed and needs to be checked with real numbers, not assumed.
• When cheaper-and-slower wins on pure cost: if scaling up doesn't achieve proportional speedup (the scaling-efficiency degradation discussed elsewhere in this topic), the larger cluster's extra cost isn't fully offset by proportional time savings, making the smaller, cheaper, longer-running option cheaper in total dollars, even though it finishes later.
• Beyond pure dollar cost, the value of time: if there's a genuine business reason time matters independent of dollar cost (a hard product deadline, or the ability to run more sequential experiments and therefore find a better model faster), that needs to be explicitly incorporated as a value (even if roughly estimated) added to the "faster" option's side of the comparison, not left as an unstated, implicit tiebreaker. This is the model's opportunity cost term: capital or GPU-hours committed to this training run cannot be spent on other experiments in the same window, so the relevant comparison is not just this run's dollar cost but that cost plus the estimated value of whatever else those GPU-hours or that calendar time could have produced instead, made explicit as a number (even a rough one) rather than an unstated intuition.

Worked example

Option A (larger cluster): 256 GPUs at $2/GPU-hour, completing training in 48 hours: total cost = $256 \times 2 \times 48 = \$24{,}576$. Option B (smaller cluster): 64 GPUs at $2/GPU-hour (same per-GPU rate, different scale, assuming the smaller cluster achieves close to linear scaling since 64 GPUs is a less demanding scaling regime than 256), completing the same total training progress in roughly 4x the wall-clock time due to 4x fewer GPUs (assuming near-linear scaling holds at this smaller scale): $64 \times 2 \times 192 = \$24{,}576$, the same total dollar cost in this case, since near-linear scaling at 64 GPUs means the two options are dollar-equivalent, and the actual decision then hinges entirely on whether the 48-hour-versus-192-hour completion-time difference has business value worth paying for (it doesn't cost extra in this particular numeric example, so faster genuinely wins here with no downside once scaling holds).

Trade-offs & pitfalls

The most common mistake in this kind of analysis is comparing only wall-clock time (bigger is obviously faster) or only sticker hourly cost (smaller is obviously cheaper) without actually computing total dollar cost for equivalent training progress at both options, which is the only apples-to-apples comparison; the real answer depends entirely on how scaling efficiency behaves at the specific GPU counts being compared, which must be measured (per the scaling-efficiency-measurement discussion elsewhere in this topic), not assumed to be either perfectly linear or badly degraded.

Direct answer

Data parallelism replicates the entire model on every device and splits the training data across devices; model parallelism splits the model itself (different parameters on different devices) while typically keeping the same data on each; pipeline parallelism is a specific form of model parallelism that splits the model into sequential stages placed on different devices, with different devices processing different micro-batches at different pipeline stages simultaneously to keep all devices busy.

Structured elaboration

• Data parallelism: every device holds an identical, full copy of the model; each device processes a different slice of the mini-batch, computing its own local gradients; gradients are then synchronized (typically via all-reduce) across devices before the (identical, since it started from identical weights and applies an identical aggregated gradient) parameter update. Chosen when the model fits comfortably in a single device's memory but training would benefit from more parallel compute across more data.
• Model parallelism: the model's parameters themselves are split across devices (e.g. different layers, or different slices of very large layers via tensor parallelism), with each device holding only a portion of the full model; necessary specifically when the model is too large to fit on a single device, regardless of batch size or data volume. Its communication pattern is denser than data parallelism's: because a split weight matrix's output activations need to be reassembled (or its input activations need to be replicated) across the devices holding the split, tensor parallelism requires an all-gather or reduce-scatter/all-reduce at essentially every split-and-rejoin point in both the forward and backward pass, on every layer that's split, which is why tensor parallelism is generally kept within a single node over fast NVLink rather than spread across a slower network.
• Pipeline parallelism: a specific model-parallelism strategy that splits the model into sequential stages (contiguous groups of layers), each stage on a different device; to avoid devices sitting idle waiting for the previous stage to finish an entire batch, the batch is further divided into smaller micro-batches, which flow through the pipeline stages in an overlapped fashion (while stage 2 processes micro-batch 1's forward pass, stage 1 can already be processing micro-batch 2's forward pass), analogous to an assembly line. Its communication pattern is much sparser than tensor parallelism's: since each device holds a distinct, non-overlapping group of layers, communication between adjacent stages is limited to point-to-point send/recv of activations (forward direction) and gradients (backward direction) at each stage boundary, once per micro-batch, rather than the dense collective communication tensor parallelism requires on every split layer; this is why pipeline parallelism tolerates being spread across nodes with slower cross-node networking far better than tensor parallelism does.
• Combining all three (3D parallelism): for very large models on large clusters, these three axes are commonly combined: tensor/model parallelism within a node (exploiting fast NVLink), pipeline parallelism across a few node groups (tolerating the necessarily-less-frequent cross-node communication), and data parallelism to scale out replicas of that combined unit across many such groups, as discussed in the earlier large-model-training-design survivors.

Worked example

Training a model that fits comfortably on one GPU but would benefit from more compute: use pure data parallelism, replicating the model across, say, 8 GPUs, each processing 1/8 of the mini-batch. Training a model too large for even one GPU's memory: use model parallelism (splitting the model's parameters across devices) as a prerequisite before data parallelism can even be applied at all, since data parallelism assumes the whole model fits on each device to begin with. Training a very large model across many GPUs with a deep layer stack: pipeline parallelism specifically addresses the idle-time problem that would otherwise occur if you naively split layers across devices without micro-batching, since without micro-batches, each device would sit completely idle while waiting for the previous device to finish the ENTIRE batch's forward pass before it could start.

Trade-offs & pitfalls

A common conceptual confusion is treating "model parallelism" and "pipeline parallelism" as synonyms; pipeline parallelism is specifically ONE WAY to do model parallelism (splitting by sequential layer groups, with micro-batching to keep devices busy), while tensor parallelism (splitting individual weight matrices) is a different way to do model parallelism, and the choice between them (or combining both) depends on the specific memory/communication trade-offs of the model and cluster in question.