Scalability Patterns and Techniques Questions

Direct answer

Attack a CPU-bound feature-computation bottleneck in priority order of how much online cost each technique removes: first eliminate work entirely by precomputing what can be computed ahead of the request, then approximate what cannot be precomputed to shrink its cost, then make what remains on the hot path run faster per unit of work, and only move to a specialized service once the computation itself no longer fits comfortably in the request path at all.

Structured elaboration

1. Precompute what does not depend on the live request.
Any feature that is a function of data known before the request arrives (a user's rolling 7-day activity count, an item's historical embedding) can be computed offline on a schedule and looked up online instead of recomputed. This removes the CPU cost from the request path entirely rather than reducing it, which is why it is the first thing to check. The cost is staleness: the feature reflects the world as of the last precompute run, not the instant of the request.

2. Approximate or quantize what must stay on the request path.
For features that genuinely depend on request-time data, ask whether full precision is needed. Quantized representations (lower-precision numeric types) and approximate algorithms (sketches, sampled aggregates instead of exact ones) trade a bounded, quantifiable accuracy loss for a real reduction in CPU and memory cost per request. This only works if the accuracy loss is measured against the actual downstream metric that matters (model accuracy, ranking quality), not assumed to be negligible.

3. Make the remaining hot-path code fast per unit of work.
Once you've minimized how much computation happens online, optimize what's left: replace interpreted per-element loops with vectorized array operations, or use just-in-time (JIT) compilation to compile hot numeric code paths to native instructions ahead of first use. This is pure engineering efficiency with no accuracy trade-off, which is why it belongs after precompute and approximation, not before them: it is wasted effort to hand-optimize code you could have avoided running at all.

4. Move heavy transforms to a specialized service only when the above are not enough.
If the feature computation is inherently expensive even after the previous steps (a transform that genuinely needs specialized hardware or a different runtime), extracting it into its own service lets it scale, batch, and be optimized independently of the request-handling tier. The cost is a network hop and an added operational dependency, so this is a structural change reserved for computations that cannot be shrunk further, not a first response to a slow function.

Worked example

This uses Amdahl's-law-style reasoning: if feature computation is a fraction $f$ of total per-request CPU cost and a technique reduces that fraction's cost by a factor $k$ (leaving it unchanged if $k=1$, removing it entirely if $k \to \infty$), the overall speedup on a fixed-size fleet is:

$$\text{speedup} = \frac{1}{(1-f) + f/k}$$

Assume, as an illustrative planning input rather than a measured benchmark: profiling shows feature computation is 75% of per-request CPU cost ($f = 0.75$).

$$\begin{aligned} \text{precompute (removes it, } k \to \infty \text{)}: &\quad \text{speedup} = \frac{1}{0.25} = 4\times \\ \text{vectorize online (3x faster on that portion, } k = 3 \text{)}: &\quad \text{speedup} = \frac{1}{0.25 + 0.25} = 2\times \end{aligned}$$

The formula makes the priority order concrete: precomputing (removing the term entirely) beats vectorizing the same work in place, because $k \to \infty$ always dominates any finite $k$. That is the quantitative reason "eliminate before optimize" is the right order, not just a rule of thumb.

Trade-offs & pitfalls

Precomputing trades accuracy-at-the-instant for speed (staleness), which is invisible until a downstream consumer needs a feature that just changed and gets the old value; it needs an explicit staleness bound the consumer can reason about. Approximation and quantization trade measured accuracy loss for speed, and the mistake is skipping the measurement step and assuming the loss is negligible. Vectorization and JIT compilation are the safest technique (no accuracy cost) but the smallest lever if the fraction of cost they address ($f$) is small, which is exactly what the speedup formula shows. Extracting a specialized service is the most expensive option operationally (new deployment, new failure mode, network latency added to the path) and should be justified by the same fraction-of-cost reasoning, not reached for first because it feels like the "real" fix.

Direct answer

A machine-learning serving stack typically caches three distinct things, and they need different keys, hit-rate expectations, and staleness tolerances: prediction results (keyed by model version plus input), feature values (keyed by entity plus a time window), and model artifacts (keyed by model version, cached mainly to avoid reloading weights on every request). Getting the cache key wrong on any of the three doesn't just hurt latency, it can silently serve an outdated or mismatched model's output, which shows up as a model-quality regression, not a normal cache bug.

Structured elaboration

Prediction result caching

• Cache key: a deterministic hash of (model_id, model_version, normalized_input), where the input is normalized (sorted feature keys, discretized continuous buckets if appropriate) so semantically identical requests produce the same key. Include a tenant or authorization identifier if predictions differ by caller.
• What drives hit rate: high for repeated, low-cardinality inputs (batch scoring over a fixed catalog, popular queries); low for high-cardinality, per-user, real-time inputs where every request is effectively unique.
• Freshness: tying the key to model_version prevents an old model's cached output from being served after a rollout, which matters more here than a time-to-live (TTL, how long a cached value stays valid) alone would.
• Staleness impact: a stale prediction isn't just slow data, it's a wrong model output; for time-sensitive decisions (fraud scoring, dynamic pricing) this can directly move a business metric, not just a latency metric.

Feature value caching

• Cache key: (entity_id, feature_set_version, window_end_timestamp) for temporal features (for example, a 7-day rolling average), with a TTL aligned to how often the underlying feature actually updates.
• What drives hit rate: high when many requests reference the same entities (popular products, active users); low when entity cardinality is enormous with no locality. Several concrete techniques raise hit rate and cut tail latency here, roughly in the order most teams reach for them: tiered caching and parallel/batched fetching are the standard starting point most feature stores need first; denormalized feature bundles, precomputed aggregates, and async read-ahead fetching are more specialized techniques layered on only once that basic pair isn't enough for a specific symptom (a slow join, an aggregation that's expensive to compute live, or a predictable access pattern worth prefetching):
  • Tiered caching: a small, fast local (in-process) cache in front of a larger shared cache, so the hottest entities never leave the process serving them.
  • Denormalized feature bundles: pre-joining several feature tables into one lookup-ready record per entity, so a single cache read returns everything the model needs instead of N separate lookups.
  • Precomputed aggregates: computing rolling statistics offline or via a streaming job ahead of request time, rather than aggregating on the request path.
  • Parallel or batched fetching: retrieving features for multiple entities in one round trip instead of serially, and feature-reduction (only fetching the minimal feature set an endpoint actually uses) to cut 95th-percentile (P95) and 99th-percentile (P99) tail latency.
  • Async, read-ahead fetch patterns for a read-heavy feature store: prefetching a feature vector before the request that needs it arrives, when access can be anticipated.
• A distinct case worth separating out: features that are expensive to produce, for example because they call an external enrichment service or require a CPU-bound transform. These justify caching the derived value itself, pre-warming it ahead of expected traffic for known-important entities, and using partial invalidation, invalidating only the specific derived feature that changed rather than the whole feature bundle, to bound how much model-accuracy staleness a stale derived feature can introduce.
• Freshness and staleness impact: stale features bias the prediction itself (outdated user behavior driving a stale recommendation), which degrades model accuracy and can move revenue, not just latency.

Model artifact caching

• Cache key: (model_id, model_version, artifact_checksum), optionally including a runtime variant like quantized versus full precision.
• What it actually protects: loading a model's weights from disk or blob storage on every single request is a concrete anti-pattern, it adds fixed input/output and deserialization cost to the hot path of every call. Caching the loaded, deserialized model in process memory removes that cost from steady-state serving; its main benefit is avoiding repeated cold-starts, not improving accuracy.
• Freshness: must be evicted and atomically swapped when a new model build deploys, so a request never mixes weights from two different versions mid-inference.
• Staleness impact: serving an old model's weights after a rollout was supposed to complete can reintroduce a known bug or a worse-performing model, and in regulated settings can create an audit problem, since the served model no longer matches the one that was approved.

A fourth thing worth caching: intermediate results in a multi-stage pipeline

Many serving stacks aren't a single model call but an ensemble or a multi-stage pipeline (an embedding stage feeding a re-ranking stage feeding a blending stage). Caching each stage's intermediate output, keyed by that stage's own inputs and model version, avoids redoing expensive upstream work when only a downstream stage needs to rerun. It also gives a natural fallback: if a downstream stage is unavailable or times out, the pipeline can serve using the last cached output from the stages that did complete (or a simpler standalone prediction) rather than failing the whole request outright.

Worked example

Hot keys in a feature cache aren't primarily about a segment's total traffic share, they're about how concentrated the requests to a small number of specific keys become. Consider a distributed feature-caching hierarchy (a local, in-process tier backed by a shared regional tier) built to serve a product at 100-million-user scale. Assume, as a planning input rather than a measured fact, that total feature-cache read traffic across the shared regional tier is 50,000 queries per second (QPS, queries per second), spread evenly by consistent-hashing-style sharding across 20 shards. Each shard's fair share of traffic is:

$\frac{50{,}000}{20} = 2{,}500 \text{ QPS per shard}$

Now suppose a high-value user segment receives materially heavier personalization than typical users, more feature lookups and more pipeline stages per request, and that this segment's keys happen to land on just 2 of the 20 shards. Even if the segment is a small fraction of the total user base, those 2 shards can see request rates well above the 2,500 QPS fair share while the fleet-wide average still looks healthy, because the imbalance is hidden by averaging across all 20 shards. Two levers address this: technically, give that segment's keys a dedicated local cache tier or replicate them across more shards so no single shard owns them exclusively; on the product side, accept a slightly lighter-weight or briefly staler personalized experience for that segment specifically when its shard is under pressure, rather than let it degrade latency for the rest of the service.

Trade-offs & pitfalls

• Version everything (predictions, features, and artifacts alike) against model_version; the single most common failure mode in machine-learning-serving caches is a rollout that updates the model but leaves a cache serving outputs computed under the old one.
• Don't apply the same freshness intuition to all three cache types: a stale artifact is mostly a latency and correctness-of-version problem, while a stale feature or prediction is a silent model-quality problem that won't throw an error, it just quietly gets the answer wrong.
• Monitor downstream business or model-quality metrics alongside cache hit rate, not just hit rate on its own; a healthy hit rate can coexist with a staleness window that's actually hurting accuracy if nobody is watching for that specific signal.
• Pre-warming and partial invalidation both add operational complexity; they're worth it for expensive, externally-enriched features specifically, not as a default applied to every feature in the store.

Direct answer

Default to stateless serving with session state externalized to a shared store (Redis or a similar key-value store): it preserves horizontal scalability and fault tolerance without extra work. Prefer in-process stateful serving only when all three of these hold at once: the latency budget genuinely can't absorb an external round trip, the session state is small enough to comfortably live in memory per replica, and the routing layer can guarantee a user's repeated requests reach the same replica (sticky routing). An online recommendation model using short-term interaction history often sits right at this boundary, which is why the right answer is usually a hybrid rather than a pure choice.

Structured elaboration

Stateless serving. Inference is a pure function of model plus input, with any needed context (recent interactions, session state) fetched from an external store on each request. Any replica can serve any request, which is what makes this pattern trivially horizontally scalable and fault tolerant.

Stateful serving. The serving process keeps session state (recent interaction history, a user's short-term embedding) in memory across requests, avoiding a store round trip on every call. This lowers per-request latency and load on the external store, but only works if requests from the same user keep landing on the same replica.

Worked example

Consider a recommendation service reading a user's last-20-interactions window to personalize the next set of results. That window is small (comfortably fits in memory per active user) and short-lived (it doesn't need to survive indefinitely), which are exactly the conditions favoring a stateful approach: route each user's requests to the same replica (this is a routing-layer requirement, not something the serving process controls on its own), keep the last-20-interactions window in an in-memory structure per active session, and periodically checkpoint it asynchronously to an external store so a replica crash doesn't lose the window outright, only whatever changed since the last checkpoint.

Contrast that with a version of the same product that personalizes off a user's full browsing history rather than a short recent window: that state is no longer small or bounded, it grows over the life of the account, and it stops meeting the "fits comfortably in memory per replica" condition that made stateful serving attractive in the first place. At that point, externalizing to a store built for large, sharded key-value access (rather than trying to keep it resident in the serving process) is the right call, even though it reintroduces the external round trip the stateful design was trying to avoid.

The practical middle ground most real recommendation services land on: keep the tiny, latency-sensitive recent-interaction window in-process for fast reads, but treat it as a cache in front of an authoritative external store, not as the only copy, so a crash degrades gracefully (falls back to the external store) instead of losing the session outright.

Trade-offs & pitfalls

• Stateful serving requires the load balancer to route a given user's repeated requests to the same replica; that routing requirement is itself a scaling constraint being traded for lower latency, and it needs to be designed deliberately rather than assumed.
• "Short-term" session state can grow unbounded if nothing enforces a size or time limit; a design that was safely small at launch can silently become a memory-pressure problem as usage patterns shift.
• Treating in-memory state as durable is the most common failure mode: without an async checkpoint to an external store, a crash doesn't just lose one request, it loses that user's entire session context.
• Externalizing everything by default avoids all of the above at the cost of a network round trip on every request; that's the right trade for most services, and only worth giving up when the latency budget and state size genuinely demand it.

Direct answer

Forecast GPU capacity by projecting current training and inference demand forward with an explicit growth assumption, translate that into a GPU-hour budget, then split that budget across tenants with quotas and priority tiers rather than first-come-first-served access, and absorb burstiness with a shared preemptible pool instead of over-provisioning every tenant's peak individually.

Structured elaboration

1. Separate the two workload shapes before forecasting.
Nightly training and real-time inference have opposite scheduling profiles: training is throughput-oriented, tolerant of queueing, and can be preempted and resumed from a checkpoint; inference is latency-oriented, cannot queue behind a long training job, and needs guaranteed capacity during business hours. Forecast and quota them separately, then decide how much of the training capacity can double as burst inference capacity (and vice versa) when idle.

2. Forecast from a baseline and a stated growth driver, not a flat trend line.
A 12-month GPU forecast should be driven by concrete inputs you can name: expected model count growth, expected training frequency per model, and expected inference traffic growth (tied to product roadmap, not just historical curve-fitting, since a 12-month horizon on GPU demand is usually driven by planned launches more than organic drift).

3. Define quotas and priorities across tenants.

• Quotas: a guaranteed floor per tenant (hard reservation) sized to their steady-state need, so no tenant is starved by another's burst.
• Priority tiers above the floor: latency-critical inference outranks best-effort training for shared/burst capacity; within training, deadline-bound jobs outrank exploratory ones.
• Fair-share reconciliation: unused quota from an idle tenant should be lend-able to others temporarily, reclaimed when the owning tenant needs it back, which is what makes shared clusters more cost-efficient than static per-tenant carve-outs.

4. Design scheduling and preemption policy around that priority order.
Give inference workloads dedicated, non-preemptible capacity for their guaranteed floor. Run training as preemptible: a training job can be checkpointed and evicted to make room for a burst in inference demand or a higher-priority training job, then resumed. This is the mechanism that lets one cluster serve both workload types without doubling the fleet.

5. Handle bursty demand with a shared elastic pool, not per-tenant peak provisioning.
Provisioning every tenant for their own peak wastes capacity, since peaks rarely align across tenants. A shared burst pool, allocated by priority when contended, covers more aggregate peak with fewer total GPUs, at the cost of occasional queueing for lower-priority work during simultaneous bursts, which the priority tiers make an explicit, accepted trade-off rather than a surprise.

6. Metrics and cost trade-offs.
Track GPU-hour utilization per tenant (catches over-provisioned quotas), queue wait time by priority tier (catches under-provisioned burst capacity), and preemption/checkpoint-restart rate for training (catches a preemption policy that is too aggressive and is wasting compute on repeated restarts). The core cost trade-off is reservation versus utilization: guaranteed floors improve latency predictability for inference but sit idle during off-peak hours; a larger shared/preemptible pool improves utilization but adds queueing risk that inference cannot tolerate, which is exactly why the two workloads get different scheduling treatment rather than one policy for the whole cluster.

Worked example

Assume, as planning inputs rather than measured facts: current combined steady-state demand across training and inference is 350 GPU-hours/day, and demand is expected to grow 8% month-over-month (compounding, reflecting planned model and traffic growth) over the next 12 months.

$$\begin{aligned} \text{forecast demand}_{12} &= 350 \times (1.08)^{12} \approx 881\ \text{GPU-hours/day} \\ \text{required fleet size} &= \frac{881}{24 \times 0.70} \approx 52.5 \;\Rightarrow\; 53\ \text{GPUs} \end{aligned}$$

(24 hours of possible daily capacity per GPU, discounted to a 70% target utilization, flagged as a planning assumption that leaves headroom for burst and preemption overhead rather than running the fleet at the edge of saturation.)

If the cluster serves three tenants weighted 50/30/20 by their steady-state share, that 53-GPU fleet implies guaranteed floors of roughly 26-27 / 16 / 10-11 GPUs, with any unused share reclaimable by the burst pool rather than sitting idle.

Trade-offs & pitfalls

The main failure mode is applying one scheduling policy to both workloads: inference queued behind a preemptible training job causes latency SLO (service-level objective) violations, while training that is never preemptible forces the fleet to be sized for the sum of every tenant's peak instead of a shared pool, which is far more expensive. A second common mistake is forecasting GPU demand the way CPU demand is forecast (a smooth trend line) when GPU demand in most orgs is lumpy and roadmap-driven; the growth assumption should be tied to named upcoming work, and revisited quarterly rather than trusted for the full 12 months unchanged.

Direct answer

For a 50ms P95 (95th-percentile) latency budget, prefer horizontal scaling with right-sized graphics processing unit (GPU) replicas as the default, and reserve vertical scaling (bigger GPUs) and model sharding (splitting one model across multiple GPUs) for cases that genuinely need them: sharding only when the model doesn't fit on one GPU's memory, and bigger single instances only when batching economics clearly favor concentration over replication for your actual traffic shape. Neither is universally cheaper; the deciding factors are cold-start behavior, batching efficiency, and how tightly your latency budget constrains batch size.

Structured elaboration

Vertical (bigger GPU instances)

• Fewer, larger units mean model loading happens less often across the fleet, and a bigger memory/compute pool can build larger batches within the same latency budget, which amortizes per-request fixed overhead (kernel launch, memory transfer) more efficiently.
• Cost scales non-linearly with GPU size; the largest instance tiers carry a real price premium per unit of compute, and a single large instance is also a larger single point of failure.

Horizontal (more replicas, or model sharding for oversized models)

• Elastic: capacity tracks demand by adding or removing replicas, and losing one replica out of many degrades capacity gracefully rather than taking the service down.
• Each replica typically loads its own full copy of the model (unless sharded), so scaling out means more frequent cold starts across the fleet during scale events, and inter-replica batching coordination is looser than within one large instance.
• Model sharding (splitting a model too large for one GPU across several, communicating over an interconnect) trades added inter-GPU communication latency for the ability to serve a model that otherwise wouldn't fit at all; it's a "how do we serve this" answer, not a "how do we get cheaper" answer, and it adds real operational complexity: shard placement, inter-shard routing, and keeping every shard on a consistent model version.

Cost per prediction, worked concretely below, does not have a universal winner; it depends on the batching efficiency curve of the actual model and hardware.

Cold start and GPU utilization. GPU instances take real, non-trivial time to become ready (instance boot plus model load into GPU memory), which is qualitatively slower than a small stateless web replica's readiness. This directly constrains autoscaling: a purely reactive horizontal policy that scales out only after load has already arrived will miss the 50ms tail-latency target during the ramp, since new replicas aren't instantly useful the moment they boot. That pushes toward a pre-warmed pool (the same pattern used for any latency-critical, slow-to-initialize service) rather than cold reactive scale-out.

Operational complexity. Vertical scaling with a handful of large, fully-loaded instances is operationally the simplest: fewer things to monitor, deploy, and keep in sync. Horizontal scaling with plain replication adds fleet-management overhead but nothing conceptually new; model sharding adds the most complexity of the three, since it introduces distributed-inference concerns (routing a request to the right shard set, keeping shards version-consistent) that a single-GPU deployment never has to solve.

Worked example

Cost-per-prediction comparison. These are illustrative, stated dollar figures for the purpose of the calculation, not live cloud pricing. Assume a large GPU instance costs $4.00/hour and, with batching, sustains 200 predictions/second:

$$\frac{\$4.00/\text{hr}}{200 \ \text{preds/s} \times 3600 \ \text{s/hr}} \approx \$5.56 \text{ per million predictions}$$

A smaller GPU instance costs $1.20/hour and, with less memory headroom for batching, sustains 50 predictions/second:

$$\frac{\$1.20/\text{hr}}{50 \ \text{preds/s} \times 3600 \ \text{s/hr}} \approx \$6.67 \text{ per million predictions}$$

Matching the big instance's 200 preds/sec aggregate throughput takes 4 small instances:

$$4 \times \$1.20/\text{hr} = \$4.80/\text{hr} \quad (\text{vs. } \$4.00/\text{hr for the one big instance})$$

In this illustrative model, the horizontally-scaled fleet costs about 20% more per prediction than the single large instance at equal aggregate throughput, purely because the larger instance's bigger batches amortize fixed per-request overhead more efficiently. This direction is not a general rule; it reverses as soon as the workload is spiky rather than steady (the four small instances can independently scale down when idle, while the one big instance is an all-or-nothing cost), or as soon as the model doesn't fit on the smaller GPU tier at all, at which point sharding or moving to the larger instance stops being optional.

Recommended architecture

A hybrid: right-sized (not maximal) GPU replicas running behind a small pre-warmed pool sized to the expected spike arrival rate, each replica running dynamic batching tuned so the batch-wait window plus inference time stays inside the 50ms P95 budget. Reserve model sharding strictly for models that don't fit on a single GPU's memory, since it's the highest-operational-complexity option and buys nothing for a model that already fits on one card. This gets most of vertical scaling's batching efficiency (via per-replica batching) while keeping horizontal's elasticity and fault isolation, and it avoids paying sharding's coordination cost for a problem sharding wasn't needed to solve.

Trade-offs & pitfalls

• Chasing cost efficiency with ever-larger batches can quietly blow the latency budget, since a bigger batch takes longer both to fill and to compute; batch size has to be tuned against the P95 target, not against throughput alone.
• Treating horizontal scaling as free, instant elasticity while ignoring GPU cold-start time is the most common mistake here; a freshly-booted GPU replica is not immediately equivalent to a warm one the way a small stateless web replica usually is.
• Reaching for model sharding as a scaling strategy (rather than a "the model doesn't fit" necessity) adds the most operational complexity of any option here for the least benefit when the model actually fits on one GPU.
• The cost-per-prediction comparison above assumes steady, batchable traffic; a spiky, unpredictable traffic pattern changes the calculus toward horizontal's elasticity regardless of the flat-throughput cost numbers.