Scalability Patterns and Techniques Questions

Direct answer

For a small team starting from a proof of concept serving roughly 1,000 requests/sec at 50ms P95 latency (95th-percentile latency), I'd start with the monolithic multi-model service and split into per-model microservices only once a specific model's traffic, resource profile, or release cadence diverges enough from the others to justify the added operational cost; premature per-model decomposition trades a real, immediate development-velocity cost for a scaling flexibility benefit the team may not need yet.

Structured elaboration

Development velocity. A monolithic service means one deployable, one test suite, one place to add a new model or fix a shared preprocessing bug. A small team without dedicated platform support pays a real tax for coordinating N separate services (N sets of CI/CD, N sets of dependency upgrades, N places a shared feature-extraction bug must be fixed identically). Early on, this usually dominates every other factor.

Deployment complexity and testability. One service means one deployment pipeline and one integration-test surface; per-model services mean each model change deploys independently (an advantage once models genuinely evolve on different schedules) but multiplies the number of moving parts that must each be tested, versioned, and rolled back independently.

Coupling. In a monolith, a bug or resource spike in one model's code can degrade another model's latency (shared process, shared memory, shared thread pool). Per-model services isolate that blast radius, at the cost of adding a network hop and a separate failure mode (the model service being unreachable) that did not exist before.

Observability. A monolith's per-model performance is harder to see cleanly, since metrics are aggregated across all models unless the code explicitly tags them. Per-model services get natural per-model dashboards and alerts for free, from the service boundary itself.

Resource utilization. This is the factor most tied to the given numbers, and the one that most often justifies decomposition. If the 1,000 requests/sec is split evenly across similarly-sized models, a monolith's shared resource pool is efficient: idle capacity for one model's low-traffic period is naturally used by another's high-traffic period. If instead demand and resource needs are skewed across models (one model dominates traffic, or one model needs a GPU while others run fine on CPU), decomposition lets you size and scale each model's replica count independently to match its own demand, instead of scaling the whole monolith (and paying for every model's replica) just to keep the hot model within its 50ms P95 latency target.

Framing as a per-model service boundary decision. The same choice can be framed as: does each model get its own per-model service boundary, and what does that do to scaling per model and to team ownership? Drawing that boundary means each model's replica count can scale per model, independently matching its own traffic instead of the aggregate, and typically means a single team (or a single engineer) becomes the clear owner of one model's service, establishing team ownership instead of shared, diffuse responsibility for one large multi-model deployable. That team-ownership clarity is a real benefit as team and model count grow, but it is not free early on, when a small team's context-switching cost across N services can exceed the coordination cost of one shared codebase.

Worked example

Assume, as an illustrative planning input, that of the 1,000 requests/sec total, one model accounts for 700 requests/sec (a hot, latency-critical recommendation model) and the remaining three models share 300 requests/sec combined.

In the monolith, meeting the 50ms P95 target for the hot model forces you to scale the whole service (all four models' code paths) to the hot model's replica count, so the three cold models are running far more replicas than their own traffic needs, purely as a side effect of sharing a deployable with the hot one. Splitting per model lets the hot model scale to meet its own 700 requests/sec target independently, while the three cold models run a much smaller, cheaper replica count sized to their combined 300 requests/sec, which is the resource-utilization case for decomposition once the traffic split across models is uneven enough to matter.

Trade-offs & pitfalls

The main pitfall is decomposing before there is evidence of the skew that justifies it: a small team that splits into per-model services on day one pays the coordination and operational cost immediately while the utilization benefit is theoretical, since a fresh proof of concept rarely has the traffic history to know which model will actually become the hot one. The better sequence is start monolithic, instrument per-model resource usage and latency from day one even inside the monolith, and split out a specific model only when its profile (traffic share, latency sensitivity, or hardware need such as a GPU) has diverged enough from its siblings to make shared deployment genuinely costly, not just theoretically suboptimal.

Direct answer

For a 50ms P95 (95th-percentile) latency budget, prefer horizontal scaling with right-sized graphics processing unit (GPU) replicas as the default, and reserve vertical scaling (bigger GPUs) and model sharding (splitting one model across multiple GPUs) for cases that genuinely need them: sharding only when the model doesn't fit on one GPU's memory, and bigger single instances only when batching economics clearly favor concentration over replication for your actual traffic shape. Neither is universally cheaper; the deciding factors are cold-start behavior, batching efficiency, and how tightly your latency budget constrains batch size.

Structured elaboration

Vertical (bigger GPU instances)

• Fewer, larger units mean model loading happens less often across the fleet, and a bigger memory/compute pool can build larger batches within the same latency budget, which amortizes per-request fixed overhead (kernel launch, memory transfer) more efficiently.
• Cost scales non-linearly with GPU size; the largest instance tiers carry a real price premium per unit of compute, and a single large instance is also a larger single point of failure.

Horizontal (more replicas, or model sharding for oversized models)

• Elastic: capacity tracks demand by adding or removing replicas, and losing one replica out of many degrades capacity gracefully rather than taking the service down.
• Each replica typically loads its own full copy of the model (unless sharded), so scaling out means more frequent cold starts across the fleet during scale events, and inter-replica batching coordination is looser than within one large instance.
• Model sharding (splitting a model too large for one GPU across several, communicating over an interconnect) trades added inter-GPU communication latency for the ability to serve a model that otherwise wouldn't fit at all; it's a "how do we serve this" answer, not a "how do we get cheaper" answer, and it adds real operational complexity: shard placement, inter-shard routing, and keeping every shard on a consistent model version.

Cost per prediction, worked concretely below, does not have a universal winner; it depends on the batching efficiency curve of the actual model and hardware.

Cold start and GPU utilization. GPU instances take real, non-trivial time to become ready (instance boot plus model load into GPU memory), which is qualitatively slower than a small stateless web replica's readiness. This directly constrains autoscaling: a purely reactive horizontal policy that scales out only after load has already arrived will miss the 50ms tail-latency target during the ramp, since new replicas aren't instantly useful the moment they boot. That pushes toward a pre-warmed pool (the same pattern used for any latency-critical, slow-to-initialize service) rather than cold reactive scale-out.

Operational complexity. Vertical scaling with a handful of large, fully-loaded instances is operationally the simplest: fewer things to monitor, deploy, and keep in sync. Horizontal scaling with plain replication adds fleet-management overhead but nothing conceptually new; model sharding adds the most complexity of the three, since it introduces distributed-inference concerns (routing a request to the right shard set, keeping shards version-consistent) that a single-GPU deployment never has to solve.

Worked example

Cost-per-prediction comparison. These are illustrative, stated dollar figures for the purpose of the calculation, not live cloud pricing. Assume a large GPU instance costs $4.00/hour and, with batching, sustains 200 predictions/second:

A smaller GPU instance costs $1.20/hour and, with less memory headroom for batching, sustains 50 predictions/second:

Matching the big instance's 200 preds/sec aggregate throughput takes 4 small instances:

In this illustrative model, the horizontally-scaled fleet costs about 20% more per prediction than the single large instance at equal aggregate throughput, purely because the larger instance's bigger batches amortize fixed per-request overhead more efficiently. This direction is not a general rule; it reverses as soon as the workload is spiky rather than steady (the four small instances can independently scale down when idle, while the one big instance is an all-or-nothing cost), or as soon as the model doesn't fit on the smaller GPU tier at all, at which point sharding or moving to the larger instance stops being optional.

Recommended architecture

A hybrid: right-sized (not maximal) GPU replicas running behind a small pre-warmed pool sized to the expected spike arrival rate, each replica running dynamic batching tuned so the batch-wait window plus inference time stays inside the 50ms P95 budget. Reserve model sharding strictly for models that don't fit on a single GPU's memory, since it's the highest-operational-complexity option and buys nothing for a model that already fits on one card. This gets most of vertical scaling's batching efficiency (via per-replica batching) while keeping horizontal's elasticity and fault isolation, and it avoids paying sharding's coordination cost for a problem sharding wasn't needed to solve.

Trade-offs & pitfalls

• Chasing cost efficiency with ever-larger batches can quietly blow the latency budget, since a bigger batch takes longer both to fill and to compute; batch size has to be tuned against the P95 target, not against throughput alone.
• Treating horizontal scaling as free, instant elasticity while ignoring GPU cold-start time is the most common mistake here; a freshly-booted GPU replica is not immediately equivalent to a warm one the way a small stateless web replica usually is.
• Reaching for model sharding as a scaling strategy (rather than a "the model doesn't fit" necessity) adds the most operational complexity of any option here for the least benefit when the model actually fits on one GPU.
• The cost-per-prediction comparison above assumes steady, batchable traffic; a spiky, unpredictable traffic pattern changes the calculus toward horizontal's elasticity regardless of the flat-throughput cost numbers.

Direct answer

A machine-learning serving stack typically caches three distinct things, and they need different keys, hit-rate expectations, and staleness tolerances: prediction results (keyed by model version plus input), feature values (keyed by entity plus a time window), and model artifacts (keyed by model version, cached mainly to avoid reloading weights on every request). Getting the cache key wrong on any of the three doesn't just hurt latency, it can silently serve an outdated or mismatched model's output, which shows up as a model-quality regression, not a normal cache bug.

Structured elaboration

Prediction result caching

• Cache key: a deterministic hash of (model_id, model_version, normalized_input), where the input is normalized (sorted feature keys, discretized continuous buckets if appropriate) so semantically identical requests produce the same key. Include a tenant or authorization identifier if predictions differ by caller.
• What drives hit rate: high for repeated, low-cardinality inputs (batch scoring over a fixed catalog, popular queries); low for high-cardinality, per-user, real-time inputs where every request is effectively unique.
• Freshness: tying the key to model_version prevents an old model's cached output from being served after a rollout, which matters more here than a time-to-live (TTL, how long a cached value stays valid) alone would.
• Staleness impact: a stale prediction isn't just slow data, it's a wrong model output; for time-sensitive decisions (fraud scoring, dynamic pricing) this can directly move a business metric, not just a latency metric.

Feature value caching

• Cache key: (entity_id, feature_set_version, window_end_timestamp) for temporal features (for example, a 7-day rolling average), with a TTL aligned to how often the underlying feature actually updates.
• What drives hit rate: high when many requests reference the same entities (popular products, active users); low when entity cardinality is enormous with no locality. Several concrete techniques raise hit rate and cut tail latency here, roughly in the order most teams reach for them: tiered caching and parallel/batched fetching are the standard starting point most feature stores need first; denormalized feature bundles, precomputed aggregates, and async read-ahead fetching are more specialized techniques layered on only once that basic pair isn't enough for a specific symptom (a slow join, an aggregation that's expensive to compute live, or a predictable access pattern worth prefetching):
  • Tiered caching: a small, fast local (in-process) cache in front of a larger shared cache, so the hottest entities never leave the process serving them.
  • Denormalized feature bundles: pre-joining several feature tables into one lookup-ready record per entity, so a single cache read returns everything the model needs instead of N separate lookups.
  • Precomputed aggregates: computing rolling statistics offline or via a streaming job ahead of request time, rather than aggregating on the request path.
  • Parallel or batched fetching: retrieving features for multiple entities in one round trip instead of serially, and feature-reduction (only fetching the minimal feature set an endpoint actually uses) to cut 95th-percentile (P95) and 99th-percentile (P99) tail latency.
  • Async, read-ahead fetch patterns for a read-heavy feature store: prefetching a feature vector before the request that needs it arrives, when access can be anticipated.
• A distinct case worth separating out: features that are expensive to produce, for example because they call an external enrichment service or require a CPU-bound transform. These justify caching the derived value itself, pre-warming it ahead of expected traffic for known-important entities, and using partial invalidation, invalidating only the specific derived feature that changed rather than the whole feature bundle, to bound how much model-accuracy staleness a stale derived feature can introduce.
• Freshness and staleness impact: stale features bias the prediction itself (outdated user behavior driving a stale recommendation), which degrades model accuracy and can move revenue, not just latency.

Model artifact caching

• Cache key: (model_id, model_version, artifact_checksum), optionally including a runtime variant like quantized versus full precision.
• What it actually protects: loading a model's weights from disk or blob storage on every single request is a concrete anti-pattern, it adds fixed input/output and deserialization cost to the hot path of every call. Caching the loaded, deserialized model in process memory removes that cost from steady-state serving; its main benefit is avoiding repeated cold-starts, not improving accuracy.
• Freshness: must be evicted and atomically swapped when a new model build deploys, so a request never mixes weights from two different versions mid-inference.
• Staleness impact: serving an old model's weights after a rollout was supposed to complete can reintroduce a known bug or a worse-performing model, and in regulated settings can create an audit problem, since the served model no longer matches the one that was approved.

A fourth thing worth caching: intermediate results in a multi-stage pipeline

Many serving stacks aren't a single model call but an ensemble or a multi-stage pipeline (an embedding stage feeding a re-ranking stage feeding a blending stage). Caching each stage's intermediate output, keyed by that stage's own inputs and model version, avoids redoing expensive upstream work when only a downstream stage needs to rerun. It also gives a natural fallback: if a downstream stage is unavailable or times out, the pipeline can serve using the last cached output from the stages that did complete (or a simpler standalone prediction) rather than failing the whole request outright.

Worked example

Hot keys in a feature cache aren't primarily about a segment's total traffic share, they're about how concentrated the requests to a small number of specific keys become. Consider a distributed feature-caching hierarchy (a local, in-process tier backed by a shared regional tier) built to serve a product at 100-million-user scale. Assume, as a planning input rather than a measured fact, that total feature-cache read traffic across the shared regional tier is 50,000 queries per second (QPS, queries per second), spread evenly by consistent-hashing-style sharding across 20 shards. Each shard's fair share of traffic is:

$\frac{50{,}000}{20} = 2{,}500 \text{ QPS per shard}$

Now suppose a high-value user segment receives materially heavier personalization than typical users, more feature lookups and more pipeline stages per request, and that this segment's keys happen to land on just 2 of the 20 shards. Even if the segment is a small fraction of the total user base, those 2 shards can see request rates well above the 2,500 QPS fair share while the fleet-wide average still looks healthy, because the imbalance is hidden by averaging across all 20 shards. Two levers address this: technically, give that segment's keys a dedicated local cache tier or replicate them across more shards so no single shard owns them exclusively; on the product side, accept a slightly lighter-weight or briefly staler personalized experience for that segment specifically when its shard is under pressure, rather than let it degrade latency for the rest of the service.

Trade-offs & pitfalls

• Version everything (predictions, features, and artifacts alike) against model_version; the single most common failure mode in machine-learning-serving caches is a rollout that updates the model but leaves a cache serving outputs computed under the old one.
• Don't apply the same freshness intuition to all three cache types: a stale artifact is mostly a latency and correctness-of-version problem, while a stale feature or prediction is a silent model-quality problem that won't throw an error, it just quietly gets the answer wrong.
• Monitor downstream business or model-quality metrics alongside cache hit rate, not just hit rate on its own; a healthy hit rate can coexist with a staleness window that's actually hurting accuracy if nobody is watching for that specific signal.
• Pre-warming and partial invalidation both add operational complexity; they're worth it for expensive, externally-enriched features specifically, not as a default applied to every feature in the store.

Direct answer

Design the policy around a priority hierarchy (latency-critical inference above cost-sensitive batch training) enforced through quotas and preemption, size placement and pre-warming to each hardware type's actual warm-up behavior instead of one blanket policy, and treat cost optimization as bin-packing (fitting multiple workloads onto the fewest possible machines or GPUs by matching each workload's resource need to available capacity, the same way you'd pack boxes of different sizes into as few shipping crates as possible) under multiple simultaneous service-level objectives (SLOs, service-level objectives) rather than a single throughput target.

Structured elaboration

1. Coordinate autoscaling between fast-warm-up CPU pods and slow-warm-up GPU pods.
CPU pods and GPU pods do not become ready on the same timescale: a fast-warm-up CPU pod is typically ready to serve within seconds of being scheduled, while a slow-warm-up GPU pod carries a much longer warm-up (driver initialization, loading model weights onto the device, warming CUDA kernels) before it can serve traffic. An autoscaler that reacts to load with a single reaction-time assumption will either over-provision CPU capacity (scaling conservatively enough for the GPU pod's slow warm-up even where it isn't needed) or under-provision GPU capacity (scaling reactively as if it were as fast as the CPU pod, then missing the SLO during the warm-up window). Coordinating them means giving each hardware type its own scale-out lead time in the autoscaling policy, and triggering GPU scale-out earlier, on a leading demand signal, rather than waiting for the same threshold that triggers CPU scale-out.

2. Design GPU-pool request routing and pre-warming around predicted demand, not just current demand.
Placement decides which model runs on which physical node; pre-warming decides how much idle-but-ready capacity to keep so a scale-out event does not have to pay the full cold-start cost on the critical path. Within a shared GPU pool, request routing should maximize throughput under the latency SLOs by directing traffic toward pre-warmed instances first and using eviction of idle pre-warmed capacity to reclaim it for other models once demand shifts, rather than routing blindly and letting some instances sit cold while others queue. For latency-sensitive endpoints, keep a small pre-warmed buffer sized to cover the warm-up window's worth of expected traffic growth, so a burst is served by pre-warmed capacity while new capacity is still coming online behind it. This directly addresses the cold-start/warmup problem taxonomy and its mitigations for latency-sensitive endpoints: pre-warming, keeping a warm standby pool, eviction of stale warm capacity, and (where acceptable) routing overflow traffic to a smaller or quantized fallback model while the primary scales up, rather than making every request wait through a cold start.

3. Bin-pack for cost, subject to the SLOs, not for utilization alone.
Model placement should pack multiple lower-traffic models onto shared hardware where their combined resource need fits without violating any model's latency or throughput SLO, rather than maximizing raw utilization irrespective of latency. Bin-packing efficiency and latency SLO compliance are in tension: packing too tightly introduces contention (from shared memory bandwidth or scheduling latency) that can push a co-located model's P95 (95th-percentile latency) past its target, so the packing algorithm needs an explicit headroom margin, not just a greedy best-fit on capacity.

4. Use priority queues and preemption to protect latency-critical work.
Give inference requests a higher scheduling priority than batch training jobs on shared hardware. Structure this as preemptible training: a training job can be evicted from its allocation (after checkpointing) to free capacity for an inference burst or a higher-priority job, then resumed later, so the cluster does not need to be sized for training and inference's simultaneous peak just to avoid the two workloads contending. This is the same mechanism that prevents training jobs from starving latency-critical inference: without preemption, inference must wait behind whatever training happened to claim the resource first.

5. Schedule cost-sensitive batch work opportunistically, meeting deadlines via spot/preemptible instances and bin-packing.
Cost-sensitive batch-job scheduling for training jobs with a deadline rather than a hard latency requirement is the right fit for spot/preemptible instances, scheduled via bin-packing to fit the gaps left by higher-priority work, as long as the job can tolerate being preempted and resumed before its deadline. This is also where preemption/reservation scheduling matters: a job close enough to its deadline should hold a reservation (guaranteed, non-preemptible capacity) rather than remain preemptible and risk missing it, which is the cost/latency trade-off made concrete at the level of an individual job rather than the whole cluster.

6. Allocate fairly across competing teams.
Multiple teams sharing the same heterogeneous fleet need a fair-share model: a guaranteed quota floor per team so no team is starved by another's burst, priority tiers above that floor for contended burst capacity, and noisy-neighbor mitigation (resource limits, and for GPUs specifically, isolation between processes sharing a device) so one team's workload cannot silently degrade another's latency on shared hardware. Cost attribution (metering usage per team against their quota and their burst consumption) is what makes the fair-share model auditable and lets a team's cost be traced back to its own workload rather than absorbed into a shared, unattributed cluster bill.

Worked example

flowchart TD
  A[Inference request arrives] --> B{Warm capacity available?}
  B -->|Yes| C[Serve from pre-warmed pool]
  B -->|No| D[Trigger scale-out on leading signal]
  D --> E{CPU or GPU pod?}
  E -->|CPU: fast warm-up| F[Ready in seconds]
  E -->|GPU: slow warm-up| G[Route overflow to fallback model during warm-up]
  G --> H[GPU pod ready, resume normal routing]
  I[Batch training job] --> J{Contends with inference for capacity?}
  J -->|Yes| K[Checkpoint and preempt training]
  J -->|No| L[Run opportunistically on spot capacity]

For the bin-packing and cost trade-off specifically: assume, as an illustrative planning input rather than a measured fact, that three small model services each need 30% of one GPU's peak throughput capacity, and that fractional-GPU scheduling (time-slicing or a multi-instance mode) achieves 90% packing efficiency in practice, reflecting scheduling and context-switch overhead rather than 100% ideal sharing.

All three models fit on a single shared GPU instead of three dedicated ones, a 3x reduction in GPU count for this slice of the fleet, as long as each model's individual latency SLO tolerates the contention that comes with sharing.

Trade-offs & pitfalls

The most common mistake is applying one autoscaling reaction time to both CPU and GPU pods, which either wastes CPU capacity or misses the GPU's SLO during its longer warm-up; the two need separately tuned lead times, not a shared threshold. A second is bin-packing purely for utilization without headroom for co-location contention, which looks efficient in a capacity dashboard right up until a shared model's P95 latency SLO starts failing under real concurrent load. A third is preemption without a deadline-awareness check: a training job repeatedly preempted right before its deadline either misses it or forces an expensive last-minute escalation to guaranteed capacity, which a policy that tracks time-to-deadline against preemption history can catch earlier. Finally, fair-share allocation without real cost attribution tends to erode over time, since teams have no visibility into whether their burst usage is reasonable, and the quota floors end up being renegotiated by escalation rather than by data.

Direct answer

Default to stateless serving with session state externalized to a shared store (Redis or a similar key-value store): it preserves horizontal scalability and fault tolerance without extra work. Prefer in-process stateful serving only when all three of these hold at once: the latency budget genuinely can't absorb an external round trip, the session state is small enough to comfortably live in memory per replica, and the routing layer can guarantee a user's repeated requests reach the same replica (sticky routing). An online recommendation model using short-term interaction history often sits right at this boundary, which is why the right answer is usually a hybrid rather than a pure choice.

Structured elaboration

Stateless serving. Inference is a pure function of model plus input, with any needed context (recent interactions, session state) fetched from an external store on each request. Any replica can serve any request, which is what makes this pattern trivially horizontally scalable and fault tolerant.

Stateful serving. The serving process keeps session state (recent interaction history, a user's short-term embedding) in memory across requests, avoiding a store round trip on every call. This lowers per-request latency and load on the external store, but only works if requests from the same user keep landing on the same replica.

Worked example

Consider a recommendation service reading a user's last-20-interactions window to personalize the next set of results. That window is small (comfortably fits in memory per active user) and short-lived (it doesn't need to survive indefinitely), which are exactly the conditions favoring a stateful approach: route each user's requests to the same replica (this is a routing-layer requirement, not something the serving process controls on its own), keep the last-20-interactions window in an in-memory structure per active session, and periodically checkpoint it asynchronously to an external store so a replica crash doesn't lose the window outright, only whatever changed since the last checkpoint.

Contrast that with a version of the same product that personalizes off a user's full browsing history rather than a short recent window: that state is no longer small or bounded, it grows over the life of the account, and it stops meeting the "fits comfortably in memory per replica" condition that made stateful serving attractive in the first place. At that point, externalizing to a store built for large, sharded key-value access (rather than trying to keep it resident in the serving process) is the right call, even though it reintroduces the external round trip the stateful design was trying to avoid.

The practical middle ground most real recommendation services land on: keep the tiny, latency-sensitive recent-interaction window in-process for fast reads, but treat it as a cache in front of an authoritative external store, not as the only copy, so a crash degrades gracefully (falls back to the external store) instead of losing the session outright.

Trade-offs & pitfalls

• Stateful serving requires the load balancer to route a given user's repeated requests to the same replica; that routing requirement is itself a scaling constraint being traded for lower latency, and it needs to be designed deliberately rather than assumed.
• "Short-term" session state can grow unbounded if nothing enforces a size or time limit; a design that was safely small at launch can silently become a memory-pressure problem as usage patterns shift.
• Treating in-memory state as durable is the most common failure mode: without an async checkpoint to an external store, a crash doesn't just lose one request, it loses that user's entire session context.
• Externalizing everything by default avoids all of the above at the cost of a network round trip on every request; that's the right trade for most services, and only worth giving up when the latency budget and state size genuinely demand it.