Scalability Patterns and Techniques Questions

Direct answer

For a small team starting from a proof of concept serving roughly 1,000 requests/sec at 50ms P95 latency (95th-percentile latency), I'd start with the monolithic multi-model service and split into per-model microservices only once a specific model's traffic, resource profile, or release cadence diverges enough from the others to justify the added operational cost; premature per-model decomposition trades a real, immediate development-velocity cost for a scaling flexibility benefit the team may not need yet.

Structured elaboration

Development velocity. A monolithic service means one deployable, one test suite, one place to add a new model or fix a shared preprocessing bug. A small team without dedicated platform support pays a real tax for coordinating N separate services (N sets of CI/CD, N sets of dependency upgrades, N places a shared feature-extraction bug must be fixed identically). Early on, this usually dominates every other factor.

Deployment complexity and testability. One service means one deployment pipeline and one integration-test surface; per-model services mean each model change deploys independently (an advantage once models genuinely evolve on different schedules) but multiplies the number of moving parts that must each be tested, versioned, and rolled back independently.

Coupling. In a monolith, a bug or resource spike in one model's code can degrade another model's latency (shared process, shared memory, shared thread pool). Per-model services isolate that blast radius, at the cost of adding a network hop and a separate failure mode (the model service being unreachable) that did not exist before.

Observability. A monolith's per-model performance is harder to see cleanly, since metrics are aggregated across all models unless the code explicitly tags them. Per-model services get natural per-model dashboards and alerts for free, from the service boundary itself.

Resource utilization. This is the factor most tied to the given numbers, and the one that most often justifies decomposition. If the 1,000 requests/sec is split evenly across similarly-sized models, a monolith's shared resource pool is efficient: idle capacity for one model's low-traffic period is naturally used by another's high-traffic period. If instead demand and resource needs are skewed across models (one model dominates traffic, or one model needs a GPU while others run fine on CPU), decomposition lets you size and scale each model's replica count independently to match its own demand, instead of scaling the whole monolith (and paying for every model's replica) just to keep the hot model within its 50ms P95 latency target.

Framing as a per-model service boundary decision. The same choice can be framed as: does each model get its own per-model service boundary, and what does that do to scaling per model and to team ownership? Drawing that boundary means each model's replica count can scale per model, independently matching its own traffic instead of the aggregate, and typically means a single team (or a single engineer) becomes the clear owner of one model's service, establishing team ownership instead of shared, diffuse responsibility for one large multi-model deployable. That team-ownership clarity is a real benefit as team and model count grow, but it is not free early on, when a small team's context-switching cost across N services can exceed the coordination cost of one shared codebase.

Worked example

Assume, as an illustrative planning input, that of the 1,000 requests/sec total, one model accounts for 700 requests/sec (a hot, latency-critical recommendation model) and the remaining three models share 300 requests/sec combined.

$$700 + 300 = 1{,}000\ \text{requests/sec (total, consistent with the given target)}$$

In the monolith, meeting the 50ms P95 target for the hot model forces you to scale the whole service (all four models' code paths) to the hot model's replica count, so the three cold models are running far more replicas than their own traffic needs, purely as a side effect of sharing a deployable with the hot one. Splitting per model lets the hot model scale to meet its own 700 requests/sec target independently, while the three cold models run a much smaller, cheaper replica count sized to their combined 300 requests/sec, which is the resource-utilization case for decomposition once the traffic split across models is uneven enough to matter.

Trade-offs & pitfalls

The main pitfall is decomposing before there is evidence of the skew that justifies it: a small team that splits into per-model services on day one pays the coordination and operational cost immediately while the utilization benefit is theoretical, since a fresh proof of concept rarely has the traffic history to know which model will actually become the hot one. The better sequence is start monolithic, instrument per-model resource usage and latency from day one even inside the monolith, and split out a specific model only when its profile (traffic share, latency sensitivity, or hardware need such as a GPU) has diverged enough from its siblings to make shared deployment genuinely costly, not just theoretically suboptimal.

Direct answer

Default to stateless serving with session state externalized to a shared store (Redis or a similar key-value store): it preserves horizontal scalability and fault tolerance without extra work. Prefer in-process stateful serving only when all three of these hold at once: the latency budget genuinely can't absorb an external round trip, the session state is small enough to comfortably live in memory per replica, and the routing layer can guarantee a user's repeated requests reach the same replica (sticky routing). An online recommendation model using short-term interaction history often sits right at this boundary, which is why the right answer is usually a hybrid rather than a pure choice.

Structured elaboration

Stateless serving. Inference is a pure function of model plus input, with any needed context (recent interactions, session state) fetched from an external store on each request. Any replica can serve any request, which is what makes this pattern trivially horizontally scalable and fault tolerant.

Stateful serving. The serving process keeps session state (recent interaction history, a user's short-term embedding) in memory across requests, avoiding a store round trip on every call. This lowers per-request latency and load on the external store, but only works if requests from the same user keep landing on the same replica.

Worked example

Consider a recommendation service reading a user's last-20-interactions window to personalize the next set of results. That window is small (comfortably fits in memory per active user) and short-lived (it doesn't need to survive indefinitely), which are exactly the conditions favoring a stateful approach: route each user's requests to the same replica (this is a routing-layer requirement, not something the serving process controls on its own), keep the last-20-interactions window in an in-memory structure per active session, and periodically checkpoint it asynchronously to an external store so a replica crash doesn't lose the window outright, only whatever changed since the last checkpoint.

Contrast that with a version of the same product that personalizes off a user's full browsing history rather than a short recent window: that state is no longer small or bounded, it grows over the life of the account, and it stops meeting the "fits comfortably in memory per replica" condition that made stateful serving attractive in the first place. At that point, externalizing to a store built for large, sharded key-value access (rather than trying to keep it resident in the serving process) is the right call, even though it reintroduces the external round trip the stateful design was trying to avoid.

The practical middle ground most real recommendation services land on: keep the tiny, latency-sensitive recent-interaction window in-process for fast reads, but treat it as a cache in front of an authoritative external store, not as the only copy, so a crash degrades gracefully (falls back to the external store) instead of losing the session outright.

Trade-offs & pitfalls

• Stateful serving requires the load balancer to route a given user's repeated requests to the same replica; that routing requirement is itself a scaling constraint being traded for lower latency, and it needs to be designed deliberately rather than assumed.
• "Short-term" session state can grow unbounded if nothing enforces a size or time limit; a design that was safely small at launch can silently become a memory-pressure problem as usage patterns shift.
• Treating in-memory state as durable is the most common failure mode: without an async checkpoint to an external store, a crash doesn't just lose one request, it loses that user's entire session context.
• Externalizing everything by default avoids all of the above at the cost of a network round trip on every request; that's the right trade for most services, and only worth giving up when the latency budget and state size genuinely demand it.

Direct answer

Forecast GPU capacity by projecting current training and inference demand forward with an explicit growth assumption, translate that into a GPU-hour budget, then split that budget across tenants with quotas and priority tiers rather than first-come-first-served access, and absorb burstiness with a shared preemptible pool instead of over-provisioning every tenant's peak individually.

Structured elaboration

1. Separate the two workload shapes before forecasting.
Nightly training and real-time inference have opposite scheduling profiles: training is throughput-oriented, tolerant of queueing, and can be preempted and resumed from a checkpoint; inference is latency-oriented, cannot queue behind a long training job, and needs guaranteed capacity during business hours. Forecast and quota them separately, then decide how much of the training capacity can double as burst inference capacity (and vice versa) when idle.

2. Forecast from a baseline and a stated growth driver, not a flat trend line.
A 12-month GPU forecast should be driven by concrete inputs you can name: expected model count growth, expected training frequency per model, and expected inference traffic growth (tied to product roadmap, not just historical curve-fitting, since a 12-month horizon on GPU demand is usually driven by planned launches more than organic drift).

3. Define quotas and priorities across tenants.

• Quotas: a guaranteed floor per tenant (hard reservation) sized to their steady-state need, so no tenant is starved by another's burst.
• Priority tiers above the floor: latency-critical inference outranks best-effort training for shared/burst capacity; within training, deadline-bound jobs outrank exploratory ones.
• Fair-share reconciliation: unused quota from an idle tenant should be lend-able to others temporarily, reclaimed when the owning tenant needs it back, which is what makes shared clusters more cost-efficient than static per-tenant carve-outs.

4. Design scheduling and preemption policy around that priority order.
Give inference workloads dedicated, non-preemptible capacity for their guaranteed floor. Run training as preemptible: a training job can be checkpointed and evicted to make room for a burst in inference demand or a higher-priority training job, then resumed. This is the mechanism that lets one cluster serve both workload types without doubling the fleet.

5. Handle bursty demand with a shared elastic pool, not per-tenant peak provisioning.
Provisioning every tenant for their own peak wastes capacity, since peaks rarely align across tenants. A shared burst pool, allocated by priority when contended, covers more aggregate peak with fewer total GPUs, at the cost of occasional queueing for lower-priority work during simultaneous bursts, which the priority tiers make an explicit, accepted trade-off rather than a surprise.

6. Metrics and cost trade-offs.
Track GPU-hour utilization per tenant (catches over-provisioned quotas), queue wait time by priority tier (catches under-provisioned burst capacity), and preemption/checkpoint-restart rate for training (catches a preemption policy that is too aggressive and is wasting compute on repeated restarts). The core cost trade-off is reservation versus utilization: guaranteed floors improve latency predictability for inference but sit idle during off-peak hours; a larger shared/preemptible pool improves utilization but adds queueing risk that inference cannot tolerate, which is exactly why the two workloads get different scheduling treatment rather than one policy for the whole cluster.

Worked example

Assume, as planning inputs rather than measured facts: current combined steady-state demand across training and inference is 350 GPU-hours/day, and demand is expected to grow 8% month-over-month (compounding, reflecting planned model and traffic growth) over the next 12 months.

$$\begin{aligned} \text{forecast demand}_{12} &= 350 \times (1.08)^{12} \approx 881\ \text{GPU-hours/day} \\ \text{required fleet size} &= \frac{881}{24 \times 0.70} \approx 52.5 \;\Rightarrow\; 53\ \text{GPUs} \end{aligned}$$

(24 hours of possible daily capacity per GPU, discounted to a 70% target utilization, flagged as a planning assumption that leaves headroom for burst and preemption overhead rather than running the fleet at the edge of saturation.)

If the cluster serves three tenants weighted 50/30/20 by their steady-state share, that 53-GPU fleet implies guaranteed floors of roughly 26-27 / 16 / 10-11 GPUs, with any unused share reclaimable by the burst pool rather than sitting idle.

Trade-offs & pitfalls

The main failure mode is applying one scheduling policy to both workloads: inference queued behind a preemptible training job causes latency SLO (service-level objective) violations, while training that is never preemptible forces the fleet to be sized for the sum of every tenant's peak instead of a shared pool, which is far more expensive. A second common mistake is forecasting GPU demand the way CPU demand is forecast (a smooth trend line) when GPU demand in most orgs is lumpy and roadmap-driven; the growth assumption should be tied to named upcoming work, and revisited quarterly rather than trusted for the full 12 months unchanged.

Direct answer

For a 50ms P95 (95th-percentile) latency budget, prefer horizontal scaling with right-sized graphics processing unit (GPU) replicas as the default, and reserve vertical scaling (bigger GPUs) and model sharding (splitting one model across multiple GPUs) for cases that genuinely need them: sharding only when the model doesn't fit on one GPU's memory, and bigger single instances only when batching economics clearly favor concentration over replication for your actual traffic shape. Neither is universally cheaper; the deciding factors are cold-start behavior, batching efficiency, and how tightly your latency budget constrains batch size.

Structured elaboration

Vertical (bigger GPU instances)

• Fewer, larger units mean model loading happens less often across the fleet, and a bigger memory/compute pool can build larger batches within the same latency budget, which amortizes per-request fixed overhead (kernel launch, memory transfer) more efficiently.
• Cost scales non-linearly with GPU size; the largest instance tiers carry a real price premium per unit of compute, and a single large instance is also a larger single point of failure.

Horizontal (more replicas, or model sharding for oversized models)

• Elastic: capacity tracks demand by adding or removing replicas, and losing one replica out of many degrades capacity gracefully rather than taking the service down.
• Each replica typically loads its own full copy of the model (unless sharded), so scaling out means more frequent cold starts across the fleet during scale events, and inter-replica batching coordination is looser than within one large instance.
• Model sharding (splitting a model too large for one GPU across several, communicating over an interconnect) trades added inter-GPU communication latency for the ability to serve a model that otherwise wouldn't fit at all; it's a "how do we serve this" answer, not a "how do we get cheaper" answer, and it adds real operational complexity: shard placement, inter-shard routing, and keeping every shard on a consistent model version.

Cost per prediction, worked concretely below, does not have a universal winner; it depends on the batching efficiency curve of the actual model and hardware.

Cold start and GPU utilization. GPU instances take real, non-trivial time to become ready (instance boot plus model load into GPU memory), which is qualitatively slower than a small stateless web replica's readiness. This directly constrains autoscaling: a purely reactive horizontal policy that scales out only after load has already arrived will miss the 50ms tail-latency target during the ramp, since new replicas aren't instantly useful the moment they boot. That pushes toward a pre-warmed pool (the same pattern used for any latency-critical, slow-to-initialize service) rather than cold reactive scale-out.

Operational complexity. Vertical scaling with a handful of large, fully-loaded instances is operationally the simplest: fewer things to monitor, deploy, and keep in sync. Horizontal scaling with plain replication adds fleet-management overhead but nothing conceptually new; model sharding adds the most complexity of the three, since it introduces distributed-inference concerns (routing a request to the right shard set, keeping shards version-consistent) that a single-GPU deployment never has to solve.

Worked example

Cost-per-prediction comparison. These are illustrative, stated dollar figures for the purpose of the calculation, not live cloud pricing. Assume a large GPU instance costs $4.00/hour and, with batching, sustains 200 predictions/second:

$$\frac{\$4.00/\text{hr}}{200 \ \text{preds/s} \times 3600 \ \text{s/hr}} \approx \$5.56 \text{ per million predictions}$$

A smaller GPU instance costs $1.20/hour and, with less memory headroom for batching, sustains 50 predictions/second:

$$\frac{\$1.20/\text{hr}}{50 \ \text{preds/s} \times 3600 \ \text{s/hr}} \approx \$6.67 \text{ per million predictions}$$

Matching the big instance's 200 preds/sec aggregate throughput takes 4 small instances:

$$4 \times \$1.20/\text{hr} = \$4.80/\text{hr} \quad (\text{vs. } \$4.00/\text{hr for the one big instance})$$

In this illustrative model, the horizontally-scaled fleet costs about 20% more per prediction than the single large instance at equal aggregate throughput, purely because the larger instance's bigger batches amortize fixed per-request overhead more efficiently. This direction is not a general rule; it reverses as soon as the workload is spiky rather than steady (the four small instances can independently scale down when idle, while the one big instance is an all-or-nothing cost), or as soon as the model doesn't fit on the smaller GPU tier at all, at which point sharding or moving to the larger instance stops being optional.

Recommended architecture

A hybrid: right-sized (not maximal) GPU replicas running behind a small pre-warmed pool sized to the expected spike arrival rate, each replica running dynamic batching tuned so the batch-wait window plus inference time stays inside the 50ms P95 budget. Reserve model sharding strictly for models that don't fit on a single GPU's memory, since it's the highest-operational-complexity option and buys nothing for a model that already fits on one card. This gets most of vertical scaling's batching efficiency (via per-replica batching) while keeping horizontal's elasticity and fault isolation, and it avoids paying sharding's coordination cost for a problem sharding wasn't needed to solve.

Trade-offs & pitfalls

• Chasing cost efficiency with ever-larger batches can quietly blow the latency budget, since a bigger batch takes longer both to fill and to compute; batch size has to be tuned against the P95 target, not against throughput alone.
• Treating horizontal scaling as free, instant elasticity while ignoring GPU cold-start time is the most common mistake here; a freshly-booted GPU replica is not immediately equivalent to a warm one the way a small stateless web replica usually is.
• Reaching for model sharding as a scaling strategy (rather than a "the model doesn't fit" necessity) adds the most operational complexity of any option here for the least benefit when the model actually fits on one GPU.
• The cost-per-prediction comparison above assumes steady, batchable traffic; a spiky, unpredictable traffic pattern changes the calculus toward horizontal's elasticity regardless of the flat-throughput cost numbers.

Direct answer

Attack a CPU-bound feature-computation bottleneck in priority order of how much online cost each technique removes: first eliminate work entirely by precomputing what can be computed ahead of the request, then approximate what cannot be precomputed to shrink its cost, then make what remains on the hot path run faster per unit of work, and only move to a specialized service once the computation itself no longer fits comfortably in the request path at all.

Structured elaboration

1. Precompute what does not depend on the live request.
Any feature that is a function of data known before the request arrives (a user's rolling 7-day activity count, an item's historical embedding) can be computed offline on a schedule and looked up online instead of recomputed. This removes the CPU cost from the request path entirely rather than reducing it, which is why it is the first thing to check. The cost is staleness: the feature reflects the world as of the last precompute run, not the instant of the request.

2. Approximate or quantize what must stay on the request path.
For features that genuinely depend on request-time data, ask whether full precision is needed. Quantized representations (lower-precision numeric types) and approximate algorithms (sketches, sampled aggregates instead of exact ones) trade a bounded, quantifiable accuracy loss for a real reduction in CPU and memory cost per request. This only works if the accuracy loss is measured against the actual downstream metric that matters (model accuracy, ranking quality), not assumed to be negligible.

3. Make the remaining hot-path code fast per unit of work.
Once you've minimized how much computation happens online, optimize what's left: replace interpreted per-element loops with vectorized array operations, or use just-in-time (JIT) compilation to compile hot numeric code paths to native instructions ahead of first use. This is pure engineering efficiency with no accuracy trade-off, which is why it belongs after precompute and approximation, not before them: it is wasted effort to hand-optimize code you could have avoided running at all.

4. Move heavy transforms to a specialized service only when the above are not enough.
If the feature computation is inherently expensive even after the previous steps (a transform that genuinely needs specialized hardware or a different runtime), extracting it into its own service lets it scale, batch, and be optimized independently of the request-handling tier. The cost is a network hop and an added operational dependency, so this is a structural change reserved for computations that cannot be shrunk further, not a first response to a slow function.

Worked example

This uses Amdahl's-law-style reasoning: if feature computation is a fraction $f$ of total per-request CPU cost and a technique reduces that fraction's cost by a factor $k$ (leaving it unchanged if $k=1$, removing it entirely if $k \to \infty$), the overall speedup on a fixed-size fleet is:

$$\text{speedup} = \frac{1}{(1-f) + f/k}$$

Assume, as an illustrative planning input rather than a measured benchmark: profiling shows feature computation is 75% of per-request CPU cost ($f = 0.75$).

$$\begin{aligned} \text{precompute (removes it, } k \to \infty \text{)}: &\quad \text{speedup} = \frac{1}{0.25} = 4\times \\ \text{vectorize online (3x faster on that portion, } k = 3 \text{)}: &\quad \text{speedup} = \frac{1}{0.25 + 0.25} = 2\times \end{aligned}$$

The formula makes the priority order concrete: precomputing (removing the term entirely) beats vectorizing the same work in place, because $k \to \infty$ always dominates any finite $k$. That is the quantitative reason "eliminate before optimize" is the right order, not just a rule of thumb.

Trade-offs & pitfalls

Precomputing trades accuracy-at-the-instant for speed (staleness), which is invisible until a downstream consumer needs a feature that just changed and gets the old value; it needs an explicit staleness bound the consumer can reason about. Approximation and quantization trade measured accuracy loss for speed, and the mistake is skipping the measurement step and assuming the loss is negligible. Vectorization and JIT compilation are the safest technique (no accuracy cost) but the smallest lever if the fraction of cost they address ($f$) is small, which is exactly what the speedup formula shows. Extracting a specialized service is the most expensive option operationally (new deployment, new failure mode, network latency added to the path) and should be justified by the same fraction-of-cost reasoning, not reached for first because it feels like the "real" fix.