Scalability Patterns and Techniques Questions

Direct answer

Default to stateless serving with session state externalized to a shared store (Redis or a similar key-value store): it preserves horizontal scalability and fault tolerance without extra work. Prefer in-process stateful serving only when all three of these hold at once: the latency budget genuinely can't absorb an external round trip, the session state is small enough to comfortably live in memory per replica, and the routing layer can guarantee a user's repeated requests reach the same replica (sticky routing). An online recommendation model using short-term interaction history often sits right at this boundary, which is why the right answer is usually a hybrid rather than a pure choice.

Structured elaboration

Stateless serving. Inference is a pure function of model plus input, with any needed context (recent interactions, session state) fetched from an external store on each request. Any replica can serve any request, which is what makes this pattern trivially horizontally scalable and fault tolerant.

Stateful serving. The serving process keeps session state (recent interaction history, a user's short-term embedding) in memory across requests, avoiding a store round trip on every call. This lowers per-request latency and load on the external store, but only works if requests from the same user keep landing on the same replica.

Worked example

Consider a recommendation service reading a user's last-20-interactions window to personalize the next set of results. That window is small (comfortably fits in memory per active user) and short-lived (it doesn't need to survive indefinitely), which are exactly the conditions favoring a stateful approach: route each user's requests to the same replica (this is a routing-layer requirement, not something the serving process controls on its own), keep the last-20-interactions window in an in-memory structure per active session, and periodically checkpoint it asynchronously to an external store so a replica crash doesn't lose the window outright, only whatever changed since the last checkpoint.

Contrast that with a version of the same product that personalizes off a user's full browsing history rather than a short recent window: that state is no longer small or bounded, it grows over the life of the account, and it stops meeting the "fits comfortably in memory per replica" condition that made stateful serving attractive in the first place. At that point, externalizing to a store built for large, sharded key-value access (rather than trying to keep it resident in the serving process) is the right call, even though it reintroduces the external round trip the stateful design was trying to avoid.

The practical middle ground most real recommendation services land on: keep the tiny, latency-sensitive recent-interaction window in-process for fast reads, but treat it as a cache in front of an authoritative external store, not as the only copy, so a crash degrades gracefully (falls back to the external store) instead of losing the session outright.

Trade-offs & pitfalls

• Stateful serving requires the load balancer to route a given user's repeated requests to the same replica; that routing requirement is itself a scaling constraint being traded for lower latency, and it needs to be designed deliberately rather than assumed.
• "Short-term" session state can grow unbounded if nothing enforces a size or time limit; a design that was safely small at launch can silently become a memory-pressure problem as usage patterns shift.
• Treating in-memory state as durable is the most common failure mode: without an async checkpoint to an external store, a crash doesn't just lose one request, it loses that user's entire session context.
• Externalizing everything by default avoids all of the above at the cost of a network round trip on every request; that's the right trade for most services, and only worth giving up when the latency budget and state size genuinely demand it.

Direct answer

Design the policy around a priority hierarchy (latency-critical inference above cost-sensitive batch training) enforced through quotas and preemption, size placement and pre-warming to each hardware type's actual warm-up behavior instead of one blanket policy, and treat cost optimization as bin-packing (fitting multiple workloads onto the fewest possible machines or GPUs by matching each workload's resource need to available capacity, the same way you'd pack boxes of different sizes into as few shipping crates as possible) under multiple simultaneous service-level objectives (SLOs, service-level objectives) rather than a single throughput target.

Structured elaboration

1. Coordinate autoscaling between fast-warm-up CPU pods and slow-warm-up GPU pods.
CPU pods and GPU pods do not become ready on the same timescale: a fast-warm-up CPU pod is typically ready to serve within seconds of being scheduled, while a slow-warm-up GPU pod carries a much longer warm-up (driver initialization, loading model weights onto the device, warming CUDA kernels) before it can serve traffic. An autoscaler that reacts to load with a single reaction-time assumption will either over-provision CPU capacity (scaling conservatively enough for the GPU pod's slow warm-up even where it isn't needed) or under-provision GPU capacity (scaling reactively as if it were as fast as the CPU pod, then missing the SLO during the warm-up window). Coordinating them means giving each hardware type its own scale-out lead time in the autoscaling policy, and triggering GPU scale-out earlier, on a leading demand signal, rather than waiting for the same threshold that triggers CPU scale-out.

2. Design GPU-pool request routing and pre-warming around predicted demand, not just current demand.
Placement decides which model runs on which physical node; pre-warming decides how much idle-but-ready capacity to keep so a scale-out event does not have to pay the full cold-start cost on the critical path. Within a shared GPU pool, request routing should maximize throughput under the latency SLOs by directing traffic toward pre-warmed instances first and using eviction of idle pre-warmed capacity to reclaim it for other models once demand shifts, rather than routing blindly and letting some instances sit cold while others queue. For latency-sensitive endpoints, keep a small pre-warmed buffer sized to cover the warm-up window's worth of expected traffic growth, so a burst is served by pre-warmed capacity while new capacity is still coming online behind it. This directly addresses the cold-start/warmup problem taxonomy and its mitigations for latency-sensitive endpoints: pre-warming, keeping a warm standby pool, eviction of stale warm capacity, and (where acceptable) routing overflow traffic to a smaller or quantized fallback model while the primary scales up, rather than making every request wait through a cold start.

3. Bin-pack for cost, subject to the SLOs, not for utilization alone.
Model placement should pack multiple lower-traffic models onto shared hardware where their combined resource need fits without violating any model's latency or throughput SLO, rather than maximizing raw utilization irrespective of latency. Bin-packing efficiency and latency SLO compliance are in tension: packing too tightly introduces contention (from shared memory bandwidth or scheduling latency) that can push a co-located model's P95 (95th-percentile latency) past its target, so the packing algorithm needs an explicit headroom margin, not just a greedy best-fit on capacity.

4. Use priority queues and preemption to protect latency-critical work.
Give inference requests a higher scheduling priority than batch training jobs on shared hardware. Structure this as preemptible training: a training job can be evicted from its allocation (after checkpointing) to free capacity for an inference burst or a higher-priority job, then resumed later, so the cluster does not need to be sized for training and inference's simultaneous peak just to avoid the two workloads contending. This is the same mechanism that prevents training jobs from starving latency-critical inference: without preemption, inference must wait behind whatever training happened to claim the resource first.

5. Schedule cost-sensitive batch work opportunistically, meeting deadlines via spot/preemptible instances and bin-packing.
Cost-sensitive batch-job scheduling for training jobs with a deadline rather than a hard latency requirement is the right fit for spot/preemptible instances, scheduled via bin-packing to fit the gaps left by higher-priority work, as long as the job can tolerate being preempted and resumed before its deadline. This is also where preemption/reservation scheduling matters: a job close enough to its deadline should hold a reservation (guaranteed, non-preemptible capacity) rather than remain preemptible and risk missing it, which is the cost/latency trade-off made concrete at the level of an individual job rather than the whole cluster.

6. Allocate fairly across competing teams.
Multiple teams sharing the same heterogeneous fleet need a fair-share model: a guaranteed quota floor per team so no team is starved by another's burst, priority tiers above that floor for contended burst capacity, and noisy-neighbor mitigation (resource limits, and for GPUs specifically, isolation between processes sharing a device) so one team's workload cannot silently degrade another's latency on shared hardware. Cost attribution (metering usage per team against their quota and their burst consumption) is what makes the fair-share model auditable and lets a team's cost be traced back to its own workload rather than absorbed into a shared, unattributed cluster bill.

Worked example

flowchart TD
  A[Inference request arrives] --> B{Warm capacity available?}
  B -->|Yes| C[Serve from pre-warmed pool]
  B -->|No| D[Trigger scale-out on leading signal]
  D --> E{CPU or GPU pod?}
  E -->|CPU: fast warm-up| F[Ready in seconds]
  E -->|GPU: slow warm-up| G[Route overflow to fallback model during warm-up]
  G --> H[GPU pod ready, resume normal routing]
  I[Batch training job] --> J{Contends with inference for capacity?}
  J -->|Yes| K[Checkpoint and preempt training]
  J -->|No| L[Run opportunistically on spot capacity]

For the bin-packing and cost trade-off specifically: assume, as an illustrative planning input rather than a measured fact, that three small model services each need 30% of one GPU's peak throughput capacity, and that fractional-GPU scheduling (time-slicing or a multi-instance mode) achieves 90% packing efficiency in practice, reflecting scheduling and context-switch overhead rather than 100% ideal sharing.

All three models fit on a single shared GPU instead of three dedicated ones, a 3x reduction in GPU count for this slice of the fleet, as long as each model's individual latency SLO tolerates the contention that comes with sharing.

Trade-offs & pitfalls

The most common mistake is applying one autoscaling reaction time to both CPU and GPU pods, which either wastes CPU capacity or misses the GPU's SLO during its longer warm-up; the two need separately tuned lead times, not a shared threshold. A second is bin-packing purely for utilization without headroom for co-location contention, which looks efficient in a capacity dashboard right up until a shared model's P95 latency SLO starts failing under real concurrent load. A third is preemption without a deadline-awareness check: a training job repeatedly preempted right before its deadline either misses it or forces an expensive last-minute escalation to guaranteed capacity, which a policy that tracks time-to-deadline against preemption history can catch earlier. Finally, fair-share allocation without real cost attribution tends to erode over time, since teams have no visibility into whether their burst usage is reasonable, and the quota floors end up being renegotiated by escalation rather than by data.

Direct answer

Attack a CPU-bound feature-computation bottleneck in priority order of how much online cost each technique removes: first eliminate work entirely by precomputing what can be computed ahead of the request, then approximate what cannot be precomputed to shrink its cost, then make what remains on the hot path run faster per unit of work, and only move to a specialized service once the computation itself no longer fits comfortably in the request path at all.

Structured elaboration

1. Precompute what does not depend on the live request.
Any feature that is a function of data known before the request arrives (a user's rolling 7-day activity count, an item's historical embedding) can be computed offline on a schedule and looked up online instead of recomputed. This removes the CPU cost from the request path entirely rather than reducing it, which is why it is the first thing to check. The cost is staleness: the feature reflects the world as of the last precompute run, not the instant of the request.

2. Approximate or quantize what must stay on the request path.
For features that genuinely depend on request-time data, ask whether full precision is needed. Quantized representations (lower-precision numeric types) and approximate algorithms (sketches, sampled aggregates instead of exact ones) trade a bounded, quantifiable accuracy loss for a real reduction in CPU and memory cost per request. This only works if the accuracy loss is measured against the actual downstream metric that matters (model accuracy, ranking quality), not assumed to be negligible.

3. Make the remaining hot-path code fast per unit of work.
Once you've minimized how much computation happens online, optimize what's left: replace interpreted per-element loops with vectorized array operations, or use just-in-time (JIT) compilation to compile hot numeric code paths to native instructions ahead of first use. This is pure engineering efficiency with no accuracy trade-off, which is why it belongs after precompute and approximation, not before them: it is wasted effort to hand-optimize code you could have avoided running at all.

4. Move heavy transforms to a specialized service only when the above are not enough.
If the feature computation is inherently expensive even after the previous steps (a transform that genuinely needs specialized hardware or a different runtime), extracting it into its own service lets it scale, batch, and be optimized independently of the request-handling tier. The cost is a network hop and an added operational dependency, so this is a structural change reserved for computations that cannot be shrunk further, not a first response to a slow function.

Worked example

This uses Amdahl's-law-style reasoning: if feature computation is a fraction $f$ of total per-request CPU cost and a technique reduces that fraction's cost by a factor $k$ (leaving it unchanged if $k=1$, removing it entirely if $k \to \infty$), the overall speedup on a fixed-size fleet is:

Assume, as an illustrative planning input rather than a measured benchmark: profiling shows feature computation is 75% of per-request CPU cost ($f = 0.75$).

The formula makes the priority order concrete: precomputing (removing the term entirely) beats vectorizing the same work in place, because $k \to \infty$ always dominates any finite $k$. That is the quantitative reason "eliminate before optimize" is the right order, not just a rule of thumb.

Trade-offs & pitfalls

Precomputing trades accuracy-at-the-instant for speed (staleness), which is invisible until a downstream consumer needs a feature that just changed and gets the old value; it needs an explicit staleness bound the consumer can reason about. Approximation and quantization trade measured accuracy loss for speed, and the mistake is skipping the measurement step and assuming the loss is negligible. Vectorization and JIT compilation are the safest technique (no accuracy cost) but the smallest lever if the fraction of cost they address ($f$) is small, which is exactly what the speedup formula shows. Extracting a specialized service is the most expensive option operationally (new deployment, new failure mode, network latency added to the path) and should be justified by the same fraction-of-cost reasoning, not reached for first because it feels like the "real" fix.

Direct answer

Forecast GPU capacity by projecting current training and inference demand forward with an explicit growth assumption, translate that into a GPU-hour budget, then split that budget across tenants with quotas and priority tiers rather than first-come-first-served access, and absorb burstiness with a shared preemptible pool instead of over-provisioning every tenant's peak individually.

Structured elaboration

1. Separate the two workload shapes before forecasting.
Nightly training and real-time inference have opposite scheduling profiles: training is throughput-oriented, tolerant of queueing, and can be preempted and resumed from a checkpoint; inference is latency-oriented, cannot queue behind a long training job, and needs guaranteed capacity during business hours. Forecast and quota them separately, then decide how much of the training capacity can double as burst inference capacity (and vice versa) when idle.

2. Forecast from a baseline and a stated growth driver, not a flat trend line.
A 12-month GPU forecast should be driven by concrete inputs you can name: expected model count growth, expected training frequency per model, and expected inference traffic growth (tied to product roadmap, not just historical curve-fitting, since a 12-month horizon on GPU demand is usually driven by planned launches more than organic drift).

3. Define quotas and priorities across tenants.

• Quotas: a guaranteed floor per tenant (hard reservation) sized to their steady-state need, so no tenant is starved by another's burst.
• Priority tiers above the floor: latency-critical inference outranks best-effort training for shared/burst capacity; within training, deadline-bound jobs outrank exploratory ones.
• Fair-share reconciliation: unused quota from an idle tenant should be lend-able to others temporarily, reclaimed when the owning tenant needs it back, which is what makes shared clusters more cost-efficient than static per-tenant carve-outs.

4. Design scheduling and preemption policy around that priority order.
Give inference workloads dedicated, non-preemptible capacity for their guaranteed floor. Run training as preemptible: a training job can be checkpointed and evicted to make room for a burst in inference demand or a higher-priority training job, then resumed. This is the mechanism that lets one cluster serve both workload types without doubling the fleet.

5. Handle bursty demand with a shared elastic pool, not per-tenant peak provisioning.
Provisioning every tenant for their own peak wastes capacity, since peaks rarely align across tenants. A shared burst pool, allocated by priority when contended, covers more aggregate peak with fewer total GPUs, at the cost of occasional queueing for lower-priority work during simultaneous bursts, which the priority tiers make an explicit, accepted trade-off rather than a surprise.

6. Metrics and cost trade-offs.
Track GPU-hour utilization per tenant (catches over-provisioned quotas), queue wait time by priority tier (catches under-provisioned burst capacity), and preemption/checkpoint-restart rate for training (catches a preemption policy that is too aggressive and is wasting compute on repeated restarts). The core cost trade-off is reservation versus utilization: guaranteed floors improve latency predictability for inference but sit idle during off-peak hours; a larger shared/preemptible pool improves utilization but adds queueing risk that inference cannot tolerate, which is exactly why the two workloads get different scheduling treatment rather than one policy for the whole cluster.

Worked example

Assume, as planning inputs rather than measured facts: current combined steady-state demand across training and inference is 350 GPU-hours/day, and demand is expected to grow 8% month-over-month (compounding, reflecting planned model and traffic growth) over the next 12 months.

(24 hours of possible daily capacity per GPU, discounted to a 70% target utilization, flagged as a planning assumption that leaves headroom for burst and preemption overhead rather than running the fleet at the edge of saturation.)

If the cluster serves three tenants weighted 50/30/20 by their steady-state share, that 53-GPU fleet implies guaranteed floors of roughly 26-27 / 16 / 10-11 GPUs, with any unused share reclaimable by the burst pool rather than sitting idle.

Trade-offs & pitfalls

The main failure mode is applying one scheduling policy to both workloads: inference queued behind a preemptible training job causes latency SLO (service-level objective) violations, while training that is never preemptible forces the fleet to be sized for the sum of every tenant's peak instead of a shared pool, which is far more expensive. A second common mistake is forecasting GPU demand the way CPU demand is forecast (a smooth trend line) when GPU demand in most orgs is lumpy and roadmap-driven; the growth assumption should be tied to named upcoming work, and revisited quarterly rather than trusted for the full 12 months unchanged.

Direct answer

For a small team starting from a proof of concept serving roughly 1,000 requests/sec at 50ms P95 latency (95th-percentile latency), I'd start with the monolithic multi-model service and split into per-model microservices only once a specific model's traffic, resource profile, or release cadence diverges enough from the others to justify the added operational cost; premature per-model decomposition trades a real, immediate development-velocity cost for a scaling flexibility benefit the team may not need yet.

Structured elaboration

Development velocity. A monolithic service means one deployable, one test suite, one place to add a new model or fix a shared preprocessing bug. A small team without dedicated platform support pays a real tax for coordinating N separate services (N sets of CI/CD, N sets of dependency upgrades, N places a shared feature-extraction bug must be fixed identically). Early on, this usually dominates every other factor.

Deployment complexity and testability. One service means one deployment pipeline and one integration-test surface; per-model services mean each model change deploys independently (an advantage once models genuinely evolve on different schedules) but multiplies the number of moving parts that must each be tested, versioned, and rolled back independently.

Coupling. In a monolith, a bug or resource spike in one model's code can degrade another model's latency (shared process, shared memory, shared thread pool). Per-model services isolate that blast radius, at the cost of adding a network hop and a separate failure mode (the model service being unreachable) that did not exist before.

Observability. A monolith's per-model performance is harder to see cleanly, since metrics are aggregated across all models unless the code explicitly tags them. Per-model services get natural per-model dashboards and alerts for free, from the service boundary itself.

Resource utilization. This is the factor most tied to the given numbers, and the one that most often justifies decomposition. If the 1,000 requests/sec is split evenly across similarly-sized models, a monolith's shared resource pool is efficient: idle capacity for one model's low-traffic period is naturally used by another's high-traffic period. If instead demand and resource needs are skewed across models (one model dominates traffic, or one model needs a GPU while others run fine on CPU), decomposition lets you size and scale each model's replica count independently to match its own demand, instead of scaling the whole monolith (and paying for every model's replica) just to keep the hot model within its 50ms P95 latency target.

Framing as a per-model service boundary decision. The same choice can be framed as: does each model get its own per-model service boundary, and what does that do to scaling per model and to team ownership? Drawing that boundary means each model's replica count can scale per model, independently matching its own traffic instead of the aggregate, and typically means a single team (or a single engineer) becomes the clear owner of one model's service, establishing team ownership instead of shared, diffuse responsibility for one large multi-model deployable. That team-ownership clarity is a real benefit as team and model count grow, but it is not free early on, when a small team's context-switching cost across N services can exceed the coordination cost of one shared codebase.

Worked example

Assume, as an illustrative planning input, that of the 1,000 requests/sec total, one model accounts for 700 requests/sec (a hot, latency-critical recommendation model) and the remaining three models share 300 requests/sec combined.

In the monolith, meeting the 50ms P95 target for the hot model forces you to scale the whole service (all four models' code paths) to the hot model's replica count, so the three cold models are running far more replicas than their own traffic needs, purely as a side effect of sharing a deployable with the hot one. Splitting per model lets the hot model scale to meet its own 700 requests/sec target independently, while the three cold models run a much smaller, cheaper replica count sized to their combined 300 requests/sec, which is the resource-utilization case for decomposition once the traffic split across models is uneven enough to matter.

Trade-offs & pitfalls

The main pitfall is decomposing before there is evidence of the skew that justifies it: a small team that splits into per-model services on day one pays the coordination and operational cost immediately while the utilization benefit is theoretical, since a fresh proof of concept rarely has the traffic history to know which model will actually become the hot one. The better sequence is start monolithic, instrument per-model resource usage and latency from day one even inside the monolith, and split out a specific model only when its profile (traffic share, latency sensitivity, or hardware need such as a GPU) has diverged enough from its siblings to make shared deployment genuinely costly, not just theoretically suboptimal.