Cloud Compute Options and Trade-offs Questions

Direct answer

Cold starts need two distinct metrics tracked separately, not blended into one number: the cold-start rate (what fraction of invocations pay the cold-start penalty) and the cold-start latency itself as its own distribution, separate from the warm-path latency distribution. Blending them into a single combined latency percentile hides exactly the tail behavior you're trying to catch, since a small cold-start rate can still produce a user-visible tail even when it barely moves an average.

Structured elaboration

Concrete metrics to define:

• cold_start_rate = cold_invocations / total_invocations, tracked as a rolling rate (e.g., per 5-minute window), not a single aggregate over a long period, since it can spike sharply after a deploy or a traffic drop that lets instances scale to zero.
• cold_start_latency_p50 / p95 / p99 (a percentile is the value below which that percentage of measurements fall; p95 latency is the number 95% of requests are faster than), computed only over invocations flagged as cold, kept as its own distribution rather than folded into the overall latency histogram.
• warm_latency_p50 / p95 / p99, the same percentiles computed only over warm invocations, so you have a clean baseline to compare against.
• Whether an invocation was cold is usually available directly from the platform's own execution metadata (an explicit cold-start flag or an init-duration field); if the platform doesn't expose one directly, a reasonable proxy is flagging any invocation whose total duration exceeds the warm p99 by a wide margin, though a direct platform signal is always preferable to a proxy.

Experiment design to measure this in production:

• Instrument every invocation to tag itself cold or warm at the point of execution (using the platform's own signal), and emit that tag alongside the latency measurement, rather than trying to infer it after the fact from aggregate statistics.
• Deliberately exercise both paths with synthetic canary traffic: a steady trickle of low-frequency canary invocations (spaced far enough apart to guarantee the platform has scaled the function down and the next canary invocation is cold) alongside the organic traffic that mostly hits warm instances, giving a controlled, comparable cold-start sample even during periods of naturally low organic traffic.
• Correlate cold-start rate against concurrency and deploy events specifically, since both a fresh deploy (which invalidates warm instances) and a period of low traffic (which lets the platform scale to zero) are the two most common causes of a cold-start spike, and knowing which one caused a given spike changes the fix.

Worked example

Say a rolling 24-hour window logs 250,000 invocations, of which 2,000 are flagged cold:

$$\text{cold\_start\_rate} = \frac{2{,}000}{250{,}000} = 0.008 = 0.8\%$$

If the warm p95 latency is 45ms and the cold p95 latency is 620ms, the expected added latency contributed by cold starts, averaged across all invocations (by linearity of expectation, which holds even though percentiles themselves don't blend this way), is:

$$0.008 \times (620\text{ms} - 45\text{ms}) = 4.6\text{ms}$$

That 4.6ms system-wide average looks negligible, and an SLO (service-level objective: a specific, measurable target for how the service should perform) written only against an overall blended p95 would likely never trip on it. But the 0.8% of users who actually hit a cold start are experiencing roughly 14 times the warm latency, a real, user-visible regression that the blended metric is specifically designed to hide. This is the case for the two-metric SLO structure below.

Turning this into SLOs with automated alerting: define two separate service-level objectives, not one, for example "cold-start rate stays below 1% over a rolling hour" and "cold p95 latency stays below 800ms," each with its own error-budget burn-rate alert (an error budget is the amount of SLO-violating behavior you're allowed before you've broken the target; a fast-burn alert fires if the rate spikes sharply after a deploy, a slow-burn alert if it creeps up gradually as traffic patterns shift). Alerting on the underlying rate and the cold-path latency separately catches a regression in either dimension that a single blended threshold would miss.

Trade-offs and pitfalls

• The single biggest pitfall is exactly the one this answer opened with: averaging cold and warm latency together into one number. It's mathematically valid for an expectation (as shown above) but not for a percentile, and teams that alert on a blended p95 routinely miss cold-start regressions until users complain.
• Canary-based synthetic cold-start measurement has to be tuned carefully: too frequent, and the canary itself keeps the function warm, defeating the purpose; too infrequent, and you don't get enough samples to trust the resulting percentile.
• Cold-start rate alone doesn't tell you the fix. A rate spike right after a deploy points at the deploy invalidating warm instances (a config or provisioned-concurrency issue); a rate spike with no recent deploy points at low traffic letting the platform scale down, which may call for a small provisioned-concurrency floor instead of a code change.

Direct answer

For serving machine-learning models, recommend containers on managed Kubernetes (a platform that runs, schedules, and scales containers across a fleet of machines) as the default, especially once the model is large, latency-sensitive, or serves meaningful sustained traffic, because a model loaded into memory is exactly the kind of state serverless's scale-to-zero, size-limited execution environment handles worst. Reserve serverless functions for small models, low or unpredictable traffic, or offline batch-style inference, where cold starts and package-size limits aren't binding constraints.

Comparison across the workload's dimensions

Cold starts. Serverless pays a cold start that's often worse for machine-learning serving than for a typical web function, because the cold path usually has to load model weights into memory in addition to bootstrapping the runtime. Containers avoid this entirely once a pod is warm and can keep the model resident continuously.

Concurrency limits. Serverless platforms cap per-account or per-function concurrent executions, which can throttle a burst of inference requests. A container deployment's concurrency is governed by however many replicas you run and scale to, which you control directly.

Model size limits. Serverless functions have deployment-package and memory limits that can make a large model, especially a large transformer, difficult or impossible to package directly. Containers have no comparable ceiling beyond the node's own memory or accelerator capacity.

Latency characteristics. Containers give consistent, predictable per-request latency once warm, with no cold-start tail. Serverless's latency distribution has a long tail driven by the cold-versus-warm mix, a poor fit for a firm latency target.

State management. A loaded model, and for some serving setups an in-memory cache of recent requests or intermediate results, is exactly the kind of long-lived in-process state serverless's stateless, ephemeral execution model works against. Containers hold the model resident across requests naturally.

Operational complexity. Serverless is operationally simpler for small, infrequent workloads, since there's no cluster to run. Containers on Kubernetes require the fleet-management, autoscaling, and monitoring investment, but that investment pays off once traffic and model size grow.

Cost at low traffic. Serverless wins clearly at low, spiky traffic, since billing is per invocation and nothing is charged when idle. A container fleet, even autoscaled to a minimum of one or two replicas for availability, pays for that floor continuously.

Cost at high traffic. Containers usually win at sustained high traffic, since per-invocation billing adds up faster than amortized instance-hours once utilization is consistently high, and a large model paying its load-time cost on every serverless cold start is pure waste at volume.

An easier worked example

An internally facing analytics model, used by a handful of employees with traffic that's unpredictable but modest and no public service-level target, is a good fit for a small Kubernetes deployment with horizontal pod autoscaling (a Kubernetes controller that adds or removes replica pods based on observed load), scaled down to a single replica at idle, rather than serverless. That's specifically because the model's load time makes serverless's cold start expensive relative to how rarely a genuinely new instance would even be needed. A minimum of one warm replica avoids ever paying the model-load cold-start tax, at a small, predictable standing cost.

Recommendations

Small batch inference. Batch inference, processing a queued set of records with no single user waiting synchronously, tolerates a cold start well, since the consumer is a queue, not a person. A serverless function triggered per batch, or per chunk of a batch, is a reasonable, low-operational-cost choice as long as the model fits within the platform's package and memory limits.

Large transformer models. Recommend containers on managed Kubernetes with accelerator-backed nodes, kept warm continuously with no scale-to-zero on serving replicas, since reloading a large transformer's weights on every cold start is both slow and expensive. Use request batching within the serving layer to improve accelerator utilization per request, and size replica count to the sustained request rate, autoscaling on a request-based metric, such as queue depth or in-flight request count, rather than processor utilization, since accelerator-bound inference often doesn't show that as the bottleneck signal.

A harder follow-up: the recommender-system case

A production recommender system combines a latency-tail requirement, since a user-facing response has to come back fast, with sustained high throughput, since traffic is constant rather than spiky. That combination is exactly the profile that most favors containers over serverless: the tail-latency requirement rules out cold starts, and the sustained-throughput profile removes serverless's main cost advantage, since paying only for idle-free capacity stops mattering once the fleet is never idle.

Trade-offs and pitfalls

A common pitfall is choosing serverless for model serving purely because the rest of the stack is serverless, without checking whether this specific model's load time and traffic pattern actually fit. A large model on a platform sized for lightweight functions will either fail to package or cold-start badly enough to violate any real latency expectation. A second pitfall is running a container-based serving fleet with zero warm replicas at idle to save cost, which reintroduces a cold-start-like penalty, since the first request after scaling to zero has to wait for a pod to schedule and the model to load, without the benefit of serverless's per-invocation billing simplicity.

Direct answer

A stateless serverless function still needs somewhere durable to put state, and the real options fall into a handful of categories: a managed database for structured, queryable state, a cache for low-latency ephemeral lookups, a queue for state that represents work still to be done, object storage for large blobs, and a workflow or state-machine service for state that represents progress through a multi-step process. The trade-off across all of them is the same axis: the closer the store sits to raw memory (a cache) the lower the latency and the weaker the durability and query power, and the further it sits toward a full database or object store the stronger the durability and query power at the cost of latency.

Structured elaboration

Worked example

An order-processing function receives a webhook, needs to look up the customer's current account balance (fast, structured, must be correct), enqueue a fulfillment task for a downstream worker (asynchronous, decoupled), and store the original webhook payload for audit purposes (large-ish, rarely read, never queried). One function ends up touching three different state patterns in a single invocation: a database read (through a connection-pooling proxy, since this function can be invoked at high concurrency during a sale) for the balance check, a queue write for the fulfillment task, and an object-storage write for the audit payload. Reaching for a single store to do all three (say, cramming the audit payload into the same relational database as a large text column) would work initially but degrade the database's query performance and storage cost for data that never needed to be there in the first place.

Trade-offs and pitfalls

• The pitfall named above (connection exhaustion under serverless's concurrency spikes) is specific enough, and common enough, that it deserves calling out on its own: a database that was never a bottleneck under a fixed-size server fleet can become one purely because serverless can multiply concurrent connections far faster than a traditional connection pool was ever sized for.
• A common wrong turn is picking one pattern (usually "just use a database for everything") out of familiarity, then discovering that a workflow's multi-step progress or a large blob doesn't actually fit well into a row in a table, and retrofitting a second store under time pressure instead of matching the pattern to the state's shape from the start.
• The deeper trade-off across the whole table: durability and latency pull in opposite directions, and a design that tries to get both from a single store (a strongly-durable database used as if it were a low-latency cache) usually gets neither well.

Direct answer

A cold start is the extra latency a serverless function pays on the first invocation of a brand-new execution environment: the platform has to provision compute, initialize the language runtime, load your code and its dependencies, and run any top-level initialization before it can execute your handler at all, on top of the handler's own execution time. A warm invocation reuses an already-initialized environment and skips all of that.

Causes, effect, and mitigation

Causes. Environment provisioning: allocating compute for a brand-new instance. Runtime bootstrap: starting the language runtime, which is why a lighter-weight runtime tends to cold-start faster than a heavier one, and why a larger deployment package with more dependencies makes bootstrap slower. Your own top-level initialization code: creating a database connection pool or loading a large configuration or model into memory, which reruns on every cold start unless it's deliberately made reusable.

Effect on latency. Cold starts add a one-time tax to whichever request happens to land on a new instance. This hurts most on low-traffic functions, where instances scale down between requests and every burst starts cold, and on sudden traffic spikes, where concurrency has to expand faster than warm instances exist to absorb it, so a cluster of requests can all pay the cold-start tax at once.

Mitigation techniques. Keeping a baseline of pre-warmed or provisioned instances so some fraction of traffic never hits a cold path. Shrinking the deployment package and trimming unused dependencies so bootstrap does less work. Choosing a lighter-weight runtime for latency-critical functions. Deferring expensive initialization until it's actually needed, so a cold start's fixed cost is smaller. At the architecture level, routing latency-critical paths to always-on containers instead of scale-to-zero functions entirely.

Cost and complexity trade-offs of those mitigations. Pre-warming or provisioned capacity converts scale-to-zero's near-zero idle cost into a standing cost proportional to the warm baseline you pay for, cutting directly into serverless's main cost advantage. It also adds an operational knob (how much to keep warm, and re-tuning it as traffic shifts) that a pure "just deploy the function" model doesn't have. Trimming dependencies and deferring initialization are close to free in dollar terms but add engineering discipline, since the benefit can silently erode as new dependencies get added over time without anyone revisiting the bundle size. Moving latency-critical paths to containers eliminates cold starts there entirely, at the cost of giving up serverless's operational simplicity for that path.

How this differs across providers, qualitatively. Providers differ in the mechanisms they expose (a dedicated provisioned-concurrency feature versus tunable idle-timeout behavior), in how heavily the choice of language runtime affects bootstrap time on that specific platform, and in whether attaching a function to a private network adds its own cold-start latency. Because the specifics and their exact magnitude are provider- and release-specific, validate actual behavior against the target platform's current documentation and your own load test, rather than assuming one provider's characteristics carry over to another.

Trade-offs and pitfalls

The most common measurement mistake is looking only at average latency, which hides cold starts entirely, since they show up in tail latency. Always look at percentile latency and the ratio of cold to warm invocations, not the mean. A second pitfall is adding provisioned concurrency and never sizing it back down as traffic changes, quietly paying container-like standing cost while still carrying serverless's other constraints, such as execution-time limits and forced statelessness. Measure your own cold-to-warm invocation ratio in production before investing in mitigation at all, since a function invoked constantly may rarely cold-start regardless of platform.

Direct answer

For a steady 100 requests-per-second baseline with a hard 95th-percentile latency ceiling of 50 milliseconds, recommend containers on managed Kubernetes (a platform that runs, schedules, and scales containers across a fleet of machines) with graphics-processing-unit-backed nodes. It gives warm, predictable-latency capacity sized to a known baseline, autoscaling for traffic variance above it, and far less operational burden than running graphics-processing-unit virtual machines directly, while avoiding the cold-start risk that serverless graphics-processing-unit options carry against a tight latency target.

Comparing the three options

Serverless graphics-processing-unit options. Cold starts on accelerator-backed serverless tend to be heavier than on processor-only serverless, since they often add driver or execution-context initialization on top of model loading, which is a serious risk against a 50 millisecond 95th-percentile ceiling unless provisioned or pre-warmed concurrency is used continuously at the full 100 requests-per-second baseline. At that point, the platform is being paid to stay always warm, and most of serverless's cost advantage is gone. This option only makes sense if traffic is genuinely spiky, and a steady 100 requests-per-second baseline is not.

Containers on managed Kubernetes with graphics-processing-unit nodes. Warm replicas eliminate cold-start risk entirely for steady baseline traffic. The managed control plane removes cluster-operations burden while the team still controls replica count, autoscaling policy, and node-pool sizing. This is the best fit for a known, steady baseline with a firm tail-latency requirement.

Virtual machines with graphics processing units. Gives full control over the accelerator driver and runtime stack, which can be the right call if the team needs operating-system-level customization the managed container platform doesn't expose. Otherwise, this option takes on all of the node, operating-system, and driver patching and scaling automation directly, for no latency advantage over the container option at this traffic level, since both are running warm processes on the same class of hardware. The extra operational burden isn't bought back by anything this workload specifically needs.

Worked capacity check

At 100 requests per second with a sub-50-millisecond 95th percentile, expected concurrent in-flight requests, by Little's Law intuition (concurrency is approximately arrival rate multiplied by time in system), works out to:

$$100 \times \frac{50\text{ms}}{1000\text{ms}} = 5 \text{ concurrent requests, at minimum}$$

If a single accelerator-backed replica can serve, say, 10 concurrent requests within that latency budget, a figure to measure by load-testing the actual model rather than assume, the bare-minimum fleet needs less than one full replica's worth of concurrency. That's exactly why bare-minimum sizing is the wrong target: real traffic isn't perfectly smooth, and provisioning only enough capacity for the textbook average leaves zero headroom for burstiness, an occasional slow request, or a single replica failing. Provision meaningfully more, spread across at least two or three replicas for failover, so ordinary traffic variance or one replica's failure doesn't blow the 95th-percentile target.

Trade-offs and pitfalls

The main pitfall is picking serverless graphics-processing-unit options because serverless is assumed to be simpler, without checking whether continuous provisioned concurrency at the actual baseline load erases both the cost and the simplicity advantage; at a steady, known baseline, that check almost always favors containers. The second pitfall is sizing the container fleet to the bare capacity minimum with no headroom, which looks correct on average load but blows the tail-latency target the moment traffic or per-request latency has any real-world variance. Always load-test the chosen replica count against the full expected traffic distribution, not the mean.