Serverless and Function-as-a-Service Architecture Questions

Direct answer

For a bursty, cost-sensitive workload that only sometimes needs a GPU, I would recommend (b) serverless containers with GPU support that scale to zero (drop to zero running instances, and zero cost, whenever there is no traffic), packaged as one container image that can also run CPU-only. Pure FaaS (a) cannot serve the GPU requests at all, and a managed GPU endpoint (c) only wins once utilization is high and steady enough that paying for an always-on instance is cheaper than the serverless premium. The deciding numbers are the fraction of hours the GPU is actually busy and how much first-request latency the product can tolerate.

What each option really is

• (a) Pure FaaS without GPU. For example AWS Lambda: CPU only, up to 10,240 MB of memory (CPU scales with memory; 1,769 MB is one vCPU, so the ceiling is roughly six), 15-minute timeout, container images up to 10 GB. Per-millisecond billing, true scale to zero, no GPU.
• (b) Serverless containers with GPU. For example Google Cloud Run with NVIDIA L4 GPUs (24 GB of GPU memory). You ship an OCI (Open Container Initiative, the standard image format that a Docker build also follows) image; the platform scales instances, including to zero. Cloud Run documents about 5 seconds for a GPU instance to start before your process can use the GPU, requires instance-based billing (you pay for the instance while it is up, not per request), and needs at least 4 CPUs and 16 GiB of memory with an L4.
• (c) Managed GPU endpoint. For example an Amazon SageMaker AI real-time endpoint on a GPU instance. Autoscaling, model monitoring and deployment tooling are built in. It can scale to zero only when the model is deployed as an inference component (SageMaker's unit of deployment that lets several models share one endpoint's instances and scale independently, instead of the older one-model-per-endpoint setup), and scaling out from zero takes several minutes during which invocations fail, so in practice it is run with at least one warm instance.

Comparison on the four requested axes

The cost crossover (the number that decides b versus c)

Let $E$ be the hourly price of an always-on GPU instance on the managed endpoint, $C$ the hourly price of a serverless GPU instance while it is up, and $u$ the fraction of the month at least one serverless instance is up (busy time plus the idle window before scale-down). $730$ is the average number of hours in a month (8,760 hours a year divided by 12).

$$\text{Always-on} = 730 \times E \qquad \text{Serverless} = 730 \times u \times C$$ $$\text{Serverless is cheaper when } u < \frac{E}{C}$$

Worked example with assumed prices (substitute your vendor's current rates): suppose the serverless instance costs 1.5 times the always-on rate, so $C = 1.5E$ and the break-even is $u = 1/1.5 = 66.7\%$. Traffic comes in 4 bursts a day of 30 minutes each, and instances linger 15 minutes before scaling down:

$$u = \frac{4 \times (30 + 15)\ \text{min}}{1440\ \text{min}} = \frac{180}{1440} = 12.5\%$$ $$\frac{\text{Serverless}}{\text{Always-on}} = u \times \frac{C}{E} = 0.125 \times 1.5 = 0.1875$$

So the serverless option costs about 19% of an always-on GPU, roughly an 81% saving, and it stays cheaper until the GPU is up two-thirds of the time. That is the bursty, cost-sensitive case the question describes.

The recommendation, and how to make it work

1. One image, two deployments. Build the model server (for example a standard inference server exposing HTTP) into one container. Deploy it CPU-only for the common requests and GPU-enabled for the heavy ones, and route by request characteristics (input size, batch size, model variant). If the CPU path meets its latency target, most traffic never touches a GPU.
2. Scale the GPU service to zero off-hours, keep one warm instance in business hours if the first request of a burst cannot wait about 5 seconds plus weight loading. That single warm instance is the knob that trades cost for tail latency (the slowest requests, for example p99, as opposed to the average); price it with the formula above.
3. Shrink cold starts at the source. Quantize the model (represent its weights with fewer bits, trading a little accuracy for a smaller, faster model) or distil it (train a smaller model to mimic a larger one), store weights next to the compute, and load them once per instance, not per request.
4. Keep the platform coupling thin. Standard container, standard HTTP contract, weights in an open format, so moving between (b) and (c) is a deployment change rather than a rewrite.

What would flip the choice

• To (c): the GPU instance is up (that is $u$ from above: busy time plus the idle linger before scale-down, not narrowly "doing inference") more than about $E/C$ of the time (steady high traffic), a need for guaranteed capacity or multi-model hosting on one instance, or the organization already runs its ML platform on that vendor.
• To (a): the model runs within the latency budget on CPU for all request types (small tabular or distilled text models). Then the GPU disappears from the design and FaaS is the cheapest and simplest.

Pitfalls

• Comparing per-hour prices without utilization. The serverless unit price is higher; the bill is lower only because idle hours cost nothing.
• Forgetting weight-loading time. Instance start is only part of a cold start; a multi-GB model must be read into GPU memory too.
• Assuming "serverless" means per-request billing. GPU serverless containers commonly bill for the whole instance lifetime, so concurrency per instance (how many requests one GPU serves at once) matters as much as request count.
• Regional GPU capacity. Scale-to-zero means you are asking for a GPU again at burst time; plan for the case where one is not immediately available.