Cloud Compute Options and Trade-offs Questions

Direct answer

Spot (or preemptible) instances are the same underlying hardware as on-demand capacity, sold at a steep discount because the provider can reclaim it with little warning when it needs the capacity back. That makes them safe for exactly one category of workload: work where losing an individual instance mid-task costs you nothing more than retrying that unit of work elsewhere. The architecture you build around them isn't really about spot instances at all, it's about making every unit of work idempotent (safe to run more than once with the same correct end result), checkpointed or cheaply restartable, and never the sole copy of anything.

Structured elaboration

• Stateless web tier: run spot instances as part of a larger autoscaling group that also includes a baseline of on-demand capacity. When a spot instance is reclaimed, the load balancer's health check simply routes around it and the autoscaler replaces it, exactly the same failure mode as a normal instance crashing, provided the tier holds no local, unreplicated state.
• Batch processing: design each job as a small, independently retryable unit rather than one long monolithic run. If an instance is reclaimed mid-unit, the orchestrator (a job queue or a workflow scheduler) simply re-dispatches that unit to another instance. The smaller the unit, the less work is lost per interruption.
• Other compute-intensive jobs (a rendering farm, a large parallel simulation): checkpoint progress periodically to durable storage outside the instance, so a reclaimed instance loses at most the work since the last checkpoint, not the whole job.

Designing for graceful interruption, concretely:

• Listen for the interruption notice. Providers give some advance warning before reclaiming an instance, but the length of that window varies a lot by provider and isn't safe to assume: AWS EC2 Spot gives a fixed two-minute notice, while Google Cloud's and Azure's default Spot/preemptible notice is far shorter, as little as zero to thirty seconds unless you explicitly opt into a longer notice window on the platforms that offer one. Design the shutdown handler for the shortest window your actual provider guarantees, not for the longest one any provider offers, since assuming a two-minute AWS-style window on a platform that only gives thirty seconds will leave in-flight work unfinished when the instance is actually reclaimed. A small agent on the instance watches for that signal and triggers an orderly shutdown: stop accepting new work, finish or checkpoint what's in flight, deregister from the load balancer, then exit.
• Diversify instance types and availability zones. Spot capacity availability and reclaim risk vary independently across instance types and zones; spreading a fleet across several types and zones means a capacity crunch in one pool doesn't take out the whole fleet simultaneously.
• Keep a floor of on-demand (or reserved) capacity for anything that must never fully disappear. A pure spot fleet can, in the worst case, be reclaimed almost entirely at once if the provider needs that capacity broadly; a baseline of guaranteed capacity underneath the spot layer keeps the service alive even in that scenario.
• Make retries idempotent. If a unit of work is re-dispatched after an interruption, running it twice (once partially, once fully) must produce the same correct result as running it once, otherwise interruption doesn't just cost time, it corrupts output.

Worked example

A video-rendering batch pipeline processes 10,000 independent frame-ranges, each taking about 3 minutes on one spot instance:

• Each frame-range is submitted as a separate job to a queue, not one 500-hour job for the whole batch.
• A worker pulls one job, renders it, and writes the output to durable object storage under a deterministic, content-addressed key (say, keyed by the input range and a version hash) before acknowledging the job as complete.
• If the instance is reclaimed mid-render, the job's visibility timeout expires unacknowledged, the queue makes it available again, and a different worker picks it up and re-renders it. Because the output key is deterministic, re-rendering the same range twice (once by the interrupted worker, if it partially wrote output before being killed, and once by the replacement worker) either overwrites the same key harmlessly or is caught by a "does this key already exist" check before starting, so no partial or duplicate output survives.
• The blast radius of losing any one instance is bounded to at most one 3-minute unit of work, not the whole 500-hour job, which is the entire point of choosing a small unit size.

Trade-offs and pitfalls

• Where spot goes wrong: putting a stateful primary (a database, a message broker holding unacknowledged state only it has) on spot capacity. Losing that instance doesn't cost you a retry, it costs you the state itself, unless it's independently replicated elsewhere, which defeats the cost savings if you now need extra replicas just to make spot safe.
• A subtler pitfall: sizing the on-demand floor too thin. Teams sometimes run "mostly spot with a token on-demand instance," which technically has a floor but not enough capacity to serve real traffic if a large fraction of spot capacity is reclaimed at once during a genuine capacity crunch.
• The real trade-off being made: significant cost savings in exchange for engineering effort spent making work interruptible, checkpointed, and idempotent. That engineering cost is often paid once and amortized across every future workload you run on spot, so it's worth investing in the primitives (a resilient job queue, a checkpointing convention, idempotent writes) rather than solving it ad hoc per workload.

Direct answer

Think of these four as one ladder of abstraction, not four unrelated choices: a monolith on a virtual machine (VM) gives full control at the cost of doing everything yourself; containers on a self-managed orchestrator trade some of that control for packaging and density; a managed platform-as-a-service (PaaS) or managed container service trades more control for a shrinking operations bill; and serverless functions trade almost all control for near-zero idle cost and infrastructure-as-a-service (IaaS)-free operations, at the price of execution-time limits and cold starts. For a three-tier web application, the right architecture is rarely "pick one for everything," it's picking per component based on that component's latency, statefulness, and burstiness, then adjusting how far up the ladder you go based on how much operational capacity the team actually has.

Structured elaboration

Security boundaries and observability are decision inputs here too, not afterthoughts: a VM gives you full audit visibility (you own every log, every syscall) but also full liability for missing something; serverless gives you the least attack surface to secure (no host to patch) but the least visibility into what's happening inside a cold-started instance during a live incident, since you can't attach a debugger to it. That trade should be named explicitly alongside cost and control, because a team that picks serverless for the API tier without also investing in structured tracing will find incidents much harder to diagnose than the "just SSH in" experience a VM gives them.

Worked example

Picture this three-tier app starting life as a single monolith running on one VM: a web/API layer, an extract-transform-load (ETL) style nightly reporting job, and a database.

• Steady web tier: traffic is predictable and mostly flat. Splitting this off the monolith into containers on a managed container service is the natural first move: you get consistent packaging (the same image runs in dev, staging, and production, removing "works on my machine" drift) and horizontal scaling without owning a control plane. A concrete before/after: on the VM, a deploy meant provisioning a new instance, waiting for it to boot, then cutting traffic over, commonly several minutes of risk window. On the managed container service, a deploy is a rolling replacement of already-warm containers, typically well under a minute, because the container process starts far faster than a whole OS.
• Bursty API (say, a webhook receiver with unpredictable spiky callers): this is where serverless earns its keep. Per-invocation billing means near-zero cost during the long idle stretches between bursts, which a fleet of always-on containers would be paying for regardless.
• Batch job: the nightly reporting job is the piece most tempted to stay a cron job on a VM, but that's usually the wrong default once you're already moving the rest of the stack. If it finishes within the serverless platform's execution-time ceiling, move it to a scheduled function or a managed batch service; if it's a genuine extract-transform-load (ETL) or feature-computation pipeline that can run long, a container-based batch runner (a scheduled job on the same managed container service, not a hand-run cron on a VM) gets you the same "someone else patches the OS" benefit without hitting a serverless time limit.

The framework holds under a messier mix too: a platform running a long-lived stateful workload (a WebSocket-backed session service), a nightly batch job, and a spiky event-driven ingestion path simultaneously doesn't need one answer, it needs the stateful piece on containers or a VM (state and connection affinity don't fit serverless's stateless-by-design model), the batch piece on a container-based scheduled job, and the event-driven piece on serverless, all coexisting in the same platform.

A separate long-running orchestration pattern worth naming alongside the batch example: for workflows with occasional multi-hour steps or a human-approval gate in the middle (not just a single long-running job), a serverless orchestration service (a durable-workflow engine that coordinates a sequence of short function calls, each within the platform's time limit, and can pause indefinitely waiting on a human click) solves the same "beyond serverless time limits" problem as a container batch runner, without giving up the pay-per-use model, at the cost of learning that orchestration service's own programming model.

Trade-offs and pitfalls

• How the recommendation shifts by team size: a small operations team should default further up the ladder than the "purest" architectural fit for every component would suggest: a managed PaaS or managed container service for everything except the truly bursty and truly long-running pieces, because every extra self-managed layer (a self-run Kubernetes control plane, a hand-rolled autoscaling group) is a 3am page waiting to happen for a team with no dedicated platform engineer. A large platform team can afford to run its own container orchestration and capture the cost and control benefits, because it has the staff to own upgrades, node patching, and control-plane availability as a first-class responsibility, not a side project.
• The most common wrong turn is picking one model for the whole application because it feels simpler to reason about; a three-tier app almost never has three tiers with the same latency, statefulness, and burstiness profile, so a single blanket choice usually either overpays for capacity the batch job doesn't need or under-serves the API tier's latency needs.
• Migrating from a monolith is itself a risk to manage explicitly: extracting the web tier first (the least stateful, most horizontally-scalable piece) and leaving the database and any genuinely stateful component for last is a safer sequencing than trying to containerize everything in one pass.

Direct answer

Host each user's session as a dedicated, isolated compute unit, either a container on an orchestrator or a lightweight virtual machine for the strongest isolation, created when the session starts and torn down or suspended when it goes idle, rather than a shared multi-tenant process. Long-lived per-user connections and predictable per-user resource limits are exactly what serverless functions cannot provide, since they cap execution duration and don't hold a warm, stateful process open for the tens of minutes to hours a coding session runs. The right compute primitive here is one dedicated unit per session, on an orchestrator designed for many small, short-lived, per-tenant workloads.

Architecture

flowchart LR
  U[User browser] --> GW[Gateway or router]
  GW --> SCH[Session scheduler]
  SCH --> C1[Per-user container or microVM]
  C1 --> VOL[(Per-session persistent volume)]
  SCH --> AS[Autoscaler: node pool]
  SCH --> REAPER[Idle-session reaper]
  REAPER --> SNAP[(Workspace snapshot store)]
  C1 --> METER[Usage metering per user]

Compute choice. Containers give fast start time and good cost efficiency when isolation between users is enforced at the operating-system level, using process namespaces and resource limits (Linux mechanisms that separate and cap what each process can see and use). If the code being run is genuinely untrusted, add a stronger sandboxing boundary, such as a lightweight virtual machine (a microVM, which boots close to container speed but gives kernel-level isolation). The right choice tracks how trusted the running code is: internal tooling can use lighter container isolation, while a public product that runs strangers' code needs the stronger microVM boundary.

Session persistence. Each session gets its own persistent volume, or a workspace snapshot saved to object storage, mounted into its container or microVM, so a session that's paused and resumed later, even on a different host, reattaches to the same files. Persist workspace state on every meaningful checkpoint, such as a periodic snapshot or an idle transition, not only on a clean shutdown, since a crashed host must never silently lose a user's work.

Autoscaling strategy for sessions. Scale the underlying node pool on aggregate concurrent-session count, not compute utilization alone, since an idle-but-open session still holds a slot. Bin-pack sessions onto nodes by their per-session resource reservation, and keep a small pool of pre-warmed, ready nodes so a new session doesn't wait on a full node boot.

Idle-session cleanup. Track per-session real activity, such as keystrokes or terminal input and output, not just an open network connection, since a connection can sit open with no real activity. On an idle timeout, snapshot the workspace, then suspend or destroy the compute unit and release its resources, while keeping the persisted snapshot available for a fast resume. Make the idle threshold and any "about to be reclaimed" warning visible to the user, so cleanup is never a surprise data-loss event.

Security isolation. Run each session's code under an isolation boundary matched to trust level, enforce per-session network policy so no session can reach another session's compute or the control plane (the orchestrator's own management and scheduling systems, separate from user workloads) by default, and cap per-session resource limits, such as compute, memory, disk, and process count, so one runaway or malicious session cannot degrade the node for everyone else.

Cost allocation per user. Meter actual resource consumption per session (compute-seconds while active, storage for the persisted workspace, and any network egress, meaning data sent out to the internet or another region, which most providers bill per byte) tagged by user or account, aggregate it into a per-user cost record, and clean up idle sessions aggressively, since the biggest silent cost driver in this architecture is sessions left open and billed while genuinely idle.

Worked example

Each session reserves 1 unit of compute and 2 gigabytes of memory; a node offers 8 units of compute and 16 gigabytes of memory, so it fits 8 sessions on either dimension, meaning the node is balanced for this session shape with no wasted headroom on either axis. If the observed idle-session ratio at any given time is 40 percent, an idle reaper reclaiming those slots after, say, 20 minutes of no keystroke activity lets the same node pool serve roughly $\frac{1}{1-0.4} \approx 1.67$ times as many registered users as concurrently active ones, so capacity planning should be based on concurrent active sessions, not total registered users.

Trade-offs and pitfalls

The most costly pitfall is sizing the node pool for average concurrency and getting caught by a correlated burst, such as a course start time or a scheduled workshop, where hundreds of sessions start within the same minute; keep a pre-warmed buffer sized to the worst observed burst, not the average. The second is treating an open connection as proof of activity, which under-detects idle sessions and wastes cost, or setting the timeout so aggressively that it kills a session mid-thought; use real activity signals, not connection liveness, to decide when a session is truly idle.

Direct answer

Recommend containers on managed Kubernetes for the compute platform. A hundred requests per second with sub-50-millisecond per-request compute is comfortably within a small, steady container fleet's capability with no cold-start risk, and the seven-year log-retention requirement is a storage and compliance decision that's largely independent of which compute model is chosen, so it shouldn't itself rule serverless in or out. It's the combination of steady, moderate traffic and a tight latency figure that favors always-warm containers here.

Compute platform choice

At 100 requests per second with a 50-millisecond compute budget, all three options, virtual machines, containers on managed Kubernetes, and serverless, can technically hit the latency target once warm. The discriminator is that serverless risks an occasional cold-start outlier against a tight budget for no offsetting benefit, since traffic here is steady rather than spiky, which is serverless's main advantage. A self-managed virtual-machine fleet takes on manual scaling and patching for no benefit over managed Kubernetes at this traffic level. Containers on managed Kubernetes, with a small autoscaled replica count sized with headroom above the 100 requests-per-second baseline, is the balanced choice: predictable latency, moderate operational burden, and room to grow.

Logging and storage for retention and compliance

Separate "hot" operational logs, recent logs used for debugging and monitoring, kept in a fast-query store for a much shorter window such as weeks to a few months, from the long-term compliance retention copy, shipped to durable, low-cost object storage with a retention policy that enforces the seven-year minimum and prevents early deletion. A compliance retention requirement usually implies genuine immutability, not just logs happening to still be around. Make the retention policy, access controls, and any required audit trail on who accessed the logs explicit and tested, since a compliance requirement is about provable retention and access control, not accidental survival.

Minimal monitoring stack

Track request rate, error rate, and latency percentiles, especially the 95th and 99th, against the 50-millisecond compute budget, since an average latency figure would hide exactly the tail risk this workload cares about. Track container and pod health and replica count as the platform-level signal. And add a specific alert on the health of the log-shipping and retention pipeline itself, since that pipeline failing silently, logs simply stopping short of the long-term store, is a distinct and easy-to-miss failure mode separate from the application being healthy.

Trade-offs considered

Choosing containers over serverless gives up serverless's near-zero idle cost, but this workload isn't idle-heavy, since 100 requests per second is steady rather than spiky, so that trade is a clear win for latency predictability. Choosing managed over self-managed Kubernetes gives up some control-plane customization (fine-grained control over the cluster's own management components, such as the API server and scheduler) in exchange for not owning control-plane operations, the right trade for a team whose actual requirement here is compliance and latency, not control-plane customization. Separating hot and long-term log storage adds a second storage system and a pipeline between them, versus dumping everything into one store for seven years, but a single store sized and priced for seven-year retention of every operational log line would be needlessly expensive and slow to query for day-to-day debugging.

Trade-offs and pitfalls

A common pitfall is treating the seven-year retention requirement as a compute-platform decision, when it's really a storage and pipeline decision; don't let it push the compute choice away from what actually fits the latency and traffic profile. A second is measuring against average latency instead of the tail, when the stated requirement implies every request needs to hit the compute budget, not just most of them; validate the 50-millisecond figure against the percentile that matters and load-test the tail specifically, not just the mean.

Direct answer

Lift-and-shift to containers is the lower-risk, faster path, since it repackages the same monolith's process into a container image without changing its architecture. Rearchitecting to serverless microservices is a much larger, slower undertaking that only pays off if the monolith's actual pain points, such as deployment coupling or the inability to scale one hot code path independently, are the kind serverless microservices actually solve. Default to lift-and-shift first, and treat a full serverless rearchitecture as an incremental decomposition done afterward, not a one-shot replacement of the original monolith.

Two migration plans

Lift-and-shift to containers.
Phases: containerize the monolith as-is, with no code changes, using a container definition that wraps the existing runtime and dependencies; validate parity in a staging environment against the current deployment, using the same configuration and data access; run both in parallel behind a router, shifting a small percentage of traffic to the container path; ramp to full traffic on containers, then decommission the old servers.
Timeline: typically the shortest of the two, since no application logic changes; mostly bounded by how much undocumented, environment-specific configuration has accumulated on the old servers over time, since a container has to reproduce that explicitly.
Key risks: hidden environment dependencies, such as files or state baked onto local disk that the monolith silently relies on and that a fresh container doesn't have on first boot; a server that's been running unmodified for a very long time can be masking bugs that containerizing suddenly exposes.
Testing strategy: parity testing, comparing container responses against the existing deployment's responses for the same inputs, plus a load test at expected peak before cutting traffic over, since containers may be scheduled differently, packed onto shared hosts, than a dedicated server was.

Rearchitect to serverless microservices.
Phases: identify seams along real bounded contexts in the business logic, not arbitrary file boundaries; strangle the monolith incrementally, extracting one bounded context behind an application programming interface and routing selected traffic to the new service while the monolith still serves the rest; repeat extraction context by context; retire monolith code paths once each is fully replaced.
Timeline: much longer, proportional to the number of bounded contexts and the amount of shared data coupling between them; realistically measured in multiple quarters or longer for a nontrivial monolith, not weeks.
Key risks: distributed-system failure modes that didn't exist in a single process, such as partial failure between services, data consistency across service boundaries, and added network latency per call; underestimating shared-database coupling is the most common trap, since modules that look separable in code are rarely separable in data without real migration work.
Testing strategy: contract testing between an extracted service and everything that calls it, plus running the old and new code paths side by side with shadow traffic or a feature-flag split, diffing outputs before fully cutting over each extracted piece one bounded context at a time.

Deciding which path to pursue

Weigh how much of the current pain is deployment or packaging friction, which favors lift-and-shift, against a genuine need for independent per-component scaling or team autonomy, which favors decomposition. Weigh the cost of the monolith's current failure mode: if a slow release cycle is costing real velocity now, lift-and-shift buys relief in weeks, while a full rearchitecture buys nothing until each piece ships. Consider team size and organizational fit, since a single small team gets little organizational benefit from microservices regardless of the technical case. And check actual bounded-context clarity in the codebase: if you can't name today where one module ends and another begins, that's a signal the codebase isn't ready for decomposition, independent of any traffic argument.

Trade-offs and pitfalls

The most common pitfall is choosing the serverless microservices rearchitecture because it's the more fashionable answer, rather than because the monolith's actual constraint is one serverless solves; a monolith with steady, predictable traffic and no scaling pain gains nothing from microservices except added operational complexity. The second is treating the container lift-and-shift as a stepping stone that hasn't really finished the job, when for teams whose real problem was deployment friction rather than architecture, it's a valid, complete answer on its own. Run lift-and-shift first regardless of the eventual target, since it de-risks the environment and dependency unknowns cheaply and buys time to do the decomposition analysis properly, rather than under incident pressure.