Cloud Compute Options and Trade-offs Questions

Direct answer

Video transcoding at over 15 minutes needs to move off the request/response serverless model entirely and onto a queue-driven worker architecture: a durable message queue holds one job per video (or per chunk of a video), and a pool of long-running workers, on containers or virtual machines, not functions, pulls jobs and processes them for as long as they take. The rest of the platform can stay serverless; only the small set of genuinely long-running steps needs to escape the time limit, and the two coexist fine as long as the handoff between them is a durable queue, not a direct synchronous call.

Structured elaboration

• Job intake: a lightweight serverless function still handles the upload trigger and enqueues a transcoding job, since that part genuinely is a fast, stateless operation well suited to serverless.
• Worker pool: a pool of containers (on a managed container service, sized to autoscale on queue depth rather than request rate) or a managed batch service consumes jobs from the queue with no execution-time ceiling, running each transcode to completion.
• A second pattern for the same underlying problem: for workflows that aren't one long computation but a sequence of steps that together exceed serverless limits, some with occasional multi-hour waits or even a human-approval gate in the middle (a manual review before publishing a transcoded video, for example), a durable-workflow orchestration service is often a better fit than a single long-running worker: it coordinates a chain of short function calls, each individually within the serverless time limit, and can pause indefinitely between them (including waiting on a human action) without holding a worker process idle the whole time. Video transcoding and a human-approval publishing workflow are two different shapes of the same "beyond serverless time limits" problem, and it's worth being able to name both.

Handling retries and idempotency so a failed job doesn't corrupt or duplicate output:

• Idempotency key: each job carries a unique, deterministic identifier (derived from the input video and a version of the transcoding config, not a random ID generated fresh on each attempt), so retrying the same logical job twice is recognizably "the same job," not two different jobs.
• Visibility timeout and safe re-delivery: the queue holds a job as invisible to other workers while one worker is processing it; if that worker crashes or times out without acknowledging, the job becomes visible again and a different worker picks it up. This is what makes "the worker died mid-transcode" a recoverable event rather than a lost job.
• Atomic output publication: a worker writes its output to a temporary, uniquely-named location first, and only makes it visible under the job's final, deterministic key once the write is fully complete (an atomic rename or a conditional "put if not already present" write). This guarantees that even if two workers process the same job concurrently (a genuine possibility if a visibility timeout fires just before a slow-but-still-alive worker finishes), only one complete, correct output ever appears under the final key, never a partial or duplicated one.
• Dead-letter queue: after a bounded number of retries, a job that keeps failing moves to a separate dead-letter queue for manual inspection, rather than retrying forever and masking a genuine, unrecoverable input problem (a corrupt source file, for instance).

Worked example

A video finishes uploading and triggers a serverless function that computes a deterministic job ID (a hash of the video's storage key plus the transcoding profile), and enqueues a job under that ID. A worker picks it up, holds it invisible to other workers via the queue's visibility timeout, and begins a 40-minute transcode. At minute 25, the worker's underlying instance is reclaimed (say, it was running on spot capacity to save cost). The visibility timeout expires without an acknowledgment, the job becomes visible again, and a second worker picks it up and restarts the transcode from the beginning. When it finishes, it writes the output to a temporary path, then atomically publishes it under the deterministic final key. If the first worker had somehow still been alive and also finished around the same time, its own publish would either be a harmless overwrite of the identical content (same input, same deterministic output) or rejected by a "put if not already present" check, so the two never race into a mixed or truncated file.

Trade-offs and pitfalls

• The worker pool trades serverless's near-zero idle cost for the ability to run arbitrarily long; that's the right trade specifically because transcoding jobs are the minority of the platform's workload, not the majority, so most of the platform still gets to keep serverless's cost profile.
• A common wrong turn is making the job ID a randomly generated identifier assigned fresh on each retry attempt; that defeats idempotency entirely, since the system can no longer recognize a retried job as "the same work," and duplicate output becomes likely rather than merely possible.
• Autoscaling the worker pool on queue depth (not on CPU utilization alone) matters here: a backlog of queued but not-yet-started jobs is the leading indicator that more workers are needed, and waiting for CPU utilization to climb means the backlog has already grown before the system reacts.

Direct answer

The four instance families trade the same total capacity across compute, memory, and disk input/output differently. General-purpose instances give a balanced ratio of the three and are the safe default when you don't yet know your bottleneck. Compute-optimized instances shift the ratio toward more processing power per unit of memory, for workloads bound by raw computation. Memory-optimized instances shift it toward more memory per processing core, for workloads that need to hold large datasets or many concurrent sessions in memory. Storage-optimized instances prioritize fast, high-throughput local disk access over compute or memory. Picking the wrong family means paying for capacity you don't need in your non-bottleneck resources while still being constrained by the one you actually need.

Families and example workloads

What to check before recommending an instance type

Profile the actual bottleneck under representative load, not the resource that feels compute-heavy or memory-heavy by intuition, since the two are often correlated and easy to misattribute. Check the working-set size, meaning how much data must stay resident in memory to avoid thrashing to disk or network, against candidate memory sizes. Check whether the workload is latency-sensitive per request, which favors more, smaller instances to keep the failure blast radius small (the blast radius is how much of the system is affected when a single instance fails), or throughput-oriented in aggregate, which favors fewer, larger instances to reduce per-instance overhead. Compare cost per unit of the bottleneck resource across families, since a cheap instance type is only cheap relative to the resource you actually need more of. Leave headroom for the resource you're not optimizing for, since sizing exactly to one bottleneck with zero slack elsewhere just creates a new bottleneck under any load variance.

Vertical versus horizontal scaling

Prefer vertical scaling, moving to a bigger instance, when the workload cannot be split across processes or nodes without a rewrite, such as a legacy application with shared in-process state or a single-writer database, when operational simplicity matters more than resilience, or when coordination overhead between many small instances would exceed the resources actually available on one right-sized instance. Prefer horizontal scaling, adding more instances, when the workload is already stateless or can be made so, when fault tolerance beyond any single instance is required, since any one instance can fail regardless of size, or when load is elastic and paying for capacity that tracks demand beats paying for a large instance's peak capacity around the clock. In practice, most horizontally-scaled fleets still choose a moderately sized instance per node rather than the smallest possible unit, because per-instance overhead, such as the operating system and background agents, doesn't shrink proportionally, so there's usually a sweet spot of vertical sizing within a horizontally-scaled fleet worth checking.

Worked example

A team notices their application's tail latency degrades under load while compute utilization sits around 40 percent but memory is pinned near 95 percent. That points to a memory-bound workload, perhaps a large in-memory cache or many concurrent connections each holding buffers, so the fix is a memory-optimized instance type, or vertically scaling to more memory within the current family, not a compute-optimized one, even though "slow under load" superficially suggests a compute problem.

Trade-offs and pitfalls

A common pitfall is resizing based on a single observed metric during one incident rather than a load-test profile across the workload's actual traffic shape, which can overcorrect for one moment and be wrong the rest of the time. Another is defaulting to horizontal scaling because it's assumed to be more resilient, even for a workload with heavy per-instance coordination cost, such as a shared lock or a costly cache warm-up, where fewer larger instances can deliver higher net throughput. Re-check the instance family periodically as the workload's bottleneck resource shifts; a service that used to be compute-bound can become memory-bound once a feature adds caching, and treating the original choice as permanent misses that.

Direct answer

A good benchmarking plan for instance selection has three parts: a representative workload that actually exercises the bottleneck you care about (here, single-thread performance and memory bandwidth, not generic throughput), a small set of low-level metrics that explain why one instance type wins rather than just that it won, and a controlled test procedure that holds everything except the instance type constant. The output isn't "which instance scored highest," it's cost per unit of useful work, since the fastest instance is often not the cheapest way to get the work done.

Structured elaboration

Representative test workloads:

• A synthetic single-threaded microbenchmark that mirrors the application's actual hot path (a tight numeric loop, a hashing routine, whatever the profiler says dominates), not a generic industry benchmark score that may weight differently than your real workload.
• The real application itself, run against a fixed, replayable input (a captured production request trace or a deterministic synthetic dataset), since a microbenchmark can miss effects like cache pressure from the rest of the process.

Low-level metrics to collect, and why:

• Instructions per cycle (IPC): the clearest signal of whether the workload is actually using the core efficiently; a low IPC despite high CPU utilization points at memory stalls, not raw compute limits, which is exactly the failure mode this workload is sensitive to.
• Cache miss rate (last-level cache misses per instruction): directly measures memory-bandwidth pressure; a workload sensitive to memory bandwidth will show its instance-to-instance performance differences show up here before they show up in wall-clock terms.
• CPU steal time (on shared/virtualized hosts): time the virtual CPU wanted to run but a noisy neighbor was using the physical core instead. Skipping this metric is how teams misattribute a noisy-neighbor problem to the instance family itself.
• Sustained clock frequency under load (not just the advertised base or boost clock): many instance types boost briefly then throttle back down once thermal or power limits kick in, and a short benchmark run can hide that if the workload runs for hours in production.

Running tests consistently across instance types:

• Same OS image, kernel version, and compiler flags on every instance type, so the only variable is the hardware.
• Pin the process to specific cores and disable simultaneous multithreading if the application is genuinely single-threaded, to remove scheduler noise from the comparison.
• Run each configuration multiple times and report the median and a high percentile (p90), not a single run, since a single sample can't distinguish real hardware differences from run-to-run noise.
• Where possible, request dedicated (non-shared) tenancy for the benchmark itself, specifically to isolate CPU steal time as a variable you control rather than one you're accidentally measuring.

Worked example

Turning results into a performance-per-cost decision, using deliberately hypothetical instance families to demonstrate the method rather than claiming real published numbers for real products:

$$\text{performance per dollar} = \frac{\text{benchmark score}}{\text{price per hour}}$$

Family A has the higher raw score, the number that would win a naive "which instance is fastest" comparison. Family B has the higher performance-per-dollar, about 18% more useful work per dollar spent, because its lower price more than compensates for its lower raw throughput. Which one is actually correct to choose depends on the constraint: if the workload is latency-bound and needs the fastest single-thread completion regardless of cost, Family A wins; if the workload is throughput-bound and horizontally scalable (run more of the cheaper instance to hit the same aggregate throughput), Family B wins on total cost for the same work done.

Trade-offs and pitfalls

• The most common mistake in this kind of exercise is running the benchmark once, on shared tenancy, and trusting the result; a single run can't separate genuine hardware differences from a noisy neighbor's CPU steal, which is exactly the effect a dedicated-tenancy, multiple-run methodology is designed to catch.
• A workload sensitive to memory bandwidth specifically needs a benchmark that actually stresses memory bandwidth (a stream-style workload touching data far larger than any cache level), not just a CPU-bound compute loop that happens to fit entirely in cache and would show no differentiation between instance types at all.
• Benchmarking once at launch and never again is itself a pitfall: instance families are periodically superseded, and a family that was the best performance-per-dollar choice a year ago may no longer be, especially once a newer generation ships at a similar or lower price.

Direct answer

Training and inference are different workloads that should usually run on different compute, sized for different bottlenecks. Training wants maximum sustained throughput across many accelerators, with tolerance for job-level restarts, while online inference wants low, predictable per-request latency with enough headroom to autoscale to traffic. A right-sized training cluster and a right-sized inference fleet are not the same purchase, even when they serve the same model.

Compute options by workload

Training. Characteristics: long-running batch jobs, where accelerator memory capacity and interconnect bandwidth (how fast accelerators can exchange data with each other) matter as much as raw accelerator count, since multi-accelerator jobs need fast accelerator-to-accelerator communication, not just many chips. Training jobs can usually tolerate preemption if checkpointing is solid, and cost is dominated by total accelerator-hours. Compute options: large multi-GPU instances (graphics processing units, now the general workhorse for parallel numeric computation) or accelerator pods purchased on spot or preemptible pricing (much cheaper capacity the provider can reclaim with short notice) where checkpointing exists, since training's restart tolerance makes it a good fit for interruptible capacity, plus a reserved or committed baseline if training runs nearly continuously.

Online inference. Characteristics: short per-request compute, where tail latency (the slow end of the response-time distribution, not the average) matters more than raw throughput per accelerator, unpredictable request volume that needs to scale quickly, and a workload that typically runs constantly, so it can't be fully spot-based without a fallback, since interruption directly hurts a live user. Compute options: smaller GPU-backed instances, or CPU-only if the model is small enough, behind an autoscaled fleet (containers on an orchestrated platform are the common shape, since they give fast, controlled autoscaling), with a reserved baseline for steady traffic plus on-demand burst capacity, and serving-time optimizations (batching requests together, quantizing the model to shrink it) to make a smaller, cheaper accelerator tier viable.

Worked example

For training, use spot or preemptible multi-GPU nodes for the bulk of the epochs, checkpointing every few minutes so a preemption loses at most that much progress, and keep a small on-demand fallback reserved for the final convergence run, where an uninterrupted window is worth paying for directly. For inference, keep a minimum of two always-on GPU instances as a high-availability floor, autoscale further on a p95-latency or queue-depth metric, and enable request batching up to a batching-latency budget small enough that it doesn't push the response past the target p95.

Trade-offs and pitfalls

A common wrong turn is sizing the inference fleet the way the training cluster was sized (fewer, bigger, less redundant instances) because "that's what worked for training." That trades away exactly the failover redundancy inference needs. Another is running training jobs on capacity reserved for inference "since it's free at night," which only works with strong isolation; a runaway training job must not be able to starve inference of accelerators during a traffic spike. Recommend explicit, separate capacity pools for the two workloads rather than one shared pool governed by priority rules you hope hold under pressure.

Direct answer

For a small CRUD (create, read, update, delete) web app with about 100 daily users and one engineer, I'd recommend serverless functions over a single virtual machine (VM) or a managed container platform. At this traffic level the app is idle almost all the time, which is exactly what per-invocation billing is built for, and a single engineer benefits the most from having no server to patch, size, or keep running at all.

Structured elaboration

Worked example

A single engineer building a to-do-list style CRUD app deploys the API as a set of serverless functions, one per route, backed by a managed database. At launch, with 100 daily users making a handful of requests each, the monthly compute bill is a few dollars, dominated by the database's minimum tier rather than the compute itself, and there's no server for the one-person team to patch, monitor for disk space, or reboot after a kernel update. If traffic grows tenfold to roughly 1,000 daily users, the same functions scale automatically to the new request volume with no redeploy or resizing decision required; the only thing likely to need attention at that point is the database, which has a genuine capacity ceiling that compute scaling doesn't remove, an engineer should watch that dependency rather than assume growth is entirely free just because the compute layer scales itself.

The consideration worth naming for this scenario specifically: even at this small scale, deploying through a continuous-integration (CI) pipeline that packages the same code the same way for local development, staging, and production matters, because a single engineer without that consistency is the person most likely to be debugging a "works on my machine" discrepancy alone, at the worst possible time.

Trade-offs and pitfalls

• A single VM would be the wrong default here mainly because of what it demands of the engineer's time, not because it couldn't technically handle 100 daily users; the opportunity cost of patching and babysitting a box is the real cost, not the server's hourly price.
• What would flip the recommendation: if the application needed a long-lived, stateful connection (a real-time collaborative feature, for instance) or consistently ran requests longer than a serverless platform's execution-time limit, the calculus changes and a managed container platform becomes the better fit even at this small scale.
• A common wrong turn at this stage is over-engineering for a growth curve that hasn't happened yet, standing up a managed container platform "to be ready" for scale a 100-user app doesn't have yet, and paying the setup and ongoing minimum-replica cost for it before there's any real justification.