Scalability Patterns and Techniques Questions
Scaling a system to handle growth in traffic and data: horizontal versus vertical scaling, statelessness, sharding and partitioning strategies, read replicas, and connection pooling. Covers capacity estimation, identifying bottlenecks, and the tradeoffs each scaling axis introduces. The general toolkit for taking a design from thousands to millions of users.
Evaluate the multi-tenant database-partitioning strategies available to a SaaS product: a shared schema for all tenants, a separate schema per tenant, and a fully isolated database per tenant. Discuss tenant isolation, operational overhead, migration complexity, mitigating a hot tenant, and how each approach affects backup/restore and compliance obligations.
Sample Answer
Direct answer
Shared schema, schema-per-tenant, and database-per-tenant trade isolation against operational overhead in a fairly direct line: shared schema is cheapest to run and hardest to isolate, database-per-tenant is the strongest isolation and the most expensive to operate at scale, and schema-per-tenant sits in between. The right default for most SaaS products is shared schema with a tenant-ID column and strict row-level access enforcement, escalating specific tenants to their own schema or database only when isolation or compliance requirements demand it.
Structured elaboration
| Dimension | Shared schema | Schema per tenant | Database per tenant |
|---|---|---|---|
| Tenant isolation | Weakest: logical only, enforced by application code or row-level security on every query | Stronger: tenants can't accidentally query across schemas, but share the underlying database engine, connection pool, and physical resources | Strongest: separate connection pool, separate resource limits, separate blast radius per tenant |
| Operational overhead | Lowest: one schema to migrate, monitor, and back up | Moderate: N schemas to migrate (usually via one script run N times), shared instance to monitor | Highest: N databases to provision, monitor, patch, and back up independently |
| Migration complexity | Simplest: one migration, applies to all tenants at once | Each tenant's schema must be migrated, typically in a loop; a failed migration on one tenant does not have to block others | Same as schema-per-tenant, but now includes coordinating N separate database instances, which is slower and has more infrastructure surface to fail |
| Hot-tenant mitigation | Hardest: a hot tenant's query load is shared with every other tenant's connection pool and cache; requires application-level rate limiting or query governance per tenant | Better: a hot tenant's queries are isolated to its own schema, but still share instance-level resources like buffer cache and disk I/O, input/output operations per second | Best: a hot tenant's dedicated instance can be scaled, resource-limited, or moved without affecting any other tenant |
| Backup, restore, compliance | Hardest to satisfy tenant-specific requirements: restoring one tenant's data means restoring (or filtering out of) a backup that contains every tenant's data, complicating any customer-specific compliance obligation to isolate or delete their data on request | Better: a schema can be backed up and restored somewhat independently, though within a shared instance's backup mechanics | Best: a tenant's data has a clean backup, restore, and deletion boundary that maps directly to compliance requirements like data residency or a right-to-deletion request |
Why shared schema is usually still the right starting default. Most SaaS products have far more small tenants than large ones. Database-per-tenant at 1,000 tenants means 1,000 databases to patch, monitor, and pay for, most of them nearly idle; that operational and cost overhead is rarely justified until a tenant's size, regulatory requirement, or noisy-neighbor impact specifically demands it. Shared schema, with a tenant_id on every row and either application-enforced or database-native row-level access control, keeps operational cost flat as tenant count grows.
Mitigating a hot tenant without changing the whole architecture. Before escalating a hot tenant to its own schema or database, first try per-tenant rate limiting or query budgets within the shared schema, and confirm indexes are structured so the tenant_id is always the leading column (so a hot tenant's queries don't force a wider scan that also slows every other tenant sharing the table). Escalate to schema or database isolation only when a tenant's load genuinely cannot be governed down to a fair share, or when the tenant has a hard compliance requirement (data residency, contractual data isolation) that shared infrastructure cannot satisfy regardless of load.
Worked example
A concrete escalation path: start every tenant on shared schema. When a specific tenant's query volume or data size grows large enough that its queries are measurably degrading other tenants' P95 latency (95th-percentile latency) even after rate limiting and index tuning, migrate that one tenant to its own schema within the same database instance, which isolates its query plans and connection usage without a full new database to operate. If that tenant later has a compliance requirement (for example, a contractual obligation for physically isolated storage) that schema-level isolation cannot satisfy, migrate that tenant again, this time to its own dedicated database. This keeps the operational overhead proportional to the number of tenants that actually need the stronger isolation, rather than paying database-per-tenant overhead for every tenant up front.
As a concrete illustration of that escalation path (illustrative planning numbers, not measured): assume a shared-schema deployment normally holds P95 query latency at 40ms for all tenants, and one tenant, "Tenant A," grows to 80 million rows (versus a typical tenant's 500,000 rows) and 1,200 queries per second at peak, roughly 100x a typical tenant's 12 QPS. Once Tenant A's queries push shared-schema P95 latency past a 100ms threshold for the other tenants sharing the instance, even after rate limiting and index tuning, that crossing is the concrete trigger to migrate Tenant A to its own schema. Suppose Tenant A later signs a contract requiring physically isolated storage, a data-residency clause naming a specific region and no shared infrastructure: schema-per-tenant cannot satisfy a physical-isolation clause like that, since the underlying database instance and its disks are still shared with other tenants, so that specific, named contractual requirement is what triggers the second migration, to database-per-tenant.
Trade-offs & pitfalls
The most common mistake is picking database-per-tenant early because it feels safest, then discovering the operational cost (patching, monitoring, connection-pool sizing) scales linearly with tenant count and becomes the team's dominant maintenance burden long before any tenant actually needed that level of isolation. The opposite mistake is staying on shared schema past the point where a specific tenant's compliance requirement (data residency, contractual isolation, a right-to-deletion request that must not risk touching other tenants' data) genuinely cannot be satisfied by row-level controls; that is not primarily a performance decision and should not be deferred purely to avoid operational overhead. The schema-per-tenant middle tier is often underused: it gives real isolation gains for hot or sensitive tenants without the full cost of a separate database instance, and is worth defaulting to as the escalation step before database-per-tenant rather than jumping straight there.
Design a caching layer for a product-details API that must sustain 10,000 requests per second with a P95 latency target of 50ms. Cover your cache topology (edge CDN, application-level, distributed cache), eviction policy, TTL strategy, how you'd guard against cache stampede, cold-start handling, and the instrumentation you'd add to measure effectiveness.
Sample Answer
Direct answer
Layer the cache so most of the 10,000 requests per second (RPS) never reach the database at all: an edge content delivery network (CDN) for public, cacheable responses, a regional distributed cache (Redis or Memcached) as the authoritative read cache per region, an optional small local in-process cache for the very hottest keys, and database read replicas underneath everything as the last line of defense for the reads that still get through. The 95th-percentile (P95) latency target of 50 milliseconds is achievable because each layer above the database answers in single-digit milliseconds; the design work is mostly about which layer owns which slice of the traffic, how eviction and time-to-live (TTL) are set per data shape, and how the system behaves when a cache entry expires or a whole region briefly loses its cache.
Topology
flowchart LR
U[Client] --> CDN[Edge CDN]
CDN -->|miss| Regional[Regional Redis cluster x3 regions]
Regional -->|miss| App[App server: local hot-key cache]
App -->|miss, singleflight| DB[(Primary DB)]
App -->|read-heavy fallback| Replica[(Read replicas)]
DB --> Regional
Regional --> CDN
A 3-region topology, one regional Redis cluster per region, is the natural shape once traffic is global: each region's application servers read from their own regional Redis cluster first, so a cross-region round trip only happens on a true miss. Cache-warming keeps this useful even right after a deploy or a regional failover: instead of a cold Redis cluster taking the full brunt of traffic, a warm-up job pre-populates the known hot key set from a snapshot or from the surviving region before that region takes production traffic. The failover mechanics themselves (how a region is declared unhealthy, how traffic is rerouted, what replication mode feeds the standby) belong to the availability and disaster-recovery side of the system; what matters here is only that cache-warming is the piece that keeps a freshly-promoted region from serving a wall of cache misses.
Database read replicas sit at the bottom of the hierarchy, not because they are unimportant, but because by design most traffic should never reach them: they exist to absorb the read load that the cache layers above did not catch, keeping the primary database free for writes.
Sizing the read/write mix and the key-popularity shape
The exact read/write ratio depends on the dataset, and it changes both the TTL strategy and the stampede-protection budget:
- An inventory-style catalog might run close to 10,000 reads/s against 100 writes/s, a roughly 99:1 read-heavy mix, where staleness is cheap because writes are rare.
- A more write-active product catalog might run closer to a 95/5 read/write split, where the TTL-length trade-off is tighter: a long TTL serves more reads from cache, but a 5% write rate means more of what's cached is out of date at any moment, so the acceptable TTL shrinks compared to the 99:1 case.
Traffic is also rarely uniform across the catalog. With 10 million products, a Zipfian distribution where roughly 95% of requests hit the top 10% of products is typical for product-detail pages: this is the number that justifies keeping a small, aggressively-warmed hot-key set (the top 10%) rather than trying to cache the long tail with the same priority.
Eviction and TTL strategy
- Use least-recently-used (LRU) eviction in the regional cache with a memory cap, since it is a reasonable default when access patterns are Zipfian: the hot 10% naturally stays resident.
- Tier TTLs by how often data actually changes: a product whose price or stock updates roughly per-minute needs a TTL well under a minute, or event-driven invalidation on write, rather than sitting on a multi-minute TTL that would show stale stock counts. Slower-changing fields (description, images) can carry a much longer TTL.
- Pair short-TTL data with a background cache-priming job for the known hot set, so the top 10% of products refresh proactively instead of every expiration turning into a cold miss during peak traffic.
Cache stampede protection
When a hot key expires, do not let every one of its concurrent readers hit the database at once. Use request coalescing (a "singleflight" pattern): the first request after a miss acquires a per-key lock and performs the real fetch, while concurrent requests for the same key wait for that result instead of issuing their own database queries. Combine this with serving the previous, slightly stale value while the refresh happens in the background rather than making every waiter block, which is what keeps the P95 latency target intact even during a miss.
Masking a slow downstream dependency
Suppose part of the product-detail response depends on a personalization or pricing microservice with 200-500ms of latency, far too slow to hit on every request at a 50ms P95 target. Caching absorbs this by storing that dependency's response keyed by product, so only the first request (or the background refresh) pays the 200-500ms cost. Cache-key design has to account for authenticated versus anonymous traffic here: an anonymous (logged-out) request can share one cache entry per product, but an authenticated request that gets a personalized result cannot be keyed by raw user ID without effectively defeating the cache (one entry per user, near-zero reuse). A workable middle ground is keying authenticated responses by product plus a coarse user segment (for example, a pricing tier or locale) rather than by individual user identity, which keeps the cache's hit rate high while still respecting personalization.
Fragment caching for personalized pages
A product page usually mixes shared page components, images, description, review counts, that are identical for everyone, with personalized widgets, like "recommended for you", that are not. Caching the whole page as one unit forces a choice between caching nothing (because of the personalized part) or caching stale personalization (because of the shared part). Fragment (partial) caching avoids that trade-off: cache the shared shell as one fragment with a longer TTL, and render or fetch the personalized widgets separately with their own short-TTL or no-cache treatment, so the expensive-to-personalize slice does not force the whole page out of the cache.
Instrumentation
Track, per layer (CDN, regional, application): hit ratio, origin request rate, P95 and P99 (99th-percentile) latency, count of stampede/lock-contention events, and TTL distribution against actual data-change frequency. Alert on a sudden hit-ratio drop or an origin request-rate spike, since both are early warning signs that a cache layer stopped doing its job, whether from a bad deploy, an expired warm-up job, or a traffic shape shift away from the assumed Zipfian pattern.
Trade-offs and pitfalls
Serving a stale value during a background refresh trades a small amount of freshness for a large amount of latency and database protection, which is almost always the right trade at this scale, but it has to be a deliberate choice with a bounded staleness window, not an accident of a missing invalidation path. The most common design mistake is applying one TTL policy to the whole product object instead of splitting it by field volatility (price and stock versus description and images), which either stales the fast-moving fields or needlessly re-fetches the slow-moving ones. A second common mistake is keying personalized cache entries by raw user identity, which looks correct in a demo with few users and quietly collapses the cache hit rate once real traffic and real personalization variety show up.
For a read-heavy product catalog service, weigh the trade-offs between replicating a full cache to every region versus partitioning (sharding) cache entries by product or region. Consider read latency, cache-miss patterns, memory and network cost, consistency, and rebalancing complexity, then recommend an approach for a global retailer that sees traffic bursts from multiple regions.
Sample Answer
Direct answer
For a global retailer with bursty, multi-region traffic, neither pure full replication nor pure partitioning wins outright: full replication gives the best latency and simplest rebalancing but pays for it in memory and cross-region sync cost, while partitioning is cheaper but concentrates risk into hotspots when demand shifts. The right default is a hybrid: keep a small, region-local cache of the hottest slice of the catalog fully replicated in every region for latency, and back it with a sharded cache for the long tail, promoting items into the local cache when a region's traffic to them justifies it.
Structured elaboration
| Dimension | Full replication (every region holds the whole cache) | Partitioned (sharded by product or region) |
|---|---|---|
| Read latency | Best: any product is a local hit | Good only when the request lands on a local shard; a remote shard adds a cross-region hop |
| Cache-miss pattern | Only on first global write or expiry; predictable | Lower miss rate per shard for that shard's hot items, but a burst on one product can overload the single shard that owns it |
| Memory & network cost | High: full catalog held N times, one per region, plus cross-region invalidation traffic | Lower: no duplication of the catalog, and update broadcasts are smaller |
| Consistency | Async replication is simplest and typical; synchronous replication for strong consistency adds real latency | Simpler for the shard that owns a given item, since there's one writer path, but reads from other regions still need a remote call or a replication mechanism |
| Rebalancing complexity | Low: adding a region just means standing up another full copy | Higher: partition migrations and consistent-hashing-style reassignment are needed; hotspots require live re-sharding or targeted replication |
Why a global retailer with bursty traffic needs the hybrid, not either extreme
Bursty, multi-region traffic on a retail catalog is rarely uniform: a small set of products (a flash sale, a viral item) drive a disproportionate share of reads at any given time, and which products are hot can shift quickly. Pure partitioning puts that risk on a single shard, since consistent-hashing-style assignment (products and cache shards are placed as points on a circular hash space, so only nearby points move when shards are added; the mechanics of the hash ring itself are covered in more depth under load balancing's consistent-hashing pattern, and what matters here is the caching consequence) doesn't know a key is about to become hot until it already is. Pure full replication avoids that risk entirely but pays a flat memory and cross-region sync tax for the entire long tail of the catalog, most of which is rarely read in any given region.
The hybrid keeps region-local, fully replicated caches sized to each region's actual working set (the products that region's users actually read), backed by a sharded cache holding the full catalog. A traffic-based promotion rule (an item crossing a per-region hit-rate threshold gets pushed into that region's local cache) handles the shifting-hotspot case without requiring the whole catalog to be replicated everywhere.
Worked example
Assume, as a planning input rather than a measured fact, a product catalog sized at 50 GB, served across 6 regions.
Full replication cost:
50GB×6regions=300GB total cache memory
Partitioned cost (no duplication, split evenly across 6 shards, plus a replication factor of 2 within each shard for availability rather than for cross-region latency):
6shards50GB≈8.3GB per shard,50GB×2=100GB total with the availability replica
The partitioned approach uses roughly a third of the memory of full replication (100 GB versus 300 GB) at this illustrative catalog size. The hybrid sits between the two: if each region's working set is, say, 10% of the catalog (5 GB), replicating just that slice to all 6 regions costs:
5GB×6=30GB
on top of the 100 GB sharded backing store, for roughly 130 GB total, a fraction of full replication's 300 GB while still giving most reads (the ones hitting each region's working set) a local hit.
Trade-offs & pitfalls
- The hybrid's promotion rule needs a threshold and a demotion path; without demotion, the region-local cache grows unbounded as items get promoted but never removed, eventually approaching full replication's cost anyway.
- Cross-region invalidation is still required for the sharded backing store even in the hybrid; underestimating that traffic (versioned, pub/sub-style invalidation messages rather than synchronous broadcasts) is a common way the "cheaper" option ends up not being cheaper.
- A single globally hot product (a flash sale item) can still overload the shard that owns it even with promotion in place, if promotion reacts slower than the traffic spike; this is the scenario that specifically motivates proactive cache warming ahead of known events rather than purely reactive promotion.
- A content delivery network (CDN, a network of edge servers that cache content close to users) is a natural complement for static product assets (images, descriptions) but doesn't solve the dynamic pricing/inventory caching problem this comparison is about; don't conflate the two layers.
- Getting the region-local cache's time-to-live (TTL, how long a cached value is considered valid before refresh) too long trades staleness (wrong price or stock shown) for the latency win; too short and the hybrid starts behaving like the sharded-only design under load.
Explain how a CDN works and when you'd reach for one in a global application. Cover edge caching, cache-control headers, TTL strategy, surrogate keys, origin failover, cache invalidation, and how you'd handle dynamic versus static content (signed URLs, edge logic). What are the cost and operational trade-offs?
Sample Answer
Direct answer
A content delivery network (CDN) is a fleet of geographically distributed edge servers that cache and serve content close to the requesting user, cutting latency and offloading the origin server. You reach for one in a global application whenever a meaningful share of traffic is cacheable and users are spread across regions, since the CDN removes both the network-distance cost and the repeated origin load for the same content served over and over.
Structured elaboration
Edge caching and cache-control
Edge nodes store a response and decide how long to keep it based on HTTP caching headers the origin sets, most commonly Cache-Control (for example public, max-age=3600 to allow shared caching for one hour) and Vary (to tell the CDN that responses differ by a request header, such as Vary: Accept-Encoding, so it must cache separate copies per variant). ETag lets the edge or the client do a conditional request and get a cheap "not modified" response instead of re-downloading unchanged content.
TTL strategy
Static, versioned assets (images, bundled JS/CSS with a hash in the filename) can take long time-to-live (TTL) values, hours to days, since a content change simply ships under a new URL. Semi-dynamic content (an API response that changes occasionally, a personalized-but-cacheable fragment) needs a short TTL, seconds to minutes, and benefits from stale-while-revalidate (serve the slightly-stale cached copy immediately while refreshing it in the background) to keep latency low without serving badly outdated data. As an illustrative split, not a universal rule: a content-heavy site might find roughly 70% of its requests are static and 30% are dynamic or personalized, which is a useful mental model for deciding where TTL and invalidation effort should concentrate, since the static 70% is where aggressive caching pays off with the least risk.
Surrogate keys and invalidation
A Surrogate-Key (or Surrogate-Control) header lets the origin tag a response with one or more logical keys (a product ID, a category, a build version) so that a single purge call can invalidate every cached object sharing that tag, without the origin needing to know every individual URL that resulted from it. This is the mechanism that makes targeted, safe invalidation of a large object set practical: purging by surrogate key when a product's price changes clears exactly the cached responses for that product, not the whole cache. Purge and invalidate are distinct operations: a purge removes the object outright (next request is a full miss), while marking an object stale lets the edge serve it once more while fetching a fresh copy in the background, which is gentler on the origin during a large invalidation event.
Origin failover
Configuring a primary and secondary origin pool, with edge-side health checks, lets the CDN route around a failing origin automatically. Origin shielding, routing all edge cache misses through one designated shield location before they reach the origin, protects the origin from a "thundering herd" of simultaneous misses across many edge nodes after a mass invalidation or cold start.
Dynamic versus static content
Static assets are served straightforwardly from edge cache with long TTLs. Dynamic or personalized content needs a different approach:
- User-uploaded static assets (profile photos, attachments) are cacheable like any static asset once uploaded, but the origin (typically an object store, not the application server) needs its access secured, commonly signed URLs or a signed cookie, so the CDN can serve the object publicly at the edge while the origin itself stays access-controlled rather than open to the world.
- Signed URLs and edge logic: for content that must be authorized per-user but is still worth caching, a signed URL with an expiry lets the edge validate the request without a round trip to the application, and edge compute (code that runs at the edge rather than only at the origin) can handle that validation, run A/B test bucketing, or assemble a personalized response from cached fragments, all close to the user.
Effectiveness metrics
The metrics that tell you whether a CDN deployment is actually working: cache hit ratio (the share of requests served from edge without reaching origin), the TTL distribution actually observed across your cached object types (confirming static assets are getting long TTLs and dynamic ones short, as designed), and origin request rate (the traffic the origin actually sees, which is what a CDN exists to reduce). These are the metrics to instrument and watch, not numbers to assume; a CDN misconfigured with no-store on cacheable responses can show a healthy-looking deployment with a near-zero hit ratio.
Cost and operational trade-offs
- Caching reduces origin compute and egress cost but adds CDN service fees and the operational surface of managing cache rules, purges, and edge logic.
- Edge compute adds real capability (auth, personalization, A/B logic close to the user) at the cost of a new runtime to test, deploy, and debug, plus more vendor-specific surface.
- Aggressive TTLs cut origin load and latency the most but raise the risk of serving stale content, especially for personalized or frequently-changing data; conservative TTLs are safer but leave more traffic hitting the origin. Tuning this trade-off is what the metrics above are for.
Illustrative cache-hit and origin-load estimate
To reason about the trade-off concretely rather than by assumption, here is a worked, fully-pinned example (all inputs stated, not measured or claimed as a real published result):
Assume a video-thumbnail service receives 1,000,000 thumbnail requests per day, and, before any CDN, all of them hit the origin. Suppose an aggressive edge-TTL policy is expected to reach a 90% cache hit ratio, while a more conservative TTL policy (chosen for a use case sensitive to staleness) is expected to reach only 60%. The origin request rate under each policy:
origin requests=total requests×(1−hit ratio) aggressive: 1,000,000×(1−0.90)=100,000 origin requests/day conservative: 1,000,000×(1−0.60)=400,000 origin requests/dayThe aggressive policy cuts origin load 4x further than the conservative one (400,000 / 100,000), which is the shape of the trade-off: every point of hit-ratio improvement is a direct, proportional reduction in origin traffic and cost, and the right policy for a given asset type depends on how much staleness that asset can tolerate, weighed against that origin-load reduction.
Worked example
A global content platform serves images through a CDN with Cache-Control: public, max-age=604800 (roughly 7 days) and a Surrogate-Key per content ID. When an image is replaced, the origin issues a single purge by surrogate key rather than needing to know every resized/format variant URL the CDN generated from it, all of which share that key. Meanwhile the platform's semi-dynamic JSON API for "current viewer count" uses Cache-Control: public, max-age=5, stale-while-revalidate=30 so the edge serves a slightly-stale count instantly while refreshing in the background, keeping origin load low without users perceiving staleness beyond a few seconds.
Trade-offs & pitfalls
- Caching personalized or dynamic content without a correctness plan (the wrong
Varyheader, or none at all) is the most common CDN mistake: it silently serves one user's personalized response to another. - A CDN can worsen correctness for content that changes faster than its TTL if invalidation is not wired up; caching is not a substitute for proper invalidation, it is a mechanism that makes fast, correct invalidation more necessary, not less.
- Origin shielding helps against mass cache-miss storms but adds a hop and a new potential bottleneck of its own if the shield location is undersized relative to peak miss traffic.
- Treating hit ratio as the only success metric misses half the picture: a high hit ratio on the wrong content (over-caching data that needed to be fresh) is a correctness bug wearing a good-looking dashboard.
A cost-conscious SaaS customer has highly spiky traffic, roughly a 10x daily swing. How would you pick instance types and an autoscaling policy that keeps P95 latency on target while minimizing cost? Discuss reserved versus spot instances, burstable instances, predictive versus reactive scaling, and container versus VM-based scaling.
Sample Answer
Direct answer
For a 10x daily traffic swing, reserve baseline capacity for the trough, autoscale the delta up to peak, and lean predictive/scheduled scaling over purely reactive, since a daily swing is exactly the regular, forecastable pattern predictive scaling is good at. Use burstable instances only where bursts are genuinely short; use spot for stateless, interruption-tolerant work; keep on-demand or reserved capacity for the latency-critical path. The instance and purchasing decisions matter less than getting this basic shape right: match committed spend to the floor, elastic spend to the swing.
Structured elaboration
Reserved vs. spot vs. on-demand
- Reserved (or savings-plan) capacity for the steady trough load, since that portion runs 24/7 regardless of the swing and benefits most from a committed discount.
- Spot for stateless, horizontally-scaled, interruption-tolerant work (background/batch consumers), paired with graceful draining so an interrupted instance doesn't corrupt in-flight work.
- On-demand as the buffer for latency-critical capacity above the reserved baseline, and as an immediate fallback when spot capacity is reclaimed.
Burstable vs. fixed instances. Burstable instances (which accumulate CPU credit during idle periods and spend it during bursts) are worth using only if the burst is genuinely short relative to how fast credits accumulate; a 10x swing that holds for hours, not minutes, will exhaust accumulated credit and fall back to a throttled baseline performance level, which is the opposite of what you want during peak. For a sustained multi-hour peak, more numerous smaller fixed instances scaled horizontally is the safer choice.
Predictive vs. reactive. A daily 10x swing is about as regular a pattern as autoscaling ever sees, which makes it a strong candidate for scheduled/predictive pre-scaling: scale up ahead of the known daily ramp rather than waiting for a reactive trigger to catch up. Keep a reactive, latency-based layer running underneath as the backstop for whatever the schedule gets wrong on an atypical day (a P95 latency target, the 95th-percentile response time, i.e. the value 95% of requests come in under, is a good backstop metric since it reflects what users actually feel).
Containers vs. VMs. A container platform with a cluster autoscaler can bin-pack (efficiently pack workloads of different sizes onto the fewest possible shared machines, matching each workload's resource need to available capacity the way you'd pack differently-sized boxes into as few shipping crates as possible) across mixed instance types (reserved, on-demand, spot) on shared nodes and starts new replicas faster than a fresh VM boots, which matters directly for how quickly the fleet can absorb the ramp into peak. Plain VM-level autoscaling is simpler operationally but coarser-grained and slower to provision; it's a reasonable choice when multi-tenant isolation or licensing requirements rule out sharing nodes.
Worked example
Total cost of ownership (TCO) comparison: fixed peak-sized warm capacity vs. autoscaling. Assume, as stated planning inputs, not measured figures: peak load requires 100 instances for 4 hours/day, trough load requires 10 instances for the remaining 20 hours/day, over a 90-day (roughly 3-month) window, at an illustrative $0.10/instance-hour.
Fixed warm capacity sized to peak, running at 100 instances around the clock:
100×24×90×$0.10=$21,600Autoscaling between trough and peak:
(100×4+10×20)×90×$0.10=600×90×$0.10=$5,400 $21,600$5,400=0.25In this illustrative model, autoscaling costs a quarter of what permanently warm peak-sized capacity would, a $16,200 saving over 3 months. That gap is exactly why "just overprovision and stop worrying about it" is rarely the right default for a regular, predictable swing like this one; it's a defensible fallback only when the swing is irregular enough that autoscaling reliably can't keep up, which a stable daily pattern isn't. (This simplified model ignores scale transition costs and the risk of a slow ramp missing the service level objective, SLO, the measurable target committed to for latency or availability; a real sizing exercise would validate the ramp time against the actual latency budget by load-testing the actual scale-out ramp: confirming that a newly-scaled instance passes its health check and starts serving real traffic before the P95 latency budget for that request is exhausted, not just that the instance count eventually reaches the target.)
Fast, extreme burst: 1-hour marketing spike from 1,000 to 100,000 requests per second (RPS). A daily-swing policy tuned for a gradual 10x ramp is the wrong tool for a sudden, 100x, hour-long spike: neither reserved nor on-demand compute autoscaling reacts fast enough on its own at that magnitude. Layer the response instead: put a content delivery network (CDN, a network of edge servers caching content close to users) in front to offload whatever fraction of that traffic is cacheable, entirely off origin compute; use serverless (functions as a service, FaaS, where the platform scales invocation concurrency per-request rather than per-instance) or a pre-warmed burst pool to absorb the residual dynamic traffic within the 1-hour window, since compute autoscaling alone is too slow to react meaningfully inside that timeframe.
Bursting to a second region vs. permanent capacity. For a short or infrequent peak, temporarily bursting overflow traffic to a second region avoids paying for duplicate capacity that sits idle everywhere the rest of the time, but it only works if the service can actually run statelessly in a freshly-spun-up region and if the operational readiness (data replication, DNS or traffic-shifting mechanics) was built and tested ahead of time, not improvised during the peak. Provisioning a single larger region permanently is simpler operationally but pays for that simplicity around the clock; the second-region approach trades operational complexity for lower steady-state cost, which is worth it exactly when the peak is genuinely short or rare.
Trade-offs & pitfalls
- Burstable-instance credit exhaustion under a sustained (not brief) peak silently degrades performance right when peak matters most; validate actual burst duration against credit accumulation before relying on this instance class for a multi-hour swing.
- Spot capacity is only appropriate for the fraction of the fleet that tolerates interruption; putting latency-critical serving on spot to save cost trades away exactly the reliability the SLO depends on.
- Predictive scaling tuned to a daily pattern will under-react to anything that isn't the daily pattern (a genuine spike layered on top of the normal swing); it needs the same kind of reactive, latency-based backstop described above for the daily-swing case: a layer that watches P95 latency directly and scales further whenever the schedule under-predicts, rather than assuming the daily schedule alone is sufficient.
- A cost comparison like the TCO example above is only as good as its throughput-per-instance and hours-at-peak assumptions; treat those as inputs to validate against real load tests, not numbers to trust by default.
Unlock Full Question Bank
Get access to all Scalability Patterns and Techniques interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.