Infrastructure Scaling, Capacity Planning, and High Availability Questions

Framing

The ask here is explicitly upstream of load testing: before spending money provisioning for a 3x forecast, first find out whether the forecast itself is trustworthy, and whether the answer is even "add capacity" at all, the launch might shift traffic MIX rather than volume, or existing headroom might already cover it.

Validating the forecast itself

• Compare the forecast's methodology against a comparable past launch if one exists. If a similar past feature launch predicted 2x and actually delivered 1.3x, that's real evidence to widen or discount the current 3x forecast's confidence, not just trust the point estimate at face value.
• Break the 3x forecast into its components: is it 3x total requests, 3x users making the same per-user request volume, or the same users making 3x more requests each? Each has different infrastructure implications, for example 3x users usually scales roughly linearly across the system, but 3x per-user activity might concentrate load on one specific endpoint or database table rather than scaling everything uniformly.
• Run a smaller, controlled real-world test where possible: a percentage rollout or a beta cohort exposed to the new feature ahead of full launch, to get an actual measured multiplier from real traffic instead of relying purely on a forecast model.

Measurements to build confidence

Instrument the beta or percentage-rollout cohort's actual traffic multiplier and compare it against the 3x forecast in real time as rollout percentage increases. Separately, confirm CURRENT headroom, if the system already runs at 30% average utilization with room to 3x before hitting any known ceiling, the forecast may not require new provisioning at all, though that conclusion should be checked against PEAK utilization, not average, since peak is the number that actually matters for headroom.

Deciding whether additional capacity is actually needed

Set the decision threshold explicitly, ahead of time, for example: "if the beta cohort's measured multiplier plus a safety margin exceeds current peak headroom, provision ahead of full launch; if it's within headroom, monitor and scale reactively instead." This avoids two common failure modes: provisioning for every forecast regardless of whether current headroom already covers it (wasted cost), and always waiting for reactive autoscaling on a launch-day traffic profile whose ramp speed reactive autoscaling's own reaction time genuinely can't keep up with.

Round-robin

How it works: cycles through backends in order, sending each new request to the next one in the list.
Performs well: stateless services where every request costs roughly the same amount of work, so equal distribution by count is a good proxy for equal distribution by actual load, for example a simple content-serving API.
Performs poorly: when request cost varies a lot (round-robin can send an expensive request and a cheap request to the same backend back to back and call it "fair"), or when backend capacity is uneven, since round-robin has no concept of a backend being slower or busier.
Example service: a stateless, read-only REST API behind a fleet of identically sized instances.

Least-connections

How it works: sends each new request to whichever backend currently has the fewest active connections.
Performs well: workloads with variable request duration, since it naturally avoids piling more work onto a backend already busy with long-running requests, better than round-robin's blind cycling.
Performs poorly: when "active connections" doesn't actually track real load, for example long-lived idle connections like WebSockets or keep-alive connections inflate the connection count without reflecting real work, or when per-request cost varies so much that "fewest connections" still isn't "least actual work."
Example service: a general-purpose backend API with mixed-duration requests, where some endpoints are fast lookups and others do heavier processing.

Consistent hashing

How it works: maps each request to a backend deterministically based on a hash of some key, like client IP, session ID, or cache key, so the same key routes to the same backend repeatedly, and only a small fraction of keys remap when a backend is added or removed, unlike plain modulo hashing, where adding one backend remaps almost everything.
Performs well: stateful services or anything where routing the same key to the same backend repeatedly has real value, most commonly cache locality, for example a caching proxy tier where hitting the same backend for the same key means hitting a warm local cache instead of a cold one.
Performs poorly: when request cost is very uneven across keys, since a "hot key" can overload one backend while others sit idle (the hash doesn't know or care about load, only about the key), and it adds real operational complexity: rebalancing cost when the backend set changes, and needing a fallback strategy when a backend is unhealthy.
Example service: a distributed cache tier, like a fleet of Redis proxies, or a stateful session-affinity requirement where losing the mapping is expensive.

Step 1: convert the flash-sale scenario into a sustained request rate

5,000 concurrent users, each making 4 requests within a 2-minute (120s) session
RPS = (5,000 x 4) / 120 = 166.7 RPS, sustained for the 10-minute window

Treating the 5,000 concurrent users as a steady-state population (users complete checkout and leave, new users arrive, holding the pool near 5,000 for the full 10 minutes) gives a sustained throughput of roughly 167 RPS to design around, not a single instantaneous number.

Step 2: how much concurrency does the app tier actually need to hold

Using Little's Law (concurrent work in flight equals throughput multiplied by average time spent per unit of work), and assuming an average request latency of 150 milliseconds:

Concurrent in-flight requests = 167 RPS x 0.15s = 25

Only about 25 requests are actually being processed at any instant, far fewer than the 5,000 "concurrent users," because most of those users are idle between clicks (reading a page, filling a form), not generating continuous load. This matters because it means the app tier's real bottleneck is throughput and headroom, not raw concurrency.

Sizing the stateless application tier

Assuming, as a stated planning assumption, that a single instance can safely sustain 50 requests/sec at the target utilization:

Baseline instances = 167 RPS / 50 RPS per instance = 3.3, round up to 4

Flash-sale traffic is inherently uncertain (you rarely know the true peak in advance), so apply a roughly 2x safety buffer plus N+1 redundancy (glossed: enough spare capacity that losing any single instance does not remove needed capacity), landing around 8 to 10 instances. Because the whole event is only 10 minutes long, reactive autoscaling alone is too slow; pre-warm this fleet ahead of the sale's start time rather than relying on scale-out to catch up mid-event.

Sizing the caching layer

Cache ops/sec = 167 RPS x 2 ops/request (session lookup + inventory check) = ~333 ops/sec

That is trivial load for even a small managed cache node, which can typically handle orders of magnitude more. The real sizing driver for the cache here is availability (a primary plus a replica, so a single node failure does not take checkout down) and a short TTL (time to live, how long a cached value is kept before it is considered stale) on inventory counts, not raw throughput.

DB approach

Write rate = 167 RPS x 1.5 writes/request (order creation + inventory decrement) = ~250 writes/sec

A single, properly indexed primary with connection pooling can typically sustain this write rate comfortably; the risk is not throughput but correctness under concurrency. Guard inventory decrements with an atomic conditional update (decrement only if stock is greater than zero, in one operation) or a pre-reservation queue, rather than a naive read-then-write, to prevent overselling when many checkout requests race for the same limited stock.

In the moment: mitigate before you diagnose

The first move is the same regardless of cause: protect the system, not solve the mystery. I would immediately confirm autoscaling has headroom to add capacity, engage rate limiting or a web application firewall (WAF, a layer that inspects and blocks malicious HTTP traffic before it reaches the app) at the edge if error rates or latency are climbing, and make sure the database and any downstream dependencies are not the thing actually falling over (they usually are the real bottleneck, not the app servers). A short-term, reversible action (temporarily lowering per-IP rate limits, enabling a CDN (content delivery network, a network of edge servers that cache content near users) "under attack" mode, shedding non-critical background jobs) buys time to diagnose without making a bad guess a permanent one.

Telling attack from organic growth

I look at signals that a bot flood and a viral spike produce very differently:

• Source concentration: a DoS (denial-of-service) attack usually comes from a small number of ASNs (Autonomous System Numbers, IDs for blocks of internet addresses each run by one network operator)/IP ranges or a botnet with unusual geographic clustering; organic viral growth tends to mirror your normal user geography, just amplified.
• Request shape: attack traffic often hits a narrow set of expensive endpoints (login, search, an unauthenticated API) with near-identical request patterns; organic traffic spreads across the site the way real users browse, with referrers pointing to the actual post or platform driving it (Twitter/X, Reddit, a press mention).
• Conversion funnel behavior: real visitors sign up, click through, spend time on page; attack traffic typically does not convert at all, or does so with anomalously fast, scripted timing.
• Cache hit ratio: viral traffic to a few popular pages should see a rising cache hit ratio as the CDN absorbs repeat requests; a credential-stuffing or scraping attack often hits varied or unique URLs, keeping cache hit ratio flat or dropping it.
• User-Agent and TLS fingerprint (a signature derived from how a client negotiates encryption, which tends to differ between real browsers and scripted bots) diversity: organic traffic has a realistic spread of browsers/devices; a lot of identical or clearly scripted User-Agents is a strong attack signal.

None of these is proof alone, so I would look at two or three together before committing to a response.

Afterward

Once things are stable, the followup matters more than the immediate save: write a short postmortem regardless of which it was, add or tighten monitoring on the specific signal that confirmed the diagnosis (so the next spike is triaged faster), and separately evaluate whether the marketing/growth side needs an early-warning channel to ops before a launch or press push, since organic spikes are recurring and worth planning capacity headroom for ahead of time rather than reacting to each one as a fire.

Validate the premise first

Before sizing to it: an 80 percent hit ratio from caching only 30 percent of the keyspace only works if access is skewed (a Zipfian/Pareto-like pattern where a minority of keys account for most traffic). If access were closer to uniform-random across all 100M keys, no 30 percent cache fraction could reach an 80 percent hit ratio, you would need to hold nearly the whole dataset. I would confirm this against real access logs before committing budget to this plan; the rest of the sizing below assumes the skew is real.

Memory sizing (basis: 1 KiB = 1024 bytes, 1 GiB = 2^30 bytes, used consistently throughout)

hot_keys           = 100,000,000 x 30% = 30,000,000
hot_raw_data        = 30,000,000 x 1 KiB               = 28.6 GiB
per_key_overhead    = 30,000,000 x ~90 bytes            = 2.5 GiB   (dict entry + object header, Redis-specific)
data_plus_overhead  = 28.6 + 2.5                         = 31.1 GiB
after_fragmentation = 31.1 x 1.3                         = 40.5 GiB (jemalloc fragmentation allowance)
required_maxmemory  = 40.5 / 0.70                        = 57.8 GiB (keep utilization <= 70% of maxmemory)

Intuition: you are not just storing the raw bytes, Redis's own bookkeeping per key and memory-allocator fragmentation add roughly 40 percent on top of raw data, and you never want a cache running near 100 percent full since eviction storms and replication buffers need slack.

Cluster sizing

At roughly 26 GiB usable memory on a mid-size managed-Redis node class, 57.8 GiB needs 3 primary shards (78 GiB total capacity, comfortable headroom over the 57.8 GiB requirement). Add one replica per shard for high availability: 6 nodes total.

Throughput

The prompt does not give a request rate, so I am stating an assumption explicitly: a mid-size SaaS product at this data scale, peak 50,000 requests/sec. A single Redis shard commonly sustains on the order of 80,000 to 100,000 simple GET ops/sec per core for roughly 1KB values without pipelining (well short of a hard limit, since pipelining and larger instance classes push this several times higher). Split evenly across 3 shards, that is about 16,700 req/sec per shard, roughly 5x headroom under the single-core ceiling, so throughput is not the binding constraint here, memory is. That headroom also absorbs the classic case of one hot key or a temporarily skewed access pattern landing disproportionately on one shard.

Cost assumptions

Roughly: 6 nodes at the chosen instance class, managed-Redis pricing typically running 20 to 40 percent above equivalent raw compute for the patching, failover, and backup automation. This is a rough planning estimate, not a quote, actual pricing should be pulled from the current provider rate card before committing budget.

What I would validate before shipping this sizing

• Real hit-ratio measurement against a shadow cache before assuming the 80/30 split holds.
• Actual per-key value size distribution (average hides a long tail; large outlier values can blow the memory budget).
• Backing-store capacity for the 20 percent miss traffic, since a cache miss under load still has to be served by something.