Scalability Patterns and Techniques Questions

Direct answer

A content delivery network (CDN) is a fleet of geographically distributed edge servers that cache and serve content close to the requesting user, cutting latency and offloading the origin server. You reach for one in a global application whenever a meaningful share of traffic is cacheable and users are spread across regions, since the CDN removes both the network-distance cost and the repeated origin load for the same content served over and over.

Structured elaboration

Edge caching and cache-control

Edge nodes store a response and decide how long to keep it based on HTTP caching headers the origin sets, most commonly Cache-Control (for example public, max-age=3600 to allow shared caching for one hour) and Vary (to tell the CDN that responses differ by a request header, such as Vary: Accept-Encoding, so it must cache separate copies per variant). ETag lets the edge or the client do a conditional request and get a cheap "not modified" response instead of re-downloading unchanged content.

TTL strategy

Static, versioned assets (images, bundled JS/CSS with a hash in the filename) can take long time-to-live (TTL) values, hours to days, since a content change simply ships under a new URL. Semi-dynamic content (an API response that changes occasionally, a personalized-but-cacheable fragment) needs a short TTL, seconds to minutes, and benefits from stale-while-revalidate (serve the slightly-stale cached copy immediately while refreshing it in the background) to keep latency low without serving badly outdated data. As an illustrative split, not a universal rule: a content-heavy site might find roughly 70% of its requests are static and 30% are dynamic or personalized, which is a useful mental model for deciding where TTL and invalidation effort should concentrate, since the static 70% is where aggressive caching pays off with the least risk.

Surrogate keys and invalidation

A Surrogate-Key (or Surrogate-Control) header lets the origin tag a response with one or more logical keys (a product ID, a category, a build version) so that a single purge call can invalidate every cached object sharing that tag, without the origin needing to know every individual URL that resulted from it. This is the mechanism that makes targeted, safe invalidation of a large object set practical: purging by surrogate key when a product's price changes clears exactly the cached responses for that product, not the whole cache. Purge and invalidate are distinct operations: a purge removes the object outright (next request is a full miss), while marking an object stale lets the edge serve it once more while fetching a fresh copy in the background, which is gentler on the origin during a large invalidation event.

Origin failover

Configuring a primary and secondary origin pool, with edge-side health checks, lets the CDN route around a failing origin automatically. Origin shielding, routing all edge cache misses through one designated shield location before they reach the origin, protects the origin from a "thundering herd" of simultaneous misses across many edge nodes after a mass invalidation or cold start.

Dynamic versus static content

Static assets are served straightforwardly from edge cache with long TTLs. Dynamic or personalized content needs a different approach:

• User-uploaded static assets (profile photos, attachments) are cacheable like any static asset once uploaded, but the origin (typically an object store, not the application server) needs its access secured, commonly signed URLs or a signed cookie, so the CDN can serve the object publicly at the edge while the origin itself stays access-controlled rather than open to the world.
• Signed URLs and edge logic: for content that must be authorized per-user but is still worth caching, a signed URL with an expiry lets the edge validate the request without a round trip to the application, and edge compute (code that runs at the edge rather than only at the origin) can handle that validation, run A/B test bucketing, or assemble a personalized response from cached fragments, all close to the user.

Effectiveness metrics

The metrics that tell you whether a CDN deployment is actually working: cache hit ratio (the share of requests served from edge without reaching origin), the TTL distribution actually observed across your cached object types (confirming static assets are getting long TTLs and dynamic ones short, as designed), and origin request rate (the traffic the origin actually sees, which is what a CDN exists to reduce). These are the metrics to instrument and watch, not numbers to assume; a CDN misconfigured with no-store on cacheable responses can show a healthy-looking deployment with a near-zero hit ratio.

Cost and operational trade-offs

• Caching reduces origin compute and egress cost but adds CDN service fees and the operational surface of managing cache rules, purges, and edge logic.
• Edge compute adds real capability (auth, personalization, A/B logic close to the user) at the cost of a new runtime to test, deploy, and debug, plus more vendor-specific surface.
• Aggressive TTLs cut origin load and latency the most but raise the risk of serving stale content, especially for personalized or frequently-changing data; conservative TTLs are safer but leave more traffic hitting the origin. Tuning this trade-off is what the metrics above are for.

Illustrative cache-hit and origin-load estimate

To reason about the trade-off concretely rather than by assumption, here is a worked, fully-pinned example (all inputs stated, not measured or claimed as a real published result):

Assume a video-thumbnail service receives 1,000,000 thumbnail requests per day, and, before any CDN, all of them hit the origin. Suppose an aggressive edge-TTL policy is expected to reach a 90% cache hit ratio, while a more conservative TTL policy (chosen for a use case sensitive to staleness) is expected to reach only 60%. The origin request rate under each policy:

$$\text{origin requests} = \text{total requests} \times (1 - \text{hit ratio})$$ $$\text{aggressive: } 1{,}000{,}000 \times (1 - 0.90) = 100{,}000 \text{ origin requests/day}$$ $$\text{conservative: } 1{,}000{,}000 \times (1 - 0.60) = 400{,}000 \text{ origin requests/day}$$

The aggressive policy cuts origin load 4x further than the conservative one (400,000 / 100,000), which is the shape of the trade-off: every point of hit-ratio improvement is a direct, proportional reduction in origin traffic and cost, and the right policy for a given asset type depends on how much staleness that asset can tolerate, weighed against that origin-load reduction.

Worked example

A global content platform serves images through a CDN with Cache-Control: public, max-age=604800 (roughly 7 days) and a Surrogate-Key per content ID. When an image is replaced, the origin issues a single purge by surrogate key rather than needing to know every resized/format variant URL the CDN generated from it, all of which share that key. Meanwhile the platform's semi-dynamic JSON API for "current viewer count" uses Cache-Control: public, max-age=5, stale-while-revalidate=30 so the edge serves a slightly-stale count instantly while refreshing in the background, keeping origin load low without users perceiving staleness beyond a few seconds.

Trade-offs & pitfalls

• Caching personalized or dynamic content without a correctness plan (the wrong Vary header, or none at all) is the most common CDN mistake: it silently serves one user's personalized response to another.
• A CDN can worsen correctness for content that changes faster than its TTL if invalidation is not wired up; caching is not a substitute for proper invalidation, it is a mechanism that makes fast, correct invalidation more necessary, not less.
• Origin shielding helps against mass cache-miss storms but adds a hop and a new potential bottleneck of its own if the shield location is undersized relative to peak miss traffic.
• Treating hit ratio as the only success metric misses half the picture: a high hit ratio on the wrong content (over-caching data that needed to be fresh) is a correctness bug wearing a good-looking dashboard.

Direct answer

Cache hit ratio is the fraction of lookups a cache can answer without going to the origin (hits divided by total lookups); a cache miss is a lookup where the value isn't present or has expired, forcing a fetch from the origin and usually a write back into the cache; cache warmup is the process of populating a cache, at startup or after a restart, before it can offer a useful hit ratio. Whether adding an application cache (memcached, Redis, or similar) is worth it comes down to a simple cost comparison: what a miss costs you (a database round trip, a slow computation, an external API call) versus what running the cache costs you (infrastructure, and the risk of serving stale data), not a fixed hit-ratio threshold you're supposed to hit.

Structured elaboration

When to add a cache

• The workload is read-heavy with requests that repeat: the same or similar queries recur often enough that a cached answer is reused rather than computed once and discarded.
• The thing being cached is expensive relative to a cache lookup: a slow database query, an external API call, or a CPU-heavy computation are all good candidates; something already fast to compute gains little.
• The data can tolerate the staleness a cache implies, or the system can actively invalidate the cache on writes; if every read must reflect the absolute latest write, caching adds a consistency problem you have to solve, not just a performance win.

What hit ratio would justify the cost

There's no universal number, because "justified" depends on the ratio between the cost avoided per hit and the cost of running the cache, not on the hit ratio in isolation. A commonly cited planning heuristic for latency- and cost-sensitive systems is to aim for roughly 70 to 80% hit ratio; treat that as a starting heuristic to validate against your own workload, not as a target derived from anything specific to your system. A system where each miss is very expensive (an external, rate-limited, or metered API call) can be worth caching even at a 50% hit ratio, while a system where the origin call is already cheap may not justify caching even at 90%.

Three practical ways to improve an existing cache's effectiveness

1. Tune time-to-live (TTL, how long a cached value is considered valid before it must be refreshed) per key type rather than using one global value: longer TTLs for stable data, shorter for volatile data, so you're not needlessly re-fetching stable data or serving stale volatile data.
2. Improve cache key design: normalize keys (strip user-specific noise that doesn't actually change the result, use consistent prefixes) so semantically identical requests share a cache entry instead of each generating its own miss.
3. Warm proactively and avoid a thundering herd on expiry: pre-populate known-hot keys at deploy or restart, and use a read-through or refresh-ahead pattern where a background job refreshes a key just before it expires, rather than letting many concurrent requests all miss on the same expired key at once.

Worked example

Assume, as a planning input rather than a measured fact, that an endpoint currently receives 10,000 requests per minute with no caching, and that each request costs 20 ms of database time. Total database time consumed per minute today:

$10{,}000 \times 20\,\text{ms} = 200{,}000\,\text{ms} = 200\,\text{s of database time per minute}$

If a cache reaches a 75% hit ratio on this endpoint, the number of requests still reaching the database is:

$10{,}000 \times (1 - 0.75) = 2{,}500 \text{ requests per minute}$

$2{,}500 \times 20\,\text{ms} = 50{,}000\,\text{ms} = 50\,\text{s of database time per minute}$

That's a 75% reduction in database load, directly tracking the hit ratio, from 200 seconds of database time per minute down to 50. Whether that reduction is "worth it" then depends on what those 150 seconds of freed-up database time are worth relative to running the cache, not on the 75% figure by itself.

Trade-offs & pitfalls

• Chasing a higher hit ratio as a goal in itself, rather than as a proxy for reduced backend cost, can lead to caching things that barely help (already-cheap lookups) while a genuinely expensive but less frequent lookup goes uncached.
• A TTL that's too long trades staleness risk for hit ratio; too short and the cache behaves closer to no cache at all under the same access pattern.
• Skewed (heavily concentrated, or Zipfian) access patterns mean a handful of keys drive most hits; eviction policy choice (LRU, least-recently-used, versus LFU, least-frequently-used) matters more under this kind of skew than under uniform access, since LRU can evict a very-frequently-used-but-not-most-recent key that LFU would keep.
• Expose hit and miss counters through your application performance monitoring (APM, application performance monitoring) or metrics stack; without that visibility, a regression in hit ratio after a code change (a key-format change, a new high-cardinality parameter) can go unnoticed until backend load spikes.

Direct answer

At 1 million requests per second (RPS), even a fully stateless service hits six distinct application-layer ceilings before the database ever enters the picture: raw network throughput, the thread/concurrency model used to handle requests, how connections are accepted and held, the CPU (central processing unit) cost of TLS (Transport Layer Security, the encryption protocol behind HTTPS) termination, the CPU and allocation cost of serialization, and garbage-collection (GC) pauses in managed runtimes. Each has its own symptom, its own mitigation, and its own way to measure how much it is actually costing you, so the right approach is to profile for all six rather than assume which one dominates.

Bottleneck, mitigation, and how to quantify it

Worked example: why the thread model has to change at this scale

As an illustrative assumption, not a measured figure, suppose the in-application processing budget (excluding network transit) needed to hit a reasonable P95 (95th-percentile latency) target is 20 ms per request. By Little's Law, the number of requests that must be handled concurrently to sustain 1,000,000 RPS at that latency is:

$N = X \times R = 1{,}000{,}000 \text{ RPS} \times 0.02\text{s} = 20{,}000 \text{ concurrent requests}$

If each of those were handled by a dedicated OS thread, and again as an illustrative assumption, suppose each thread reserves roughly 2 MB of stack:

$20{,}000 \text{ threads} \times 2\text{ MB/thread} = 40{,}000\text{ MB} = 39.06 \text{ GiB in thread stacks alone}$

That is memory spent before a single byte of request data or response buffer is allocated, and it scales linearly with load: doubling throughput or the latency budget doubles the stack overhead in this model. This is the concrete argument for an event-driven or lightweight-concurrency runtime: the same 20,000-way concurrency can be held by a small, fixed pool of OS threads multiplexing many logical requests, so memory and context-switch overhead stop scaling with request count.

Trade-offs and pitfalls

• Optimizing the bottleneck that is easiest to fix instead of the one profiling actually shows dominates. All six of these can look plausible from first principles; the profiling step is what separates a real fix from a guess.
• Fixing serialization or TLS cost with a change that only helps if both client and server can be changed together (a binary format, session resumption). Confirm you actually control both ends before committing to the approach.
• Tuning a garbage collector for lower pause times without checking the throughput cost; low-pause collectors typically trade some raw throughput or memory overhead for shorter, more predictable pauses, which is usually the right trade at the tail but should be a deliberate choice.
• Solving connection-handling and thread-model problems independently when they interact: an async runtime that still blocks on a synchronous crypto or serialization call in the hot path reintroduces the same contention it was meant to remove.
• Load-testing with unrealistic request shapes (uniform tiny payloads, no TLS) that hide exactly the bottlenecks (serialization at scale, real handshake cost) this question is about; test with payload sizes and TLS configuration that match production.

Direct answer

For a global retailer with bursty, multi-region traffic, neither pure full replication nor pure partitioning wins outright: full replication gives the best latency and simplest rebalancing but pays for it in memory and cross-region sync cost, while partitioning is cheaper but concentrates risk into hotspots when demand shifts. The right default is a hybrid: keep a small, region-local cache of the hottest slice of the catalog fully replicated in every region for latency, and back it with a sharded cache for the long tail, promoting items into the local cache when a region's traffic to them justifies it.

Structured elaboration

Why a global retailer with bursty traffic needs the hybrid, not either extreme

Bursty, multi-region traffic on a retail catalog is rarely uniform: a small set of products (a flash sale, a viral item) drive a disproportionate share of reads at any given time, and which products are hot can shift quickly. Pure partitioning puts that risk on a single shard, since consistent-hashing-style assignment (products and cache shards are placed as points on a circular hash space, so only nearby points move when shards are added; the mechanics of the hash ring itself are covered in more depth under load balancing's consistent-hashing pattern, and what matters here is the caching consequence) doesn't know a key is about to become hot until it already is. Pure full replication avoids that risk entirely but pays a flat memory and cross-region sync tax for the entire long tail of the catalog, most of which is rarely read in any given region.

The hybrid keeps region-local, fully replicated caches sized to each region's actual working set (the products that region's users actually read), backed by a sharded cache holding the full catalog. A traffic-based promotion rule (an item crossing a per-region hit-rate threshold gets pushed into that region's local cache) handles the shifting-hotspot case without requiring the whole catalog to be replicated everywhere.

Worked example

Assume, as a planning input rather than a measured fact, a product catalog sized at 50 GB, served across 6 regions.

Full replication cost:

$50\,\text{GB} \times 6\,\text{regions} = 300\,\text{GB total cache memory}$

Partitioned cost (no duplication, split evenly across 6 shards, plus a replication factor of 2 within each shard for availability rather than for cross-region latency):

$\frac{50\,\text{GB}}{6\,\text{shards}} \approx 8.3\,\text{GB per shard}, \qquad 50\,\text{GB} \times 2 = 100\,\text{GB total with the availability replica}$

The partitioned approach uses roughly a third of the memory of full replication (100 GB versus 300 GB) at this illustrative catalog size. The hybrid sits between the two: if each region's working set is, say, 10% of the catalog (5 GB), replicating just that slice to all 6 regions costs:

$5\,\text{GB} \times 6 = 30\,\text{GB}$

on top of the 100 GB sharded backing store, for roughly 130 GB total, a fraction of full replication's 300 GB while still giving most reads (the ones hitting each region's working set) a local hit.

Trade-offs & pitfalls

• The hybrid's promotion rule needs a threshold and a demotion path; without demotion, the region-local cache grows unbounded as items get promoted but never removed, eventually approaching full replication's cost anyway.
• Cross-region invalidation is still required for the sharded backing store even in the hybrid; underestimating that traffic (versioned, pub/sub-style invalidation messages rather than synchronous broadcasts) is a common way the "cheaper" option ends up not being cheaper.
• A single globally hot product (a flash sale item) can still overload the shard that owns it even with promotion in place, if promotion reacts slower than the traffic spike; this is the scenario that specifically motivates proactive cache warming ahead of known events rather than purely reactive promotion.
• A content delivery network (CDN, a network of edge servers that cache content close to users) is a natural complement for static product assets (images, descriptions) but doesn't solve the dynamic pricing/inventory caching problem this comparison is about; don't conflate the two layers.
• Getting the region-local cache's time-to-live (TTL, how long a cached value is considered valid before refresh) too long trades staleness (wrong price or stock shown) for the latency win; too short and the hybrid starts behaving like the sharded-only design under load.

Direct answer

Use the temporary read replica as the mechanism that turns an in-place resize (which can mean real downtime) into a controlled cutover: provision the replica already at the larger CPU/memory spec, let it fully catch up to the primary, briefly pause writes, promote the replica to primary, and repoint the application. The actual write-unavailability window is bounded by how long it takes to drain in-flight writes and flip the connection target, not by the resize operation itself, which is why this pattern minimizes downtime even though it isn't strictly zero-downtime.

Structured elaboration

Pre-checks

• Confirm the exact target CPU/memory spec against measured load, not a guess, and confirm a maintenance window and a communicated service level objective (SLO, the measurable target for allowed downtime or latency impact) for the operation.
• Take a fresh on-demand snapshot immediately before starting, independent of the replica strategy, as a last-resort fallback.
• Verify current replication lag baseline and that the environment supports creating a same-region replica sized larger than the current primary.
• Confirm the automation (scripts, IaC) for promotion and connection-string cutover has been tested outside of this incident, not written live.

Rolling steps

1. Create a read replica provisioned at the new, larger instance spec. Let it catch up and monitor replication lag until it's negligible and stays that way for a sustained period, not just a single low reading.
2. Briefly quiesce writes on the current primary (put the application into a short read-only or write-paused mode).
3. Promote the replica to primary. Because it was fully caught up at the moment of promotion, this preserves the data that existed at quiesce time.
4. Repoint the application's write target to the newly-promoted primary (a connection-string or routing change, ideally something that doesn't require an application redeploy).
5. Resume writes and run smoke tests against critical read and write paths.
6. Rebuild redundancy: the original (smaller) instance can be resized and re-added as a replica, or replaced, restoring the topology's normal read-replica count.

Fallback options

• If the replica fails to catch up before the maintenance window closes, abort the promotion; investigate whether replication is network- or I/O-bound, and either wait for a longer window or address the bottleneck before retrying.
• If a data mismatch or unexpected inconsistency is detected after promotion, the fallback is the pre-operation snapshot, not the old primary (which may now be behind); restore from snapshot to a fresh instance if this happens.

Monitoring

• Before: replication lag trend over a real observation window, not a single point-in-time check, plus baseline CPU/memory/connection counts to compare against post-cutover.
• During: replication lag right up to the promotion moment, since promoting a replica that's meaningfully behind means losing whatever writes happened after its last applied transaction.
• After: error rates, write and read latency, and a targeted data check (row counts or checksums on a few critical tables, or confirming the most recent known transactions are present) rather than assuming success from the absence of alarms.

Worked example

A safe promotion policy might require replication lag to stay under 1 second for a sustained 5-minute window before promotion is allowed (a stated operational threshold for this runbook, not a universal rule that applies to every workload). The actual write-pause duration during cutover isn't a fixed number worth quoting as a general fact, since it depends on the application's connection pool behavior, not on the database resize itself: it's bounded by how long the app takes to drain in-flight writes and how quickly its clients reconnect to the new endpoint after the connection target flips, which is exactly why this pattern is described as minimizing downtime, not eliminating it. If the application's reconnect and retry logic is slow or missing, the same database-side runbook produces a much longer perceived outage even though the database steps themselves didn't change.

Trade-offs & pitfalls

• This is not truly zero-downtime: promoting a replica that isn't fully caught up loses whatever writes landed after its last applied transaction, so the safety of the whole procedure hinges on verifying lag is genuinely near zero at the moment of promotion, not assuming it.
• If the application cannot tolerate even a brief write-pause, this pattern isn't sufficient on its own; a true zero-downtime requirement needs a different approach, such as a proxy layer that queues writes during cutover.
• Before building a custom replica-promotion runbook, check whether the cloud provider's native "modify instance class" operation already performs an equivalent internal promote-and-swap; if it does, it may be simpler and better-tested than a hand-rolled version of the same idea.
• The replica-promotion mechanics here (catching up, promoting, cutting over) are a practical means to a scaling end; the deeper mechanics of replication modes and failover consensus are a related but distinct topic from the scaling procedure itself.