Scalability Patterns and Techniques Questions

Direct answer

Think of caching as putting copies of data closer to the people who want it, at progressively larger "stores": a content delivery network (CDN) keeps copies at edge locations near users worldwide, a regional cache keeps copies in a handful of data-center regions, and an application cache keeps hot data in memory right next to the servers that handle requests. For a photo app with 50 million daily active users and bursty traffic, the priority order is CDN first, regional cache second, application cache third, because most of the cost and latency risk comes from serving the same popular photos over and over to a global audience, and the CDN is what absorbs that at the lowest cost per request.

Why this order, in plain terms

• CDN (edge caching), highest priority. Photos are, once uploaded, mostly unchanging. Storing copies at CDN points of presence around the world means a user in another country gets the photo from a nearby server instead of round-tripping to wherever the app's servers actually run. This is what makes bursty traffic (a post suddenly going viral) survivable: the CDN absorbs the spike instead of it hitting the origin servers directly.
• Regional caches, second priority. These sit closer to the origin than the CDN, in each region where the app runs servers. They catch requests the CDN missed (a photo nobody has viewed recently in that area) and reduce how often a request has to cross regions to reach wherever the primary data lives, which matters for both speed and cost.
• Application cache, third priority. This is in-memory data held directly by the app servers, mainly for things that change more often or need to be assembled per-request, like a user's session or a feed's metadata (like counts, captions), rather than the photo bytes themselves.

Invalidation strategy and how much staleness to accept

• The photos themselves: treat as immutable. When a user replaces a photo, give the new version a new URL rather than overwriting the old one in place; that sidesteps invalidation entirely, since the old URL simply stops being referenced. Cache these for a long time.
• Thumbnails and resized versions: shorter cache lifetimes, or serve a slightly stale version while a fresh one is generated in the background, since users rarely notice a resize regenerating a few seconds late.
• Metadata (likes, comment counts): short cache lifetimes and update the cache on write, since this changes constantly and users do notice when a like count looks frozen.
• Deletions and privacy actions: the one case that should never be "eventually consistent." If a user deletes a photo, that removal should propagate immediately, not wait out a cache expiration, because a deleted-but-still-cached photo is a privacy and trust problem, not just a UX nitpick.

As a rule of thumb: the more a piece of data resembles "a fact that was published once," the longer a cache can hold it; the more it resembles "a live counter or a permission decision," the shorter that window needs to be.

A concrete walkthrough: one photo, start to finish. Say a user uploads photo.jpg at 2:00:00 PM; it's cached at the CDN edge with a long time-to-live (TTL, how long a cached value stays valid before it's considered expired) since photo bytes are treated as immutable. At 2:05:00 PM that same user deletes the photo. The delete triggers an immediate invalidation call that purges the object from the CDN, the regional cache, and any application-cache entry referencing it, rather than waiting for the normal TTL to lapse. By roughly 2:05:02 PM, all three layers have confirmed the purge; a friend who opens that user's profile at 2:05:03 PM sees no photo at all, instead of the deleted image loading one more time from a stale edge copy. Contrast that with the like-count metadata next to the same photo: if it uses a 5-second cache lifetime instead of immediate invalidation, a viewer might briefly see a like count that is a few seconds behind reality, an acceptable trade-off for that specific piece of data, unlike the deleted-photo case above.

Metrics to monitor

• Cache hit ratio at each layer (CDN, regional, application): the single best signal that the tiering is doing its job.
• Origin request rate: should stay low and flat even during traffic spikes if the CDN and regional caches are absorbing load correctly.
• Latency at the 95th and 99th percentile (the response time that 95% and 99% of requests beat), since averages hide the slow outliers users actually complain about.
• How quickly a deletion or update propagates through the cache layers, since that is the metric that catches a privacy-invalidation bug before a user does.

Trade-offs and pitfalls

The main trade-off is staleness versus cost and speed: caching for longer serves more traffic cheaply but risks showing outdated content, while caching for a shorter time keeps things fresher but pushes more load back to origin servers, which is expensive at 50 million daily users. The most common mistake in this design is treating every kind of data the same way, for example, applying one blanket cache duration to both the photo bytes (safe to cache for a long time) and the like counter next to it (which looks broken if it is stale for more than a few seconds). The second most common mistake is not treating deletions as a special, urgent case: a cache design that is otherwise well-tuned for performance can still create a real privacy incident if a deleted photo keeps serving from cache for its normal time-to-live.

Direct answer

Capacity planning for 10x growth isn't one calculation, it's figuring out what breaks first as load rises, buying enough lead time to fix each thing before it does, and having a plan for when reality outpaces the forecast. The most useful anchor number for a product manager or engineering manager to hold onto is the ratio of peak concurrent load to your total user base, because it turns an abstract user-growth target into a concrete load number engineering can actually plan against.

Structured elaboration

Telemetry to collect, and why each matters

• Business and usage signals: monthly active users (MAU, monthly active users), session length, requests per user, and which features drive the most usage. This tells you where growth is actually coming from, not just that it's happening.
• Traffic signals: peak concurrent requests, how bursty traffic is (a spike lasting seconds looks very different from sustained peak load), and geographic distribution. Bursty traffic needs headroom that average traffic doesn't.
• Performance signals: response time at the 95th percentile (P95, the value below which 95% of requests are faster; a better planning number than the average, because it reflects what your slower users actually experience) and error rates. These tell you where the system is already close to its limit today.
• Infrastructure signals: server utilization, database load, and how much of current capacity is autoscaled versus fixed. This tells engineering how much of a 10x jump can be absorbed by "turning a dial" versus requiring new design work.

Modeling the growth, in plain terms

Don't model user growth and system load as the same number; they're related but not identical. Build a small set of scenarios (conservative, expected, aggressive) for user growth, and separately translate each into a load number using your current ratio of peak load to user base as a starting anchor: if peak load has historically been roughly 10% of your monthly active user count, apply that same ratio to your growth target as a first estimate, while treating that ratio as something that can shift as the product evolves, not a law.

What kinds of architectural changes this usually forces

You don't need to design these yourself, but knowing the categories helps you ask the right questions of engineering and set a realistic timeline:

• Absorbing more load without touching the core system: a content delivery network (CDN, a network of servers positioned close to users that serves cached content) for anything that doesn't need to hit your servers on every request, and caching for data that's read far more often than it changes.
• Spreading read load: adding read replicas, additional copies of the database that can serve read traffic, so reads don't all compete with writes on one machine.
• Spreading write and storage load: sharding, splitting data across multiple databases, once a single database's capacity becomes the actual constraint rather than reads.
• Smoothing spikes: moving non-urgent work (sending a notification, generating a report) onto an asynchronous queue, so a traffic spike doesn't force every piece of work to happen synchronously in the request path.
  Each of these is a real engineering project with its own timeline, not a switch you flip; the earlier you know which ones the forecast requires, the more lead time engineering has.

Contingency plan if growth exceeds the forecast

• Short term (hours): pre-agreed emergency levers, like temporarily disabling a non-critical, expensive feature, or throttling lower-priority traffic, to protect the core product experience.
• Medium term (days to weeks): accelerate whichever planned change (more caching, more read capacity) is already closest to ready, rather than starting something new under pressure.
• Long term (months): treat sustained over-forecast growth as a signal to revisit the architecture itself, not just add more of the same capacity.
• Have this agreed with engineering and leadership before you need it: what a service-level objective (SLO, an internal target for how the system should perform, for example a target response time) breach looks like, who decides to pull an emergency lever, and what budget is pre-approved for burst capacity so that isn't a debate happening during the incident itself.

Worked example

Assume the numbers given: 1,000,000 monthly active users (MAU) and 100,000 peak concurrent requests today. The ratio of peak load to user base:

$\frac{100{,}000}{1{,}000{,}000} = 0.10$

If that same ratio holds at 10x user growth (10,000,000 MAU), the projected peak concurrency is:

$0.10 \times 10{,}000{,}000 = 1{,}000{,}000 \text{ peak concurrent requests}$

This is a planning assumption, not a guarantee: it treats the 10% ratio as constant, which holds only if usage patterns per user don't change materially as the product grows. If the product becomes stickier (longer sessions, more features used per visit as it matures), the ratio itself could rise, which is why it should be re-measured from real telemetry each quarter rather than locked in once at the start of the 12-month window.

Trade-offs & pitfalls

• Treating the user-growth number and the load number as interchangeable is the most common planning mistake; a 10x user target does not automatically mean 10x load if usage intensity per user changes at the same time.
• Setting architectural targets around average load rather than P95 or peak load under-provisions for exactly the moments (launches, marketing pushes, viral moments) capacity planning is meant to protect against.
• A contingency plan that exists only as a document, without pre-approved budget and a named decision-maker for pulling emergency levers, tends to fail exactly when it's needed, because the debate about whether to act happens during the incident instead of before it.
• Re-forecasting only once, at the start of the 12-month window, means the plan quietly goes stale; the ratio and the scenarios both need periodic revisiting against real telemetry.

Direct answer

As the engineering manager, the job is to run two tracks in parallel: protect customers with fast, reversible mitigations while the team runs a structured, evidence-driven investigation into why the tail is spiking even though the median looks fine. A stable median with a spiking P99 (99th percentile latency, the response time that only the slowest 1% of requests exceed) almost always points to something that affects a subset of requests intermittently, such as contention for a shared resource, garbage-collection pauses, cold caches, or a dependency that is occasionally slow, rather than a problem with the service's typical-case code path. My role is less about running the profiler myself and more about sequencing the investigation, keeping it evidence-based instead of guess-driven, and making the call on when to stop mitigating and start shipping a real fix.

The investigation, phase by phase

What the technical investigation actually tells you

A manager does not need to run these tools personally, but needs to know what question each one answers well enough to review the findings critically:

• Instrumentation and metrics: are p95 and p99 tracked as separate, alertable signals, not folded into an average? An average or median can look perfectly healthy while a small percentage of requests are badly affected; if only the average is monitored, this class of problem is invisible until customers complain.
• Distributed tracing: for one specific slow request, where did the time actually go, across every service and network hop it touched? This turns "the service is slow sometimes" into "this specific downstream call is slow on this specific request."
• Flamegraphs and profiling: within one process, during a slow window, which function or code path was actually consuming CPU (central processing unit, the compute resource that runs the code) or blocked? This is what distinguishes "the code is doing too much work" from "the code is waiting on something."
• Traffic correlation: does the spike line up with a traffic pattern (a burst, a specific client, a batch job, a particular hour) rather than being random? A correlated spike is a much smaller search space than a random one.
• Dependency analysis: is the tail coming from inside this service, or from something it calls (a database, cache, or another service)? This decides which team should even be investigating further.
• Hypothesis-driven experiments: once there is a specific suspected mechanism, can it be reproduced in a controlled setting (replayed traffic, a toggle that disables the suspected component) to confirm the theory before shipping a fix based on it?

For example, tracing plus dependency analysis might show that spikes cluster in a narrow, recurring window that coincides with a scheduled batch job saturating a connection pool (a fixed, reusable set of open database connections that requests share, since opening a brand-new connection for every request is slow) shared with the customer-facing path. Confirming that theory means reproducing the pattern under controlled load with and without the batch job running, not just noting the correlation and shipping a fix on faith.

Trade-offs and pitfalls

• Chasing root cause before stabilizing customer impact. A rollback or capacity bump that you don't fully understand yet is still the right first move if it demonstrably reduces customer pain; waiting for certainty before mitigating trades customer harm for tidiness.
• Treating a correlated pattern as a confirmed cause without the experiment step. Two things happening around the same time is a lead, not proof; shipping a fix based on correlation alone risks solving the wrong problem while the real cause keeps recurring.
• Setting a hard deadline for full resolution before the investigation phase is even done. Mitigation timelines (hours) and full architectural resolution timelines (weeks) are genuinely different kinds of commitments, and conflating them either creates false urgency on the durable fix or false calm about customer impact.
• Only tracking the average or median in the first place. If p99 is not already an alertable signal, the team finds out about tail-latency problems from customer complaints instead of from monitoring, which is itself worth fixing regardless of this specific incident's outcome.

Direct answer

As an engineering manager evaluating this design, the useful lens is: each caching layer trades cost, staleness, and operational complexity for latency, and the invalidation strategy and eviction policy paired with each layer follow directly from how far it sits from the origin and how quickly its data changes. Client-side and edge/content delivery network (CDN) caching are cheap and fast but coarse-grained; a reverse proxy and an in-memory service cache give finer control at the cost of infrastructure to run; a persistent caching tier and the database itself are the fallback of record. Getting this right is less about picking the "best" layer and more about not making every layer behave the same way.

Caching layers

Two terms in the table are worth spelling out plainly, since this question is aimed partly at a Technical Product Manager audience: an ETag is a version tag the client can check to see if its cached copy is still fresh, and a surrogate key is a label attached to cached content so many different URLs sharing that label can be purged together in one call, instead of purging URL by URL.

The persistent caching tier is worth calling out as distinct from the in-memory service cache above it: an in-memory cache is fast but starts empty after every restart or deploy, which means a deploy can itself cause a temporary spike in origin load as the cache refills. A persistent tier (a disk-backed cache, or an in-memory cache configured to snapshot and reload) avoids that cold-start cost at the price of slightly higher latency than pure in-memory and some added operational surface to manage.

Invalidation strategies

• Time-to-live (TTL): the cache entry simply expires after a fixed window. Simplest to reason about, and appropriate when some staleness is acceptable.
• Cache-aside (lazy loading): the application checks the cache first; on a miss, it reads from the database and writes the result into the cache. The most common pattern, since it only caches what's actually requested.
• Write-through: the cache is updated synchronously as part of every write, so reads are always consistent with the cache, at the cost of added write latency.
• Write-back: the write lands in the cache first and is flushed to the database later. This is faster for writes but risks data loss if the cache fails before the flush happens, so it needs a durability plan (like a write-ahead log) before it's safe to use.

Choose based on the consistency requirement of the data: user-facing counts and prices tolerate a short TTL; anything where "stale" means "wrong in a way a user or auditor would flag" needs write-through or event-driven invalidation instead.

Eviction policies

• LRU: a safe default, evicts whatever hasn't been used recently.
• LFU: better when a stable set of items stays hot over time, since it protects popular-but-recently-quiet items that LRU would wrongly evict.
• TTL-based / FIFO (first-in first-out): simple and predictable, useful less for memory pressure and more for enforcing a maximum staleness window.

Trade-offs and pitfalls

The recurring failure mode across teams is not choosing a bad individual layer, it's applying one policy uniformly across data with very different consistency needs, which either under-caches fast-moving data (wasting the performance benefit) or over-caches slow-moving data as if it were volatile (adding unneeded invalidation complexity). The second common gap is skipping the persistent tier and treating the in-memory cache as if it always stays warm, which understates the load spike a deploy or restart actually produces on the origin.

Direct answer

Anchor the plan on a per-instance capacity number you can actually benchmark, not a guess: measure current queries per second (QPS, queries per second) and the 95th-percentile (P95, the value below which 95% of observations fall) CPU and memory per instance, project the 3x traffic target onto that per-instance capacity to get a target instance count, and only then work out what that delta costs in infrastructure spend versus what it costs in engineering time and headcount. Those are two different questions: "how many more instances" is usually a budget and autoscaling-configuration decision, while "does the architecture even support that many instances cleanly" is the one that turns into a hiring conversation.

Structured elaboration

Inputs to gather

• Current peak QPS and its trend over recent months, not just a single snapshot.
• P95 CPU and memory utilization per instance at current peak load; P95 rather than average, because average hides the moments the system is actually under stress.
• A benchmarked (not assumed) maximum sustainable QPS per instance, measured under realistic load, not theoretical hardware limits.
• Current autoscaling configuration: minimum and maximum instance counts, and how long a new instance takes to become ready (cold-start time), since that affects how much buffer you need above the bare-minimum forecast.
• Recruiting lead time for the team, if the forecast implies new engineering work rather than just more of the same infrastructure.

Key calculation

$\text{required instances} = \left\lceil \frac{\text{peak QPS} \times \text{growth factor} \times (1 + \text{safety buffer})}{\text{QPS per instance}} \right\rceil$

Assume, as a planning input rather than a measured fact, a current peak QPS of 3,000, a benchmarked capacity of 150 QPS per instance, a 3x growth target, and a 20% safety buffer for headroom above the raw forecast:

$\left\lceil \frac{3{,}000 \times 3 \times 1.20}{150} \right\rceil = \left\lceil \frac{10{,}800}{150} \right\rceil = \left\lceil 72 \right\rceil = 72 \text{ instances}$

For comparison, today's instance count under the same 20% buffer:

$\left\lceil \frac{3{,}000 \times 1.20}{150} \right\rceil = \left\lceil 24 \right\rceil = 24 \text{ instances}$

Instance count scales linearly with traffic here (from 24 to 72, a 3x increase matching the 3x traffic target), because per-instance capacity was held constant. That linearity check is itself useful: if the projected instance count did not scale roughly with the traffic multiplier, it would signal that something other than raw compute, a shared dependency like a database connection ceiling, is the real constraint, not instance count.

Turning the forecast into decisions

If the forecast requires new engineering work, translate the estimated effort into a hiring ask against your team's actual recruiting lead time (commonly a few months for a senior engineer, a planning assumption you should validate against your own team's recent hiring, not a fixed constant) rather than assuming headcount can be added instantly once budget is approved.

Trade-offs & pitfalls

• The formula assumes per-instance capacity stays constant as load grows; if the bottleneck is actually a shared resource (a database, a single-instance cache, a rate-limited third-party API), adding instances past that point doesn't help and the linear projection will be wrong in a way the math alone won't reveal.
• Skipping the safety buffer and rounding down "to save cost" removes exactly the headroom meant to absorb the difference between a forecast and reality; a moderate buffer is worth its cost until you have data suggesting otherwise.
• Treating this as a one-time calculation rather than a recurring check misses the point: re-run it with fresh telemetry each quarter, because both the QPS-per-instance benchmark and the growth trend can shift as the product and traffic mix change.
• Converting a capacity gap directly into a headcount number without first checking whether it's actually an autoscaling or budget problem leads to over-hiring for what could have been solved by turning a dial.