Scalability Patterns and Techniques Questions
Scaling a system to handle growth in traffic and data: horizontal versus vertical scaling, statelessness, sharding and partitioning strategies, read replicas, and connection pooling. Covers capacity estimation, identifying bottlenecks, and the tradeoffs each scaling axis introduces. The general toolkit for taking a design from thousands to millions of users.
Define edge caching and origin caching in plain terms for a cross-functional audience. For a photo-sharing app with 50 million daily active users and highly bursty traffic, which caching layers would you prioritize: CDN, regional caches, or application cache, and why? Briefly describe your invalidation strategy, how much staleness you'd accept, and the performance metrics you'd monitor.
Sample Answer
Direct answer
Think of caching as putting copies of data closer to the people who want it, at progressively larger "stores": a content delivery network (CDN) keeps copies at edge locations near users worldwide, a regional cache keeps copies in a handful of data-center regions, and an application cache keeps hot data in memory right next to the servers that handle requests. For a photo app with 50 million daily active users and bursty traffic, the priority order is CDN first, regional cache second, application cache third, because most of the cost and latency risk comes from serving the same popular photos over and over to a global audience, and the CDN is what absorbs that at the lowest cost per request.
Why this order, in plain terms
- CDN (edge caching), highest priority. Photos are, once uploaded, mostly unchanging. Storing copies at CDN points of presence around the world means a user in another country gets the photo from a nearby server instead of round-tripping to wherever the app's servers actually run. This is what makes bursty traffic (a post suddenly going viral) survivable: the CDN absorbs the spike instead of it hitting the origin servers directly.
- Regional caches, second priority. These sit closer to the origin than the CDN, in each region where the app runs servers. They catch requests the CDN missed (a photo nobody has viewed recently in that area) and reduce how often a request has to cross regions to reach wherever the primary data lives, which matters for both speed and cost.
- Application cache, third priority. This is in-memory data held directly by the app servers, mainly for things that change more often or need to be assembled per-request, like a user's session or a feed's metadata (like counts, captions), rather than the photo bytes themselves.
Invalidation strategy and how much staleness to accept
- The photos themselves: treat as immutable. When a user replaces a photo, give the new version a new URL rather than overwriting the old one in place; that sidesteps invalidation entirely, since the old URL simply stops being referenced. Cache these for a long time.
- Thumbnails and resized versions: shorter cache lifetimes, or serve a slightly stale version while a fresh one is generated in the background, since users rarely notice a resize regenerating a few seconds late.
- Metadata (likes, comment counts): short cache lifetimes and update the cache on write, since this changes constantly and users do notice when a like count looks frozen.
- Deletions and privacy actions: the one case that should never be "eventually consistent." If a user deletes a photo, that removal should propagate immediately, not wait out a cache expiration, because a deleted-but-still-cached photo is a privacy and trust problem, not just a UX nitpick.
As a rule of thumb: the more a piece of data resembles "a fact that was published once," the longer a cache can hold it; the more it resembles "a live counter or a permission decision," the shorter that window needs to be.
A concrete walkthrough: one photo, start to finish. Say a user uploads photo.jpg at 2:00:00 PM; it's cached at the CDN edge with a long time-to-live (TTL, how long a cached value stays valid before it's considered expired) since photo bytes are treated as immutable. At 2:05:00 PM that same user deletes the photo. The delete triggers an immediate invalidation call that purges the object from the CDN, the regional cache, and any application-cache entry referencing it, rather than waiting for the normal TTL to lapse. By roughly 2:05:02 PM, all three layers have confirmed the purge; a friend who opens that user's profile at 2:05:03 PM sees no photo at all, instead of the deleted image loading one more time from a stale edge copy. Contrast that with the like-count metadata next to the same photo: if it uses a 5-second cache lifetime instead of immediate invalidation, a viewer might briefly see a like count that is a few seconds behind reality, an acceptable trade-off for that specific piece of data, unlike the deleted-photo case above.
Metrics to monitor
- Cache hit ratio at each layer (CDN, regional, application): the single best signal that the tiering is doing its job.
- Origin request rate: should stay low and flat even during traffic spikes if the CDN and regional caches are absorbing load correctly.
- Latency at the 95th and 99th percentile (the response time that 95% and 99% of requests beat), since averages hide the slow outliers users actually complain about.
- How quickly a deletion or update propagates through the cache layers, since that is the metric that catches a privacy-invalidation bug before a user does.
Trade-offs and pitfalls
The main trade-off is staleness versus cost and speed: caching for longer serves more traffic cheaply but risks showing outdated content, while caching for a shorter time keeps things fresher but pushes more load back to origin servers, which is expensive at 50 million daily users. The most common mistake in this design is treating every kind of data the same way, for example, applying one blanket cache duration to both the photo bytes (safe to cache for a long time) and the like counter next to it (which looks broken if it is stale for more than a few seconds). The second most common mistake is not treating deletions as a special, urgent case: a cache design that is otherwise well-tuned for performance can still create a real privacy incident if a deleted photo keeps serving from cache for its normal time-to-live.
A service uses in-memory caches to reduce database load, but it's experiencing cache thrashing during traffic spikes. As the engineering manager reviewing this with your team, what are the likely causes, and what fixes would you expect the team to propose? Consider eviction-policy tuning, tiered caching (edge, regional, origin), request coalescing, warm caches, and circuit breakers. How would you quantify the expected reduction in database load for a given improvement in hit rate?
Sample Answer
Direct answer
As the engineering manager reviewing this with the team, cache thrashing during a traffic spike almost always traces back to one of a few root causes: the cache is too small or the time-to-live (TTL) too short for the actual working set, the eviction policy is discarding items that are still hot, the cache starts cold right when the spike hits, or a thundering herd of requests floods the origin the moment a popular key expires. The fixes the team should propose map one-to-one onto those causes, and the value of any fix should be provable in terms of database load reduction before it's called done, not just "latency felt better."
Likely causes and the fixes to expect
- Cache too small for the working set, or eviction discarding hot-but-sparse keys: least-recently-used (LRU) eviction can wrongly evict an item that is popular but was not accessed in the last few seconds, in favor of something accessed once and never again. Tuning toward least-frequently-used (LFU) or a hybrid, or simply increasing cache size to match the real working set, addresses this directly. Adding admission filtering (only caching a key after it's been requested more than once) avoids wasting cache space on one-off keys in the first place.
- Tiered caching gaps: a single cache tier means every miss goes straight to the database. Edge, regional, and origin-level tiers each reduce origin hits multiplicatively; a gap in the tiering (for example, no regional cache) means the origin absorbs load that a middle tier should have caught.
- Request coalescing missing: without it, a popular key expiring under load produces one database query per concurrent reader instead of one query total. This is the specific mechanism behind a thundering herd, and it is usually the single highest-leverage fix when thrashing coincides with traffic spikes rather than steady-state load.
- Cold caches at spike time: if the cache was never warmed (after a deploy, a restart, or an autoscaling event added new instances with empty local caches), the first wave of spike traffic pays full origin cost. Predictive or scheduled pre-warming for known hot keys addresses this.
- No backpressure on the database itself: backpressure means signaling an overloaded downstream component, here the database, to slow down or reject new work instead of letting requests queue up indefinitely; without a circuit breaker (a guard that tracks recent failures and temporarily stops sending calls to a struggling dependency once a threshold is crossed, so it fails fast instead of piling on) or a hard cap on concurrent database calls, a wave of cache misses can take the database down entirely rather than just running slow, turning a performance problem into an outage.
Quantifying the expected database load reduction
Database load is, to a first approximation, proportional to the cache miss rate, so it scales with (1−hit rate). If a fix raises the hit rate from h1 to h2, the resulting database load relative to before is:
DB loadoldDB loadnew=1−h11−h2As an illustrative example (not a claim about this specific service, these are example inputs to show the method): if the team's current hit rate is h1=70% and the proposed fixes are projected to raise it to h2=90%:
1−0.701−0.90=0.300.10=0.333That means database load after the fix is expected to be about 33.3% of what it was before, a roughly 66.7% reduction. The team should measure the actual before/after hit rate in production (or in a load test with realistic traffic replay) and plug the real numbers into this same ratio, rather than trusting a projected hit-rate improvement without confirming it landed.
As the reviewing manager, what to ask for
- Instrumentation before the fix ships: per-key access distribution, miss-storm rate (misses per second, not just overall hit rate), and cache occupancy, so the team has a baseline to compare against.
- A pilot on one service or one traffic shard rather than a global rollout, so the hit-rate improvement (and any regression) shows up in a comparison the team can actually read.
- A specific number to report back: measured database queries per second before and after, converted into the same hit-rate ratio shown above, not just "it feels faster."
Trade-offs and pitfalls
The trap in this kind of review is accepting "we improved the eviction policy" as sufficient without a before/after measurement, since eviction tuning, request coalescing, and cache warming can each look individually reasonable while still leaving the actual bottleneck (often the missing coalescing) unaddressed. A second trap is prioritizing the fix that's easiest to implement (often eviction-policy tuning) over the fix with the largest leverage during a spike specifically (usually request coalescing, since it caps the herd effect directly rather than just improving steady-state hit rate).
You are a Technical Product Manager for a cloud developer platform. Define horizontal scaling versus vertical scaling in concrete terms, then give two product scenarios (one favoring horizontal, one favoring vertical) and explain the trade-offs in cost, downtime risk, operational complexity, observability, and developer experience. How would you influence engineering's choice, and what metrics would you monitor to validate it?
Sample Answer
Direct answer
Horizontal scaling means running more copies of a service side by side (more instances behind a load balancer) so the same work is split across a wider set of machines. Vertical scaling means making one existing machine bigger (more CPU, memory, or disk on the same box). As a technical product manager, the question I'd push engineering on isn't "which is better" in the abstract; it's "which one fits this specific service's constraints right now," because the two options carry very different cost, risk, and speed-to-ship trade-offs.
Structured elaboration
Definitions, concretely:
- Horizontal scaling: going from 1 app instance to 5 instances, each handling a fifth of the traffic, coordinated by a load balancer.
- Vertical scaling: taking that same single instance and moving it to a larger machine, for example doubling its CPU and memory.
Scenario favoring horizontal: a multi-tenant API serving many short, independent requests.
- Cost: higher baseline (more machines running), but better cost efficiency per request once traffic is high and steady.
- Downtime risk: lower; instances can be replaced one at a time without taking the whole service down.
- Operational complexity: higher upfront; needs load balancing and service discovery in place, which is infrastructure work, not a business decision by itself.
- Observability: needs request-level and fleet-level visibility (how is load distributed across instances), not just one machine's health.
- Developer experience: scaling is "add another instance," which is fast to execute once the infrastructure exists, but the service has to be stateless first (see below).
Scenario favoring vertical: a legacy or stateful component that can't easily be split, such as a single-process cache or an analytics-ingest service holding state in memory that isn't designed to be split across machines.
- Cost: a bigger machine has worse cost-per-unit-capacity at the high end, but it's often the fastest way to buy headroom without an engineering rewrite.
- Downtime risk: higher; resizing frequently requires a restart, and there's a single point of failure the whole time.
- Operational complexity: lower day-to-day (one machine to watch), but scaling further is capped by the largest machine available and harder to automate safely.
- Observability: narrower, focused on that one machine's CPU, memory, and health.
- Developer experience: no code changes required, which is attractive under deadline pressure, but it's a deferral of the real fix, not a substitute for it.
Signals that should trigger a move from vertical to horizontal, even if the team's instinct is to keep resizing:
- You're already near the largest machine size available, or the next size up costs disproportionately more for a shrinking capacity gain, so vertical simply runs out of room as a lever.
- A resize requires downtime, and that downtime window is now colliding with real user traffic instead of fitting inside a quiet maintenance period, meaning the "safe" vertical option has stopped being safe.
- Growth has become spiky rather than steady. A single bigger machine can absorb a slow, predictable climb, but it can't add capacity fast enough for short-lived spikes the way a set of instances that scale out and back in can.
User-visible impact during the transition. Moving from one big machine to several smaller ones is not free for users if the service was holding state in memory (a logged-in session, an in-progress upload). Unless that state is externalized to a shared store first, users can be logged out or lose in-progress work mid-cutover. There is also typically a short window of uneven response times while new instances warm up behind the load balancer, before their health checks stabilize.
How I'd influence engineering's choice:
- Translate the business need into concrete decision criteria: expected request volume, response-time targets, cost ceiling, and how soon this needs to ship.
- Ask directly whether the service is stateless (safe to run many identical copies) or stateful (holds data on one machine that would need to move first); this single question usually decides which path is realistic, more than a general cost debate does.
- Propose starting with the cheapest safe option (often vertical, if there's headroom left) while scoping the refactor that horizontal scaling requires, rather than treating it as an all-or-nothing choice.
- Get explicit agreement on a timeline: at what point does the team commit to the horizontal path even if vertical is still technically an option, so the decision doesn't get re-litigated every time a resize buys another few months.
Worked example
A concrete story: a developer-platform API starts on a single, reasonably large instance. Over several months, traffic grows steadily and the team resizes the instance twice, each time buying a few months of headroom with a short maintenance-window restart. On the third approaching resize, the team discovers they're already near the largest instance size the cloud provider offers for that machine family, and the next tier up costs far more for a proportionally smaller capacity increase. That's the vertical-headroom-exhausted signal firing. At the same time, product has just launched a feature that drives short, unpredictable traffic spikes around specific events rather than steady growth, which is the spiky-growth signal. Together, these push the team to invest in making the service stateless (moving session data out of the process and into a shared store) so it can run behind a load balancer as multiple instances, even though that refactor takes real engineering time the earlier vertical resizes didn't.
Trade-offs & pitfalls
- Horizontal scaling isn't free just because it's more "modern." It requires the service to be stateless first; skipping that step and scaling horizontally anyway produces inconsistent behavior (a user's session data only living on one of several instances) that is worse than staying vertical until the refactor is actually done.
- A string of "just one more vertical resize" decisions can quietly become the expensive path, if nobody is tracking how close the team is to the largest available machine size.
- The transition itself has a user-visible cost that's easy to leave out of the plan. Budgeting the refactor without budgeting for the state-externalization work, or without warning users about a rockier-than-usual cutover window, turns a well-reasoned architecture decision into a rough surprise for customers.
- The right metrics for validating this decision are the same ones that should have driven it: response-time percentiles (the response time under which a given percentage of requests complete), utilization per instance, cost per request, and how often scaling events happen; a decision that isn't being watched with these after the fact is a guess, not a validated choice.
A service has a stable median latency, but production telemetry shows periodic P99 spikes that are generating customer complaints. As the engineering manager, walk through the investigation you'd run: instrumentation, tracing, flamegraphs or profiling, traffic correlation, dependency analysis, and experiments. What temporary mitigations would you put in place to protect customers while you dig in, and roughly how long would you expect mitigation versus full resolution to take?
Sample Answer
Direct answer
As the engineering manager, the job is to run two tracks in parallel: protect customers with fast, reversible mitigations while the team runs a structured, evidence-driven investigation into why the tail is spiking even though the median looks fine. A stable median with a spiking P99 (99th percentile latency, the response time that only the slowest 1% of requests exceed) almost always points to something that affects a subset of requests intermittently, such as contention for a shared resource, garbage-collection pauses, cold caches, or a dependency that is occasionally slow, rather than a problem with the service's typical-case code path. My role is less about running the profiler myself and more about sequencing the investigation, keeping it evidence-based instead of guess-driven, and making the call on when to stop mitigating and start shipping a real fix.
The investigation, phase by phase
| Phase | Timebox | What happens | The EM's (engineering manager's) role |
|---|---|---|---|
| Immediate protection | 0-4 hours | Reduce customer-visible pain without knowing the root cause yet: check whether a recent deploy or config change lines up with when spikes started and roll it back if so; give the affected service temporary extra capacity headroom; if a specific low-value traffic pattern (a batch job, a specific client) correlates with spikes, throttle or reschedule it | Ask "what changed recently" first, authorize the rollback or capacity bump, and set expectations with stakeholders that this reduces pain, it does not explain the cause |
| Fast triage | 0-8 hours, can overlap with the above | Correlate the timing of spikes against deploys, traffic volume, time of day, region, and specific endpoints or customers, using existing dashboards | Ask for a timeline overlay (spikes vs. deploys vs. traffic) before anyone opens a profiler; this alone often narrows the search a lot |
| Deep investigation | 1-3 days | Distributed tracing, profiling, and dependency analysis (details below) to find the actual mechanism | Understand what each technique tells you well enough to ask sharp questions and sanity-check conclusions, without doing the tracing yourself |
| Temporary code or config fix | 1-7 days | A targeted change addressing the confirmed mechanism: fixing a slow query path, resizing a connection pool, adding backpressure to a hot path | Review that the fix targets the confirmed cause, not just the first plausible theory |
| Durable resolution | 2-8 weeks | Architectural follow-up (isolating a noisy workload, redesigning a hot path, adding permanent tail-latency monitoring) plus a written postmortem | Sponsor the follow-up work against competing roadmap priorities, since tail-latency fixes rarely feel urgent once the immediate pain is gone |
What the technical investigation actually tells you
A manager does not need to run these tools personally, but needs to know what question each one answers well enough to review the findings critically:
- Instrumentation and metrics: are p95 and p99 tracked as separate, alertable signals, not folded into an average? An average or median can look perfectly healthy while a small percentage of requests are badly affected; if only the average is monitored, this class of problem is invisible until customers complain.
- Distributed tracing: for one specific slow request, where did the time actually go, across every service and network hop it touched? This turns "the service is slow sometimes" into "this specific downstream call is slow on this specific request."
- Flamegraphs and profiling: within one process, during a slow window, which function or code path was actually consuming CPU (central processing unit, the compute resource that runs the code) or blocked? This is what distinguishes "the code is doing too much work" from "the code is waiting on something."
- Traffic correlation: does the spike line up with a traffic pattern (a burst, a specific client, a batch job, a particular hour) rather than being random? A correlated spike is a much smaller search space than a random one.
- Dependency analysis: is the tail coming from inside this service, or from something it calls (a database, cache, or another service)? This decides which team should even be investigating further.
- Hypothesis-driven experiments: once there is a specific suspected mechanism, can it be reproduced in a controlled setting (replayed traffic, a toggle that disables the suspected component) to confirm the theory before shipping a fix based on it?
For example, tracing plus dependency analysis might show that spikes cluster in a narrow, recurring window that coincides with a scheduled batch job saturating a connection pool (a fixed, reusable set of open database connections that requests share, since opening a brand-new connection for every request is slow) shared with the customer-facing path. Confirming that theory means reproducing the pattern under controlled load with and without the batch job running, not just noting the correlation and shipping a fix on faith.
Trade-offs and pitfalls
- Chasing root cause before stabilizing customer impact. A rollback or capacity bump that you don't fully understand yet is still the right first move if it demonstrably reduces customer pain; waiting for certainty before mitigating trades customer harm for tidiness.
- Treating a correlated pattern as a confirmed cause without the experiment step. Two things happening around the same time is a lead, not proof; shipping a fix based on correlation alone risks solving the wrong problem while the real cause keeps recurring.
- Setting a hard deadline for full resolution before the investigation phase is even done. Mitigation timelines (hours) and full architectural resolution timelines (weeks) are genuinely different kinds of commitments, and conflating them either creates false urgency on the durable fix or false calm about customer impact.
- Only tracking the average or median in the first place. If p99 is not already an alertable signal, the team finds out about tail-latency problems from customer complaints instead of from monitoring, which is itself worth fixing regardless of this specific incident's outcome.
As an engineering manager, describe a simple capacity-planning approach for a service expected to grow 3x in traffic over the next 12 months. What inputs would you gather, such as current QPS and P95 CPU/memory per instance? Walk through the key calculations for forecasting instance or shard counts, and how you'd turn that forecast into hiring, infrastructure, or autoscaling decisions.
Sample Answer
Direct answer
Anchor the plan on a per-instance capacity number you can actually benchmark, not a guess: measure current queries per second (QPS, queries per second) and the 95th-percentile (P95, the value below which 95% of observations fall) CPU and memory per instance, project the 3x traffic target onto that per-instance capacity to get a target instance count, and only then work out what that delta costs in infrastructure spend versus what it costs in engineering time and headcount. Those are two different questions: "how many more instances" is usually a budget and autoscaling-configuration decision, while "does the architecture even support that many instances cleanly" is the one that turns into a hiring conversation.
Structured elaboration
Inputs to gather
- Current peak QPS and its trend over recent months, not just a single snapshot.
- P95 CPU and memory utilization per instance at current peak load; P95 rather than average, because average hides the moments the system is actually under stress.
- A benchmarked (not assumed) maximum sustainable QPS per instance, measured under realistic load, not theoretical hardware limits.
- Current autoscaling configuration: minimum and maximum instance counts, and how long a new instance takes to become ready (cold-start time), since that affects how much buffer you need above the bare-minimum forecast.
- Recruiting lead time for the team, if the forecast implies new engineering work rather than just more of the same infrastructure.
Key calculation
required instances=⌈QPS per instancepeak QPS×growth factor×(1+safety buffer)⌉
Assume, as a planning input rather than a measured fact, a current peak QPS of 3,000, a benchmarked capacity of 150 QPS per instance, a 3x growth target, and a 20% safety buffer for headroom above the raw forecast:
⌈1503,000×3×1.20⌉=⌈15010,800⌉=⌈72⌉=72 instances
For comparison, today's instance count under the same 20% buffer:
⌈1503,000×1.20⌉=⌈24⌉=24 instances
Instance count scales linearly with traffic here (from 24 to 72, a 3x increase matching the 3x traffic target), because per-instance capacity was held constant. That linearity check is itself useful: if the projected instance count did not scale roughly with the traffic multiplier, it would signal that something other than raw compute, a shared dependency like a database connection ceiling, is the real constraint, not instance count.
Turning the forecast into decisions
| Lever | What it addresses | When it's the right call |
|---|---|---|
| Autoscaling configuration | Routine, gradual demand within the existing architecture | The projected instance count fits comfortably within what the current design already tolerates; mostly a cost and configuration conversation |
| Infrastructure spend | Buying more of what you already run | The 72-instance target is a straightforward extension of the current stateless, horizontally-scaled design |
| New engineering work (headcount) | A structural limit the current design won't clear, for example a shared database that can't take 3x the connections, or a single component that isn't horizontally scalable | Profiling shows the bottleneck isn't instance count but a shared dependency; this needs a project (sharding, a caching layer, async processing) and a timeline, not just more servers |
If the forecast requires new engineering work, translate the estimated effort into a hiring ask against your team's actual recruiting lead time (commonly a few months for a senior engineer, a planning assumption you should validate against your own team's recent hiring, not a fixed constant) rather than assuming headcount can be added instantly once budget is approved.
Trade-offs & pitfalls
- The formula assumes per-instance capacity stays constant as load grows; if the bottleneck is actually a shared resource (a database, a single-instance cache, a rate-limited third-party API), adding instances past that point doesn't help and the linear projection will be wrong in a way the math alone won't reveal.
- Skipping the safety buffer and rounding down "to save cost" removes exactly the headroom meant to absorb the difference between a forecast and reality; a moderate buffer is worth its cost until you have data suggesting otherwise.
- Treating this as a one-time calculation rather than a recurring check misses the point: re-run it with fresh telemetry each quarter, because both the QPS-per-instance benchmark and the growth trend can shift as the product and traffic mix change.
- Converting a capacity gap directly into a headcount number without first checking whether it's actually an autoscaling or budget problem leads to over-hiring for what could have been solved by turning a dial.
Unlock Full Question Bank
Get access to all 9 Scalability Patterns and Techniques interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.