Infrastructure Scaling, Capacity Planning, and High Availability Questions

Growth assumptions to state up front

State the assumed growth curve explicitly. Going from 10K to 10M DAU (daily active users) over 3 to 5 years is roughly a 1000x multiplier, and that growth is unlikely to be linear; it is typically closer to exponential in the early years and slows as the product matures. Also state the assumed data volume per user and its own growth, since more DAUs often comes with more events per user too as the product adds features, so total data volume can grow faster than DAU alone.

Capacity inflection points

• Around 10x DAU, roughly 100K: usually still comfortably vertical or config-level scaling, but this is the point where the Kafka cluster's default partition count and consumer group sizing typically need a deliberate re-plan rather than default settings.
• Around 100x DAU, roughly 1M: the nearline processing layer, Spark or Flink, usually needs to move from a shared, general-purpose cluster to dedicated, workload-isolated capacity, since noisy-neighbor contention starts actually costing SLA (service-level-agreement) misses at this volume. This is also typically where hot and cold storage separation in the data lake starts paying for itself, since most queries touch recent data but storage cost is dominated by the growing cold tail.
• Around 1000x DAU, 10M: this is where structural changes become necessary rather than optional, sharded Kafka clusters and likely regionalization, both discussed below.

Structural changes and their triggers

• Sharded Kafka: the trigger is operational, not purely throughput. When a single cluster's broker count makes routine operations, rolling upgrades and rebalances, too slow or risky to do safely, shard by logical domain rather than round-robin, so a hot spot in one domain does not require rebalancing everything.
• Hot and cold storage separation: the trigger is when storage cost, not compute, becomes the dominant line item in the data lake bill, typically once retained history spans multiple years at meaningfully higher volume. Move cold data to cheaper storage tiers with a documented and tested rehydration path for the rare query that needs it.
• Regionalization: the trigger is either a latency requirement the growing user base's geography no longer lets a single region meet, or a single-region capacity ceiling, network or compute quota, that is cheaper to solve by adding a region than by continuing to scale one.

Staffing and budget implications

Each structural change above is also an operational-ownership change. Sharded Kafka needs a team that owns cross-shard operational tooling, not just per-shard administration; regionalization needs on-call coverage that either follows the sun or a documented single-region-first, multi-region-fallback ownership model. Budget the headcount growth curve alongside the infrastructure growth curve explicitly, since infrastructure scaling without a matching operations-capacity plan is how a growing platform ends up constrained on people rather than machines.

Risk mitigation

• Re-forecast on a fixed cadence, for example quarterly, against actual growth, since a 3-to-5-year roadmap built once and never revisited will be wrong by year two regardless of how good the initial assumptions were.
• Build the structural changes, sharding, hot and cold separation, regionalization, before being forced into them under incident pressure; each is meaningfully riskier to attempt for the first time during an active capacity crisis than as planned, tested work.
• Keep an explicit "what if growth is 2x faster than planned" contingency for the two or three inflection points closest to the current position, since those are the ones most likely to be hit soonest with the least lead time to react.

This is a behavioral question, so the answer below is a worked example of the kind of structured, specific story an interviewer is looking for. Use the STAR shape (Situation, Task, Action, Result) and swap in your own real experience with a comparable trade-off.

Situation

At a mid-size company running an ML platform, the infrastructure team operated a fixed GPU cluster used both for nightly model retraining jobs and for serving a real-time recommendation API with a 200ms p99 (the latency experienced by the slowest 1% of requests) latency SLO (service level objective, the internal performance target the team held itself to). Midway through a quarter, finance asked infrastructure spend to come down by roughly a third within a month, and the GPU cluster was the single largest line item.

Task

I owned finding a way to hit the cost target without breaking the recommendation API's latency SLO, and without simply asking for more budget, which had already been ruled out.

Action

I mapped out where the GPU hours were actually going: most of it was nightly retraining, a smaller but latency-critical slice was live inference. I modeled two options: delay the retraining cadence from nightly to every 3 days, freeing up the bulk of the GPU time immediately with a quantifiable, boundable quality cost (a model a few days staler), or reduce inference concurrency to shrink the serving fleet, which directly risked breaching the SLO during peak traffic. I built a small before/after estimate for each option showing dollar impact and the specific risk it introduced, then brought both options plus my recommendation (delay retraining, leave inference concurrency untouched) to the team lead and the product stakeholder who owned the recommendation feature, explaining the trade-off in terms they could evaluate: staleness versus latency risk, not just abstract cost numbers.

Result

The team agreed to the retraining-cadence change. It hit the cost target within the month, the recommendation API's SLO was never at risk since concurrency was untouched, and we set a follow-up check a few weeks later to confirm model quality had not measurably degraded from the staler retraining cadence, which it had not. The concrete artifact that made the decision easy to sign off on was the side-by-side cost/risk comparison rather than a single recommendation stated without alternatives.

How to adapt this to your own story: pick a real trade-off you owned, be specific about the numbers and the options you actually considered (not just the one you picked), name who you communicated it to and how, and be honest about the outcome, including anything you would do differently next time.

Baseline math (everything kept in requests per second, RPS)

Current: 50,000 RPS at 60% CPU
Capacity at 100% CPU (today's fleet) = 50,000 / 0.60 = 83,333 RPS

Utilization at 60 percent while serving 50,000 RPS tells us the fleet's real ceiling today is about 83,333 RPS if it ran flat out, which we never actually want to do.

Baseline in 6 months with 30% growth = 50,000 x 1.30 = 65,000 RPS
Capacity needed to hold 60% target utilization = 65,000 / 0.60 = 108,333 RPS

Holding the same 60 percent target utilization at the grown baseline needs about 108,333 RPS of capacity, roughly 30 percent more fleet than today, which makes sense: if the target utilization does not change, fleet size scales linearly with traffic.

Flash peak on top of the grown baseline = 65,000 x 5 = 325,000 RPS
Capacity to hold 60% utilization at that peak = 325,000 / 0.60 = 541,667 RPS

That is about 6.5x today's compute footprint (541,667 / 83,333). Holding that permanently, year-round, for an "occasional" spike is not economical, so the plan treats it as an elastic ceiling, not a standing fleet.

Horizontal vs vertical scaling

Horizontal (more stateless instances) is the primary lever: it scales close to linearly and improves fault tolerance. Vertical scaling (bigger instances) is reserved for anything that cannot be sharded easily, such as a single-writer database, and even there it is a stopgap since it has a hard ceiling and usually needs a resize/restart.

Autoscaling policy and buffer sizing

Run target-tracking autoscaling around the same 60 percent utilization target the fleet runs at today, with the maximum instance count set high enough to cover the 5x flash case (the 541,667 RPS figure above). If flash events are known in advance (an announced sale), add scheduled pre-warming ahead of the event, because reactive-only autoscaling cannot launch, boot, and warm 6.5x the fleet inside a sudden spike without some queueing and latency degradation first.

Peak concurrency modeling

Use Little's Law (a queueing-theory result: average concurrent requests in flight equals throughput multiplied by average request latency) to size not just instance counts but connection pools and downstream (database, cache) query rates at each of the three traffic levels above, since the DB or cache can become the real ceiling even when the app tier autoscales fine.

Validating with load testing

Run a ramp test to the 6-month baseline (65,000 RPS) to confirm steady-state headroom holds, and a separate spike test that ramps from baseline to the 325,000 RPS flash level over a realistic ramp time to confirm the autoscaler (or pre-warmed buffer) keeps p95 latency inside SLO during the spike, not just after it settles.

Presenting cost vs headroom to stakeholders

Frame it as a curve, not a single number: holding permanent capacity for the 5x case year-round costs roughly 6.5x today's compute bill for a scenario that only happens occasionally, versus an autoscaling-plus-pre-warm approach that costs close to the 30 percent growth-driven increase most of the time, with a stated, honestly communicated risk window (the minutes it takes to scale out) for any flash event that arrives without advance notice.

Direct answer

Reframe the buffer from an abstract cost line into the specific outage risk it prevents, quantify that risk in terms finance and sales already use (revenue at risk, service-level agreement penalty exposure, the cost of a past incident), and bring more than one option so the room negotiates a risk trade-off explicitly instead of just pushing back on a single number.

Structured elaboration

Present the trade-off in business terms: translate the buffer into "capacity to absorb a given traffic surge, or one region or availability-zone failure, without customer-visible degradation," and translate the leaner alternative into the specific failure modes it reintroduces, queueing or latency degradation at peak, a failed failover, a customer-facing incident, along with how often the historical data says that surge or failure actually happens.

Quantify risk, not just cost: pull recent incident history, a past outage's duration and its customer or revenue impact, as the anchor for what an under-provisioned system actually costs when it fails, set against the buffer's known, bounded monthly cost. This reframes the conversation as a small, certain cost versus a large, uncertain one, rather than a pure cost-reduction ask.

Bring options, not one number: present a leaner option (a smaller buffer, naming exactly which failure modes it stops covering), the recommended option (the original figure, tied to the specific service-level objective it protects), and if useful a richer option, so finance and sales are choosing a risk posture on a spectrum rather than approving or rejecting a single ask.

Align stakeholders: run this as a joint working session with sales, finance, and site reliability engineering (SRE) in the room together, not sequential one-on-ones where positions harden before anyone hears the trade-off. Let sales state the margin pressure and finance state the budget constraint explicitly, and let the group jointly own whichever option is picked, so a later incident reads as a shared, informed decision rather than "engineering's plan failed."

Close with a decision, not just information: propose a specific owner and a specific revisit date, for example re-evaluating the buffer size after the next two peak events with real data, so the conversation produces an action rather than just an airing of views.

Worked example

Suppose the recommended 20% buffer costs an extra $18,000/month over a no-buffer (0%) baseline, the cost each buffer tier adds on top of running with no headroom at all. A leaner option at 10% buffer saves $9,000/month but, based on the last two peak events, would have been breached once, and the most recent breach of a similar size cost about four hours of degraded checkout performance, worth naming a rough revenue-at-risk figure for if the business tracks conversion-per-hour at peak. Presenting it as three options, 10% ($9,000/month saved, one likely breach per year based on recent history), 20% (the recommended figure, no breaches in the same lookback window), and 30% (an extra $9,000/month beyond 20% for a service with a tighter uptime commitment), turns the conversation into picking a point on a named spectrum instead of defending or attacking one number.

Trade-offs and pitfalls

Leading with the technical justification, redundancy semantics, percentile math, before the business framing tends to lose the room; translate to dollars and risk first, and keep the technical detail in reserve for whoever wants to go deeper.

Presenting only the recommended number invites a binary yes-or-no fight; presenting a spectrum of named options turns it into a negotiation that can be partially won instead.

Do not agree to a leaner buffer without a documented, time-boxed re-evaluation plan; an unmonitored reduction is exactly the kind of decision that looks fine until the next surge exposes it.

Translating forecasts into procurement timing

Convert the capacity forecast's growth curve into a resource-acquisition timeline that accounts for lead time: for hardware, that is vendor manufacturing and shipping lead time, which can be weeks to many months depending on component availability; for cloud commitments, that is the commitment term you are willing to lock into. The procurement trigger point should be the forecast date minus the lead time, not the date you actually need the capacity, so you are never ordering after the need has already arrived.

Vendor selection criteria

• Lead time reliability, not just the quoted lead time, since a vendor that is usually two weeks late needs that variance buffered into your trigger point.
• Total cost at your actual committed volume, not list price, since large commitments usually unlock negotiated discounts.
• Exit flexibility: how painful it is to reduce or cancel a commitment if the forecast turns out wrong, since the whole exercise is forecasting under uncertainty.
• Support and SLA (service level agreement) terms appropriate to how critical the capacity is; a batch-analytics cluster and a customer-facing serving tier do not need the same support tier.

Contract types

• On-demand: no commitment, highest per-unit price. Use for the uncertain tail of the forecast, the part you are least confident will materialize, or for genuinely unpredictable bursty load.
• Reserved or committed-use: lower per-unit price in exchange for a term commitment, typically one to three years. Appropriate for the portion of the forecast you have high confidence is baseline, steady-state demand.
• A blended strategy, reserving the confident baseline and using on-demand or spot capacity (spot: spare cloud capacity sold at a discount that the provider can reclaim with little notice) for the uncertain growth margin, is usually the right default rather than an all-or-nothing choice, since it directly matches contract type to forecast confidence.

Risk mitigation

• Component lead times: build a buffer into the procurement trigger equal to the vendor's worst observed lead time, not their best-case quote, and revisit that buffer after every procurement cycle.
• Spot or reserved-instance shortages: do not rely on spot capacity or a specific instance type being available on demand during a demand spike, since everyone else may be trying to buy the same thing at the same time; treat spot as a cost optimization for flexible workloads, not as guaranteed capacity for anything on the critical path.
• Forecast uncertainty itself: present a range, low, base, and high growth scenarios, rather than a single number, so procurement and finance can see what they are committing against and what the downside of over- or under-committing looks like in each scenario.

Presenting the plan to procurement and finance

Lead with the business driver, expected growth and what happens if capacity is not there in time, before the technical detail, since that is what finance is actually evaluating. Show the cost difference between the recommended blended commitment strategy and the two extremes, all on-demand and all reserved for the high-growth scenario, so the trade-off is explicit rather than implied. Flag the re-forecast cadence, for example quarterly, up front, so procurement and finance expect the plan to be revised rather than treating it as a one-time, unchangeable commitment.