Scalability Patterns and Techniques Questions

Direct answer

Horizontal partitioning (sharding) splits rows across nodes to scale storage and throughput; vertical partitioning splits columns or tables by function to isolate workloads and shrink row size. They solve different problems and are often used together. Choosing a shard key well up front matters more than almost anything else about a sharded system, because a bad key is expensive to discover late and disruptive to fix.

Structured elaboration

Shard-key selection criteria

• Evenness: the key should distribute load uniformly across shards; a skewed key concentrates traffic on a subset of shards regardless of how many shards exist.
• Query-pattern alignment: colocate data that's typically accessed or joined together on the same shard, since cross-shard joins are expensive or impossible.
• High cardinality: a key with few distinct values (like a boolean flag) can't spread load across many shards no matter how it's hashed.
• Stability: a key that rarely or never changes for a given row avoids the operational cost of moving a row between shards when its key value changes.

Planning and executing a reshard

Prefer consistent hashing (with virtual nodes, so each physical shard owns many small hash ranges rather than one large contiguous range) over naive modulo-based hashing, because of how differently the two schemes behave when the shard count changes. Under modulo hashing (hash(key) mod N), changing N from 4 to 5 shards invalidates the mapping for nearly every key, since very few keys land on the same shard under both divisors. Under consistent hashing, adding the 5th shard to a 4-shard ring moves only the keys that fall between the new shard's position and its nearest predecessor:

versus, for the naive scheme:

That gap (roughly a fifth of keys moving instead of roughly four-fifths) is the practical reason resharding plans lean on consistent hashing: the migration is proportional to the capacity added, not to the total dataset size.

The migration itself, in a cloud environment, typically runs online: stream ongoing changes from old to new shard layout (change data capture, CDC, capturing row-level changes as an ordered feed) while backfilling historical data in the background, validate the new layout against the old with checksums or row counts, then cut reads and writes over, ideally behind a feature flag or routing layer that can revert quickly if validation fails.

Operational challenges

• Rebalancing: moving data between shards is I/O- and network-intensive; throttle the migration and run it in the background rather than as a blocking step, and schedule any unavoidable cutover for low-traffic periods.
• Hotspots: even a well-chosen key can develop a hot shard as traffic patterns shift; mitigate with key-splitting (subdividing an overloaded key's range) or a cache layer absorbing read pressure in front of the hot shard.
• Migration windows: aim for a fully online migration so there's no hard window at all; if some downtime is unavoidable, keep it as short and as clearly scoped as possible, and communicate it like any other planned maintenance.

Worked example

A user-activity table sharded by user_id using consistent hashing across 4 shards needs to add a 5th shard as write volume grows. Using the calculation above, roughly 20% of keys need to move to the new shard (versus roughly 80% under naive modulo hashing), which is the concrete reason the migration is scoped as "move a fifth of the data," not "re-shuffle nearly everything." The team streams changes to the affected key ranges via CDC while backfilling their historical data, validates row counts on both old and new shard for the migrated ranges, then flips routing for those keys to the new shard, all without a hard maintenance window.

Trade-offs & pitfalls

• A monotonically increasing key (an auto-incrementing ID, or a timestamp) is a classic hotspot trap: every new row lands on whichever shard currently owns the "latest" range, so all write traffic concentrates on one shard no matter how many shards exist.
• Horizontal partitioning enables cross-node scale but makes cross-shard joins and multi-row transactions expensive or unavailable; that cost is inherent to sharding, not a bug to be engineered away.
• Vertical partitioning helps with row size and workload isolation but does nothing for a table whose row count is the actual bottleneck; picking the wrong partitioning axis for the actual problem wastes the migration effort.
• Consistent hashing reduces data movement on resize but doesn't guarantee even load by itself; a skewed key distribution can still produce uneven shard load even with a well-behaved hashing scheme.

Direct answer

The likely cause is a gap between when the database write commits and when the cache reflects it, combined with concurrent writers racing each other: one request's update gets silently overwritten by another (a lost update), and a reader in between sees data that matches neither writer's final state (a read-after-write violation surfaced through the cache rather than the database). The fix is to remove the ambiguity about ordering, either by making the cache update part of the same atomic step as the write, or by making concurrent writes detect and reject conflicting the ones based on stale reads, and to roll it out behind instrumentation and a staged rollout so a repeat of the corruption is caught before it reaches every order.

Structured elaboration

How the race actually happens

Two request paths that both read-modify-write the same order, with an asynchronous cache refresh in between, create two related but distinct failure patterns:

• Lost update: Request A reads the order, computes a new value based on that read, and writes it. Request B does the same concurrently, based on the same stale read. Whichever writes last overwrites the other's change, and the DBMS has no way to know a conflict occurred because neither write technically failed.
• Read-after-write (surfaced via the cache): even if the database itself resolves writes correctly, the cache refresh that's supposed to reflect a new write can fire out of order relative to a concurrent write, or can itself be built from a stale read, leaving the cache holding a value that never actually existed as a single consistent database state.

Fix options, with the trade-off each one makes

• Write-through caching: update the cache synchronously as part of the same operation that writes the database, instead of via a separate asynchronous invalidation step. This closes the window between "database write commits" and "cache reflects it" entirely, at the cost of higher write latency and the need to handle a cache write failure without silently diverging from the database.
• Versioned objects with optimistic compare-and-swap: attach a version (or timestamp) to the order, and require a write to specify the version it read; the write only succeeds if the version still matches, otherwise it's rejected and the caller must re-read and retry. This directly prevents lost updates without taking a lock, but it pushes retry logic onto every caller and adds a field to reason about.
• Single-flight cache refresh: ensure only one in-flight refresh per key at a time (a mutex or an in-process/distributed single-flight guard), so concurrent cache misses for the same order don't trigger redundant, possibly conflicting reads and writes back to the cache. This prevents cache stampede and duplicate refresh work, but needs a coordination mechanism that works across all instances if the service runs on more than one.
• Append-only event store: instead of overwriting order state in place, store each change as an immutable event and derive current state by replaying (or materializing) the event stream. This removes the overwrite race entirely, since nothing is ever overwritten, but it's a genuine architecture change: existing read paths, the cache itself, and any code that assumes "the order row" as the source of truth all need to be rebuilt around the event stream.

Safe rollout plan

1. Add instrumentation before changing anything: log version numbers and cache-refresh timing so you can see the race happening in production traces (application performance monitoring, APM, tooling is the natural place for this), rather than inferring it from corrupted orders after the fact.
2. Reproduce the failure deliberately in staging with concurrent writers hitting the same order, to confirm the fix actually closes the window rather than just narrowing it.
3. Start with the least invasive fix that addresses the immediate cause, typically single-flight cache refresh, since it requires no data-model change and directly stops the concurrent-refresh part of the problem.
4. Introduce optimistic versioning on the write path behind a shadow mode (write the version check's would-be result to logs without enforcing it yet) to confirm it wouldn't reject legitimate traffic before making it authoritative.
5. Roll out write-through caching or the versioning enforcement behind a canary on a small slice of order traffic, watching for rejected-write rate and cache-DB divergence checks, with a fast rollback (a feature flag reverting to the old refresh path).
6. Treat the append-only event store as a separate, longer-term project scoped to one bounded context first, not part of the immediate corruption fix.

Worked example

A concrete interleaving that produces both failure patterns in one incident, starting from order O123 with qty = 2, version = 5, cached:

At t3, any reader hitting the cache sees qty=2 even though the database already committed qty=3, a read-after-write violation. By t4, Request A's quantity update is gone entirely, overwritten by Request B's write, which was computed from a snapshot that predates A's change: a lost update. Neither the database's final state (status=cancelled, qty=2) nor the cache's intermediate state at t3 (qty=2, v5) reflects what actually happened; both writers "succeeded," but the system lost information a version check would have caught at t4 (Request B's write would have been rejected because its assumed version, 5, no longer matched the current version, 6).

Trade-offs & pitfalls

• Fixing only the cache side (single-flight refresh, write-through) does not fix the underlying database-level lost update if two writers can still race each other there; the two failure modes need addressing separately, even though they were discovered together.
• Optimistic versioning shifts work onto callers (they must handle a rejected write and retry), which is a real API contract change, not just an internal implementation detail; every caller needs to be audited, not just the one that triggered this incident.
• A rollback plan that only reverts code but leaves already-corrupted orders in place doesn't fix the incident; the rollout plan needs a separate reconciliation step to identify and correct orders affected before the fix shipped.
• Testing "under load" without deliberately forcing the specific interleaving (two writers targeting the same key, timed to overlap) can pass even with the race still present, since races are timing-dependent and won't reliably reproduce under generic load alone.

Direct answer

Capacity planning for 4x growth is a five-step loop: clarify the target SLOs (service level objectives, meaning your latency/error/cost targets), collect the metrics that let you translate traffic into resource needs, forecast the growth curve rather than assuming it is linear, convert the forecast into a required footprint with explicit safety buffers, and validate the plan with load tests before you need it for real.

Structured elaboration

1. Clarify requirements. Pin down target latency and error-rate SLOs, acceptable cost ceiling, and what "growth" means concretely (is 4x steady over 12 months, or a step change at a known date?).

2. Collect metrics (traffic, resource, application, business).

• Traffic: requests per second (RPS), request size, per-endpoint queries per second (QPS), and percentile splits (p50/p95/p99, the 50th/95th/99th-percentile response times), not just averages.
• Resource: CPU, memory, disk I/O, network throughput, connection/thread counts.
• Application: latency histograms, error rates by type, queue depths.
• Business: planned launches, marketing pushes, seasonality that could accelerate growth faster than the trend line.

3. Forecast, don't just extrapolate the headline number. A flat 4x multiplier on today's RPS is a starting point, not the plan: fit the trend against historical data (linear or exponential depending on the growth pattern observed) and project percentiles separately, since tail behavior does not always scale with the average.

4. Convert to a required footprint with safety buffers. Measure resource-per-request from load tests or production data, then size for the forecast plus buffers: operational headroom for bursts, a failure-domain buffer so the system tolerates losing a node or a zone, and a smaller uncertainty buffer for forecast error.

5. Validate before you need it. Load-test at intermediate multiples (not just the final 4x), watch SLOs hold, and set a review cadence to re-forecast as real growth data comes in rather than trusting a 12-month-old projection.

Worked example

Take the stated numbers: current load 20,000 RPS, target 4x growth.

$4 \times 20{,}000 = 80{,}000 \text{ RPS target}$

Assume, as an illustrative input not given in the question, that load testing shows each node sustains 2,500 RPS at the target CPU/latency envelope:

$\text{Current fleet size} = \frac{20{,}000}{2{,}500} = 8 \text{ nodes}$
$\text{Fleet size at 4x, no buffer} = \frac{80{,}000}{2{,}500} = 32 \text{ nodes}$

Add a 25% operational headroom buffer for bursts:

$80{,}000 \times 1.25 = 100{,}000 \Rightarrow \frac{100{,}000}{2{,}500} = 40 \text{ nodes}$

So the plan targets roughly a 5x increase in fleet size (8 to 40 nodes) to serve a 4x traffic increase, once burst headroom is included, before adding any failure-domain buffer on top.

Trade-offs & pitfalls

• A single multiplier hides percentile risk. If p99 latency is already close to its SLO at today's load, a naive 4x-everything projection can understate how close the tail gets to breaching SLO, since tail latency often degrades faster than the average as utilization rises.
• Buffers stack, and stacking them uncritically gets expensive fast. Operational headroom, failure-domain buffer, and forecast-uncertainty buffer applied one after another can inflate cost well beyond what the traffic forecast alone justifies; size each buffer deliberately and be able to justify its percentage.
• Forecasts go stale. A plan built on a 12-month linear projection needs a review cadence; treat the 4x figure as a hypothesis to keep validating, not a fixed target to build once and forget.
• Load-testing only at the final target multiple misses problems that show up earlier. Testing at 1x, 2x, and 4x surfaces bottlenecks (a database connection limit, a downstream dependency's own capacity) that a single test at 4x can mask or misattribute.

Direct answer

Combine the three tools by role, not by stacking them arbitrarily: writes go through a single authoritative store that also emits a change event per write; a cache absorbs the hottest reads in front of everything else; read replicas serve the reads a cache cannot (cold keys, range queries); and a CQRS-style (Command Query Responsibility Segregation) read model, built from the change events, serves the reads that need a shape the write store cannot produce efficiently. Consistency is kept eventual and bounded by propagating changes through an ordered event stream with idempotent, versioned consumers; write conflicts are avoided by giving the primary store sole write authority and using optimistic concurrency for concurrent updates to the same record; and divergence is caught by comparing a version number on each record between the write store and every read path on a schedule, alerting when the gap exceeds a defined budget.

Structured elaboration

Architecture.

1. A single write-authoritative primary store (sharded if write volume requires it) accepts all commands.
2. Every committed write also appends an event to an ordered log (e.g., a Kafka-style stream), in the same logical operation as the write, so the event is never lost if the write succeeds.
3. Downstream consumers apply those events to: (a) a fast key-value cache for the hottest, latency-critical reads, and (b) one or more denormalized read models optimized for the query shapes the application actually needs (search, joins collapsed into one row, aggregates).
4. Read replicas of the primary remain available for queries that need full SQL (Structured Query Language) expressiveness the read models were not built for, at the cost of typical replication lag.

flowchart LR
    Cmd[Command / write] --> Primary[(Write-authoritative primary)]
    Primary --> Log[Ordered event log]
    Primary --> Replica[(Read replica)]
    Log --> Cache[Cache: hot keys]
    Log --> RM[(CQRS read model)]
    Read[Read request] --> Cache
    Cache -- miss --> RM
    Read --> Replica

Keeping read models eventually consistent.

• Order matters: events for the same entity must apply in the order they were produced, which a partitioned log guarantees only within a partition, so partition by entity key (for example order ID) to preserve per-entity ordering.
• Idempotency matters more than ordering in practice: consumers must be safe to re-apply the same event (after a retry or a rebalance) without corrupting state, typically by making the update a "set to version N" operation rather than a blind increment.
• Each read-model row carries the write-side version or sequence number it was built from, so a consumer can detect and discard an out-of-order or duplicate event instead of silently regressing a newer value to an older one.
• Cache entries carry a short time-to-live (TTL, time-to-live) as a safety net independent of the event pipeline, so a missed or delayed invalidation event self-heals within a bounded window rather than serving stale data indefinitely.

Handling write conflicts.

• Because the primary is the sole write authority, there is no multi-master conflict to resolve; the remaining conflict is concurrent updates to the same record racing each other.
• Use optimistic concurrency: a command carries the version it expects to update; the primary rejects the write if the stored version has moved on, and the client retries with the fresh version. This keeps single-record writes fast (no locking) while still preventing a stale write from silently overwriting a newer one.
• For an operation that must touch multiple records or shards, use a saga: a sequence of local transactions coordinated by explicit compensating actions if a later step fails, since a distributed transaction across the primary and any sharded peers is not available. Each saga step should be idempotent for the same reason event consumers must be.

Monitoring divergence.

• Emit the write-side version per entity from the primary and the version currently reflected in the cache and each read model.
• Run a sampling job that compares those versions for a subset of keys on a fixed interval and computes two numbers: version lag (how many versions behind) and time lag (how long since the read side last updated).
• Instrument and alert on: consumer lag on the event log (how far behind the read-model consumers are from the latest offset), replica replication lag on the read replicas, and cache staleness (hit rate plus age of served entries).
• Surface a single dashboard metric that a non-implementer can reason about: the percentage of reads in the last window that were served from data older than the team's staleness budget, alongside the raw lag metrics engineers use to debug it.

Worked example

Say the service serves 20,000 read requests/sec at peak and the team sets a target cache hit ratio of 95% (stated here as a design target, not a measured fact, since it has to be validated against real traffic after launch).

That 1,000 requests/sec is the origin load the cache miss path has to absorb across the read replicas and read model combined, roughly a 20x reduction from the 20,000/sec that would hit origin with no cache in front at all. Split evenly across 3 read replicas, that is about 333 requests/sec per replica, comfortably inside a single replica's typical capacity.

Now take a concrete divergence scenario. Normal event throughput is 3,000 events/sec and the read-model consumer keeps pace at the same rate, so lag stays near zero. A 60-second burst pushes production to 8,000 events/sec while the consumer's steady processing capacity stays at 5,000 events/sec:

After the burst, production returns to 3,000 events/sec while the consumer still processes at 5,000 events/sec, so the backlog drains at:

That 90-second figure is exactly what the consumer-lag alert threshold should be checked against: if the team's staleness budget is, say, 30 seconds of lag, this burst would breach it for a full minute and the alert should fire well before the 90-second mark, not only once the backlog is fully drained.

Trade-offs & pitfalls

• Skipping the version number on read-model rows is the most common mistake: without it, an out-of-order or replayed event silently regresses a record to an older state, and there is no way to detect it after the fact.
• Relying on cache TTL alone, with no invalidation event, trades correctness for simplicity in a way that is easy to defend in design review and painful in an incident, since staleness then scales with the TTL rather than with actual event lag.
• Treating optimistic concurrency retries as free is a mistake under real contention: a hot record with many concurrent writers produces a retry storm, which is a signal to reconsider the access pattern (batching, or a different partition key) rather than just widening the retry budget.
• Sagas without idempotent, well-defined compensating actions turn a partial failure into a data-integrity incident instead of a handled edge case; this is where interviewers probe hardest on this design.
• A design that only reports "is the cache warm" without also reporting version or time lag on the read models gives false confidence, since a cache can be perfectly warm while serving data that is several versions behind the primary.

Direct answer

For a 10x daily traffic swing, reserve baseline capacity for the trough, autoscale the delta up to peak, and lean predictive/scheduled scaling over purely reactive, since a daily swing is exactly the regular, forecastable pattern predictive scaling is good at. Use burstable instances only where bursts are genuinely short; use spot for stateless, interruption-tolerant work; keep on-demand or reserved capacity for the latency-critical path. The instance and purchasing decisions matter less than getting this basic shape right: match committed spend to the floor, elastic spend to the swing.

Structured elaboration

Reserved vs. spot vs. on-demand

• Reserved (or savings-plan) capacity for the steady trough load, since that portion runs 24/7 regardless of the swing and benefits most from a committed discount.
• Spot for stateless, horizontally-scaled, interruption-tolerant work (background/batch consumers), paired with graceful draining so an interrupted instance doesn't corrupt in-flight work.
• On-demand as the buffer for latency-critical capacity above the reserved baseline, and as an immediate fallback when spot capacity is reclaimed.

Burstable vs. fixed instances. Burstable instances (which accumulate CPU credit during idle periods and spend it during bursts) are worth using only if the burst is genuinely short relative to how fast credits accumulate; a 10x swing that holds for hours, not minutes, will exhaust accumulated credit and fall back to a throttled baseline performance level, which is the opposite of what you want during peak. For a sustained multi-hour peak, more numerous smaller fixed instances scaled horizontally is the safer choice.

Predictive vs. reactive. A daily 10x swing is about as regular a pattern as autoscaling ever sees, which makes it a strong candidate for scheduled/predictive pre-scaling: scale up ahead of the known daily ramp rather than waiting for a reactive trigger to catch up. Keep a reactive, latency-based layer running underneath as the backstop for whatever the schedule gets wrong on an atypical day (a P95 latency target, the 95th-percentile response time, i.e. the value 95% of requests come in under, is a good backstop metric since it reflects what users actually feel).

Containers vs. VMs. A container platform with a cluster autoscaler can bin-pack (efficiently pack workloads of different sizes onto the fewest possible shared machines, matching each workload's resource need to available capacity the way you'd pack differently-sized boxes into as few shipping crates as possible) across mixed instance types (reserved, on-demand, spot) on shared nodes and starts new replicas faster than a fresh VM boots, which matters directly for how quickly the fleet can absorb the ramp into peak. Plain VM-level autoscaling is simpler operationally but coarser-grained and slower to provision; it's a reasonable choice when multi-tenant isolation or licensing requirements rule out sharing nodes.

Worked example

Total cost of ownership (TCO) comparison: fixed peak-sized warm capacity vs. autoscaling. Assume, as stated planning inputs, not measured figures: peak load requires 100 instances for 4 hours/day, trough load requires 10 instances for the remaining 20 hours/day, over a 90-day (roughly 3-month) window, at an illustrative $0.10/instance-hour.

Fixed warm capacity sized to peak, running at 100 instances around the clock:

Autoscaling between trough and peak:

In this illustrative model, autoscaling costs a quarter of what permanently warm peak-sized capacity would, a $16,200 saving over 3 months. That gap is exactly why "just overprovision and stop worrying about it" is rarely the right default for a regular, predictable swing like this one; it's a defensible fallback only when the swing is irregular enough that autoscaling reliably can't keep up, which a stable daily pattern isn't. (This simplified model ignores scale transition costs and the risk of a slow ramp missing the service level objective, SLO, the measurable target committed to for latency or availability; a real sizing exercise would validate the ramp time against the actual latency budget by load-testing the actual scale-out ramp: confirming that a newly-scaled instance passes its health check and starts serving real traffic before the P95 latency budget for that request is exhausted, not just that the instance count eventually reaches the target.)

Fast, extreme burst: 1-hour marketing spike from 1,000 to 100,000 requests per second (RPS). A daily-swing policy tuned for a gradual 10x ramp is the wrong tool for a sudden, 100x, hour-long spike: neither reserved nor on-demand compute autoscaling reacts fast enough on its own at that magnitude. Layer the response instead: put a content delivery network (CDN, a network of edge servers caching content close to users) in front to offload whatever fraction of that traffic is cacheable, entirely off origin compute; use serverless (functions as a service, FaaS, where the platform scales invocation concurrency per-request rather than per-instance) or a pre-warmed burst pool to absorb the residual dynamic traffic within the 1-hour window, since compute autoscaling alone is too slow to react meaningfully inside that timeframe.

Bursting to a second region vs. permanent capacity. For a short or infrequent peak, temporarily bursting overflow traffic to a second region avoids paying for duplicate capacity that sits idle everywhere the rest of the time, but it only works if the service can actually run statelessly in a freshly-spun-up region and if the operational readiness (data replication, DNS or traffic-shifting mechanics) was built and tested ahead of time, not improvised during the peak. Provisioning a single larger region permanently is simpler operationally but pays for that simplicity around the clock; the second-region approach trades operational complexity for lower steady-state cost, which is worth it exactly when the peak is genuinely short or rare.

Trade-offs & pitfalls

• Burstable-instance credit exhaustion under a sustained (not brief) peak silently degrades performance right when peak matters most; validate actual burst duration against credit accumulation before relying on this instance class for a multi-hour swing.
• Spot capacity is only appropriate for the fraction of the fleet that tolerates interruption; putting latency-critical serving on spot to save cost trades away exactly the reliability the SLO depends on.
• Predictive scaling tuned to a daily pattern will under-react to anything that isn't the daily pattern (a genuine spike layered on top of the normal swing); it needs the same kind of reactive, latency-based backstop described above for the daily-swing case: a layer that watches P95 latency directly and scales further whenever the schedule under-predicts, rather than assuming the daily schedule alone is sufficient.
• A cost comparison like the TCO example above is only as good as its throughput-per-instance and hours-at-peak assumptions; treat those as inputs to validate against real load tests, not numbers to trust by default.