Infrastructure Scaling, Capacity Planning, and High Availability Questions

Framing

This is a two-part problem: design a load test that tells the truth about a representative Spark workload's resource needs, then convert that measured behavior into a production sizing and scheduling plan, rather than sizing from first-principles theory alone. Two Spark-specific terms recur throughout: an executor is a worker process that runs tasks in parallel on the cluster, and a shuffle is Spark redistributing data across the network and disk to regroup it for a join or aggregation, the single most expensive operation in a job like this.

Representative job variants

Test 2-3 variants that stress different things, not one synthetic job: a wide, shuffle-heavy job (many joins/aggregations, stresses network and disk spill), a read-heavy scan job (stresses I/O and partition pruning), and, if applicable, a skewed-key job (stresses whether a few executors become stragglers), because these three failure modes call for different fixes.

Dataset sampling

Iterate fast against a stratified 10-20% sample that preserves the real key-skew distribution (a uniform random sample of skewed data hides the skew problem entirely), then validate the chosen configuration against the FULL 10TB at least once before calling it production-ready, sampled results alone can mislead on total shuffle volume and skew severity.

Configs to vary

• Executor count and cores per executor: more, smaller executors improve parallelism and fault isolation but add coordination overhead; fewer, larger executors reduce overhead but risk longer GC (garbage collection, the JVM reclaiming unused memory) pauses and a bigger blast radius per executor failure.
• Executor memory and spark.memory.fraction: headroom for shuffle buffering and spill.
• spark.sql.shuffle.partitions: the classic tuning knob, too few partitions means each task handles too much data and spills to disk, too many means per-task scheduling overhead dominates.

What to measure

Stage-level timing from the Spark UI/history server to find the actual bottleneck stage (shuffle read/write time specifically), executor GC time (a signal of memory misconfiguration), disk spill bytes (a signal that shuffle partitions are too coarse or executor memory too low), and straggler task duration (max task time versus median, a large gap is the signature of key skew).

Failure injection

Deliberately kill a subset of executors mid-job (Spark's dynamic allocation and task retry should absorb this) and briefly cut a worker node's network, to confirm the job still completes, with retries, inside the 2-hour SLO under a realistic partial-failure scenario. A cluster sized only for the happy path will miss its SLO the first time a spot executor gets reclaimed mid-run, which is a real, common occurrence, not an edge case.

Converting to production sizing

Take the configuration that hit the SLO on the sampled runs and scale it by the ratio of full data volume to sample volume as a starting point (for example, if a 2TB sample needed 40 executors at 4 cores / 16GB to finish in 24 minutes, a naive linear scale for 10TB is 200 executors), then add 15-20% headroom because shuffle cost doesn't scale perfectly linearly at 5x volume, and validate the scaled configuration against the actual full dataset before trusting it.

Scheduling for concurrency

If multiple daily jobs share the window, use a scheduler-level fair-share or capacity queue (YARN, Hadoop's cluster resource manager, capacity scheduler, or a per-namespace resource quota for Kubernetes-mode Spark) rather than sizing one dedicated cluster per job. Define explicit roll-forward and rollback criteria: if a config change (say, a new shuffle-partition count) causes the next run to blow past the 2-hour SLO or spike its retry rate, flag that run's configuration automatically and fall back to the last known-good Spark configuration, rather than silently absorbing a regression into the next day's baseline.

Framing

Read latency during peak hours usually means either the database is doing more work per query than it needs to (missing/inefficient indexes, expensive queries), or it is simply out of spare read capacity for the query volume, or both. Walk the cheap, low-risk options first, then the ones that add real infrastructure.

Query and index optimization

Check for missing indexes, full table scans, or N+1 query patterns (running one extra query per row of a result set instead of one query for the whole batch) first, since a query-level fix can remove the bottleneck entirely without adding any infrastructure. Capacity implication: none, existing capacity just goes further. Cost: essentially free, engineering time only. Complexity: low, though it requires actually profiling slow queries rather than guessing.

Connection pooling / query tuning at the app layer

If latency is partly from connection overhead or the database thrashing under too many concurrent connections, a connection pooler smooths that out. Capacity implication: effectively increases usable throughput from the same database without adding hardware. Cost: low, one more lightweight component to run. Complexity: low to moderate, mainly around correctly sizing the pool and handling pooled-connection quirks.

Caching (read-through cache in front of the database)

Put a cache in front of the hottest, most repeated read queries. Capacity implication: removes a large share of read load from the database entirely for cacheable queries, the biggest capacity lever on this list for read-heavy hot spots. Cost: moderate, a new stateful component to run and monitor. Complexity: moderate to high, mainly cache invalidation and cold-cache or stampede behavior (a stampede is many requests missing the cache at the same moment and all hitting the database at once) after a deploy or cache flush.

Read replicas

Add one or more read replicas and route read traffic to them, keeping writes on the primary. Capacity implication: scales read throughput roughly linearly with the number of replicas, until the primary itself becomes the bottleneck for something else. Cost: moderate to high, full extra database instances. Complexity: moderate, mainly replication lag and deciding which reads can tolerate slightly stale data.

Vertical scaling of the database

Move to a bigger instance. Capacity implication: immediate, and can improve latency directly if the bottleneck is genuine resource contention. Cost: scales up steeply at the high end and does not scale forever. Complexity: low to execute, but it is a stopgap, not a structural fix, and does not address the underlying query pattern if that was the real problem.

Sharding/partitioning

Split data across multiple database instances by some key. Capacity implication: the highest ceiling of any option here, scales close to linearly with shard count. Cost and complexity: by far the highest on this list, query routing, cross-shard queries, re-sharding, a last resort reserved for when the simpler options are genuinely exhausted, not a first response to a peak-hour latency complaint.

Putting it together

Start with query/index fixes and connection pooling since they are nearly free and often fix the problem outright. Add caching or a read replica if the workload is genuinely read-heavy and those are not enough. Treat vertical scaling as a short-term buffer while the real fix ships. Reserve sharding for when the write/data volume itself, not just read latency, has outgrown a single database.

Discrete-time steps vs event-driven simulation

These are the two standard ways to build this kind of simulator, and they trade off differently:

• Discrete-time: advance the clock in fixed steps (say, every second or every minute), and within each step, generate arrivals, serve what capacity allows, and carry any backlog forward. Simple to implement and reason about, but accuracy depends on the step size being fine enough relative to service times; too coarse a step blurs real queueing dynamics within the step.
• Event-driven: maintain a priority queue of future events (next arrival, next service completion) and jump the clock directly to the next event, with no fixed step at all. This is exact (no time-resolution error) and can be more efficient when the system is mostly idle between events, but is more complex to implement correctly, especially once you add a time-varying scaling policy that needs to check state on its own schedule, not just at event times.

I would default to discrete-time for a policy simulator like this, since scaling policies themselves typically evaluate on a fixed cadence (a metrics-polling interval), which maps naturally onto discrete steps; event-driven earns its complexity when you need exact timing (e.g. validating a queueing-theory result against a closed-form answer) rather than validating a policy's behavior over time.

To make the trade-off concrete, here is a minimal event-driven core (using a min-heap of (time, event_type) pairs, jumping straight to the next thing that happens rather than stepping through fixed intervals):

import heapq
import random

def event_driven_sim(arrival_rate, service_rate, num_servers, sim_seconds, seed=99):
    rng = random.Random(seed)
    events = []
    t = 0.0
    heapq.heappush(events, (rng.expovariate(arrival_rate), "arrival"))
    queue_len, busy_servers = 0, 0
    area_under_queue, last_t = 0.0, 0.0
    arrivals = departures = 0

    while events and t < sim_seconds:
        t, etype = heapq.heappop(events)
        if t > sim_seconds:
            break
        area_under_queue += queue_len * (t - last_t)
        last_t = t
        if etype == "arrival":
            arrivals += 1
            if busy_servers < num_servers:
                busy_servers += 1
                heapq.heappush(events, (t + rng.expovariate(service_rate), "departure"))
            else:
                queue_len += 1
            heapq.heappush(events, (t + rng.expovariate(arrival_rate), "arrival"))
        else:
            departures += 1
            if queue_len > 0:
                queue_len -= 1
                heapq.heappush(events, (t + rng.expovariate(service_rate), "departure"))
            else:
                busy_servers -= 1
    return {"arrivals": arrivals, "departures": departures,
            "avg_queue_len": area_under_queue / last_t if last_t else 0.0}

if __name__ == "__main__":
    print(event_driven_sim(arrival_rate=9.0, service_rate=1.0, num_servers=10, sim_seconds=20000))

Output (deterministic, re-run twice, identical both times):

{'arrivals': 179671, 'departures': 179658, 'avg_queue_len': 6.107049003476066}

At arrival rate 9/sec against 10 servers each handling 1/sec (utilization rho = 0.9, a heavily loaded system), the time-averaged queue length of about 6.1 is expected: high utilization queueing systems build meaningful backlog even without being over capacity, which is itself a useful thing for a capacity simulator to surface.

Data structures

Discrete-time needs little beyond arrays or a rolling window of recent state per server group. Event-driven needs a priority queue (a min-heap, as above) keyed by event time, since the core operation is always "what happens next", and a heap gives that in O(log n) per event.

Limitations of this kind of simulator, either version

• Assumes arrivals and service times follow simple, memoryless distributions (Poisson arrivals, exponential service times); real traffic and real service times are rarely this clean (bursty, correlated, heavy-tailed).
• A simple average-utilization scaling policy reacts to a single aggregate signal, missing per-endpoint or per-shard hotspots that a real system would need to handle.
• Neither version models scale-up lag (the real time it takes new capacity to come online and become useful), which is often the dominant factor in whether a scaling policy actually prevents a queue backlog in practice; bolting that in requires modeling capacity itself as a delayed, not instantaneous, response to the policy's decision.
• Both versions model a single aggregated queue; real systems often have per-shard or per-region queues with their own dynamics, and this simplification can hide load-imbalance failure modes.

Direct answer

Capacity planning a high-throughput streaming pipeline (Kafka or Pub/Sub, both distributed log/message systems that decouple producers from consumers) means sizing four coupled dimensions from one basis, the sustained ingest rate, and then separately accounting for peak load and disaster recovery (DR, the plan for continuing service after a major regional outage) on top of the steady-state numbers.

Structured elaboration

Start from the single given number, 2 TB/hour ingest, and derive everything else from it so every downstream calculation shares the same basis. Then layer in: storage (how much retained data, and multiplied by however many copies durability requires), partitioning (how the stream is split for parallelism), network (ingest, internal replication, and consumer read traffic are three separate flows), and CPU (driven mostly by compression and replication overhead, not raw byte-copying). Finally, provision for the gap between average and peak, and for a DR region.

Worked example

Converting the ingest rate to a per-second basis: $2\text{ TB/hour} = \frac{2 \times 10^{12}\text{ bytes}}{3600\text{ s}} \approx 555.6\text{ MB/s} \approx 4.44\text{ Gbps}$

Storage. Over the 7-day retention window: 2 TB/hour * 24 hours * 7 days = 336 TB of raw, unreplicated data. Both Kafka and Pub/Sub replicate each message to multiple brokers for durability; at a replication factor (RF) of 3 (a common default, each message stored on 3 separate brokers), that is 336 * 3 = 1,008 TB. Add roughly 15% for index and segment-file overhead and the storage footprint to provision is close to 1.16 PB (petabytes) for the retention window.

Partitioning. Partitions are the unit of parallelism, each partition is read and written by at most one consumer within a consumer group at a time, so partition count controls maximum consumer parallelism as much as throughput. Using a common rule of thumb of roughly 10 MB/s of sustained throughput per partition (actual per-partition throughput depends on message size and compression, so validate this against your own broker benchmarks): 555.6 / 10 ≈ 56 partitions at minimum. Adding 30% headroom for peak and for future consumer-group parallelism needs gives roughly 73, round to a clean 80 partitions.

Network. Three distinct flows exist on the broker side: the 4.44 Gbps of producer ingest itself, internal replication traffic to the other RF - 1 = 2 broker replicas of each message (another 4.44 * 2 ≈ 8.9 Gbps), and consumer fan-out (if 3 independent downstream consumer groups each read the full stream, that is another 4.44 * 3 ≈ 13.3 Gbps of egress). Total broker-side network load is roughly 4.44 + 8.9 + 13.3 ≈ 26.7 Gbps at steady state, dominated by replication and fan-out, not the raw ingest number most people size against first.

CPU. Broker CPU is driven mainly by compression/decompression and replication bookkeeping rather than pure I/O. As an illustrative rule of thumb (validate against your own broker's measured throughput per core, since this varies heavily with compression codec and message size), at roughly 300 Mbps of compressed throughput handled per core, the 26.7 Gbps total broker-side traffic implies on the order of 90 cores of aggregate CPU across the cluster at average load, before peak headroom.

Peak vs average. If the 2 TB/hour figure is a sustained average and traffic follows a typical diurnal or bursty pattern, assume a peak-to-average ratio (validate this from real traffic, a commonly used starting assumption is 2 to 3x) of 3x: peak ingest becomes roughly 4.44 * 3 ≈ 13.3 Gbps, and every downstream number (partitions, network, CPU) should be provisioned against the peak figure, not the average, or the pipeline falls over during the exact moments it matters most.

DR. Replicate the topic (or the subset of it that is DR-critical) to a second region using the platform's native cross-region mirroring capability, sized to the same peak throughput, with its own retention policy (full 7-day retention in DR is expensive; many teams retain a shorter DR window and rely on the primary region for the full history).

Trade-offs & pitfalls

• Sizing off the average ingest rate alone is the single most common mistake; every dimension above should ultimately be checked against the PEAK number.
• Under-partitioning caps consumer parallelism even if raw throughput capacity looks fine, a common cause of consumer lag that looks like a "slow consumer" problem but is actually a partition-count problem.
• Replication and consumer fan-out traffic together can dwarf the raw ingest number, sizing network capacity off ingest alone will under-provision the cluster.
• DR retention does not have to match production retention; matching it by default multiplies storage cost for capacity that is rarely read.

The three tiers

• Hot tier (recent logs, say the last 7 days): the working set actively queried, needs low latency (single-digit milliseconds) and high throughput, so it belongs on fast storage, SSD or NVMe (a very low-latency solid-state storage interface) backed, sized to comfortably hold the retention window's data plus growth headroom.
• Warm tier (last 90 days): queried less frequently and can tolerate noticeably higher latency (tens to low hundreds of milliseconds), so it belongs on standard, cheaper storage, often a mid-tier object storage class rather than block storage attached to compute.
• Cold tier (older than 90 days): rarely queried, and when it is, a retrieval delay of minutes to hours is usually acceptable, so it belongs on archival-class object storage, the cheapest tier available, accepting the higher retrieval latency as the trade-off for a large reduction in storage cost.

Automated tiering

Define lifecycle policies that move data between tiers automatically based on age (data crossing the 7-day boundary moves from hot to warm, crossing the 90-day boundary moves from warm to cold), rather than relying on manual intervention. This keeps the hot tier's size bounded and predictable regardless of how much total data has accumulated, since only the last 7 days ever lives there.

Autoscaling storage nodes

Size the hot tier's node count from ingestion rate and retention window: if logs arrive at a known rate (for example, X GB/day) and the hot tier retains 7 days, required hot-tier capacity is roughly 7 times the daily ingestion rate, and that capacity should autoscale as ingestion rate grows, adding storage nodes when sustained ingestion or query load approaches the current tier's provisioned ceiling, rather than provisioning once for a guessed peak. The warm and cold tiers, being backed by elastic object storage rather than fixed compute-attached storage, generally scale near-transparently with volume and need far less manual node management.

Coordinating with compute autoscaling

The compute layer (whatever reads and processes the data, for example a query engine or log-search service) needs to scale in step with the tier it is serving:

• For the hot tier, query and ingestion compute should scale roughly with hot-tier node count and traffic, and ideally sits close to (or is co-located with) the hot storage nodes to minimize the latency of the most frequent, most latency-sensitive queries; data locality matters here in a way it does not for cold data.
• For cold-tier access, compute does not need the same tight coupling: since cold queries already tolerate minutes of latency, compute serving cold-tier requests can scale reactively and even tolerate cold starts (spinning up on demand), since the latency budget already assumes a slow path.

Putting it together: the hot tier is the one that needs both fast storage and tightly coupled, low-latency compute scaling; the warm and cold tiers trade query speed for storage cost and can rely on more elastic, loosely coupled infrastructure since nobody expects a fast answer from them.