Infrastructure Scaling, Capacity Planning, and High Availability Questions

Approach

A single-node job processing thousands of files sequentially has an SLA (service-level agreement, the target completion time the pipeline is expected to meet) problem that is fundamentally about parallelism, not raw hardware speed: on one machine, wall-clock time is file count times per-file processing time regardless of how big that machine is. The fix is to spread files across multiple workers, then decide how those workers are provisioned and scaled.

Strategies

1. Partition the work across a fixed or autoscaled worker pool. Split the file list into shards, or better, use a pull-based queue that workers pop files from, so a worker that finishes early can pick up more work instead of sitting idle while another worker is still processing a shard of unusually large files.
2. Use a distributed data-processing framework (for example Spark) if the per-file transform is expressible as a data-parallel operation. This buys built-in scheduling, retries, and fault tolerance, at the cost of cluster-management overhead that can be more machinery than a straightforward per-file transform needs.
3. Use serverless fan-out, one function invocation per file or small batch of files. Scales trivially to thousands of near-simultaneous invocations with no cluster to manage, but pays per-invocation cold-start overhead that can dominate cost and latency when per-file work is very small.
4. Use a managed batch or job-queue service sized dynamically to the day's file count. Sits between the two extremes: autoscaled worker provisioning without owning a persistent cluster, at the cost of somewhat higher per-job startup latency than an already-warm function fleet.

Trade-offs

Recommended shape

The fastest, lowest-risk fix to a missed SLA is usually converting the sequential loop into a pull-based worker pool sized so that worker count times per-file time covers the file count comfortably inside the SLA window, since it requires the least new infrastructure and directly attacks the root cause: no parallelism. If file count or per-file time keeps growing, migrating that same worker logic behind serverless fan-out or a managed batch service removes the burden of owning worker-pool capacity planning, at the cost of adopting a new platform's operational model.

Worked check

If one node processes one file in 2 seconds and there are 5,000 files, sequential runtime is 5,000 times 2 seconds, about 2.8 hours. Spreading that across 20 parallel workers, assuming near-linear scaling, brings it to roughly 10,000 seconds divided by 20, about 8.3 minutes, comfortably inside most nightly SLA windows. The worker count needed for any target SLA is simply total file-seconds divided by target seconds.

When to add nodes

Both Cluster Autoscaler and Karpenter work off the same core signal: a pod is pending because no existing node can fit it. The autoscaler simulates whether a new node would let it schedule, and if so, provisions one. Scale-down triggers on the opposite condition, a node's actual usage stays below a threshold (commonly under 50%) for a sustained window, and everything on it could be rescheduled elsewhere without violating constraints.

Which instance type

For Cluster Autoscaler, define multiple node groups (one per instance family/size) and pick an expander strategy: least-waste chooses whichever group's instance type leaves the least unused capacity for the pending batch of pods, priority lets you explicitly rank preferred types (for example, try spot first, fall back to on-demand). Karpenter skips pre-defined node groups entirely: a single flexible NodePool lists allowed instance types and Karpenter picks the best (usually cheapest sufficient) fit for the actual pending pods at provisioning time, which adapts better to heterogeneous requirements without hand-maintaining a matrix of groups.

GPU handling

A dedicated, tainted GPU node pool (or NodePool spec) is only provisioned when a GPU-requesting pod is pending, same tainting and toleration pattern used to keep CPU-only pods off GPU nodes. Bin-pack GPU pods using a MostAllocated scheduling strategy so small GPU jobs consolidate onto fewer, fuller multi-GPU nodes rather than fragmenting availability across many partially-used ones.

Avoiding churn when pods have long startup times (model loading)

The real risk isn't scaling down an idle node, it's scaling down a node that LOOKS idle mid-warm-up, right after a pod starts but before it's actually doing work, or scaling it down just as loading finishes. Concrete mitigations:

• Mark warming pods cluster-autoscaler.kubernetes.io/safe-to-evict: "false", or back them with a PodDisruptionBudget requiring minAvailable, so the autoscaler won't remove the node under them mid-load.
• Extend the scale-down "unneeded" evaluation window for this node pool (for example 20-30 minutes instead of a default 10), so a brief idle-looking reading right after startup doesn't trigger removal.
• Keep a small always-on baseline (minimum pool size above zero) for latency-critical inference, so scale events apply only to genuine burst capacity, not the steady-state fleet.
• Reduce the effective startup window itself: pre-pull the model container image and cache the model weights on faster local storage, so there's less time during which a node looks idle-but-warming.

Cost minimization

Prefer spot for training and batch (tolerant of interruption), on-demand baseline for latency-sensitive inference, and once churn risk is controlled, enable active consolidation (Karpenter's built-in repacking, or a periodic descheduler pass for Cluster Autoscaler) to actually shrink the fleet back down once bin-packing has room to improve.

This is a behavioral question, so the answer below is a worked example of the kind of structured, specific story an interviewer is looking for. Use the STAR shape (Situation, Task, Action, Result) and swap in your own real experience with a comparable trade-off.

Situation

At a mid-size company running an ML platform, the infrastructure team operated a fixed GPU cluster used both for nightly model retraining jobs and for serving a real-time recommendation API with a 200ms p99 (the latency experienced by the slowest 1% of requests) latency SLO (service level objective, the internal performance target the team held itself to). Midway through a quarter, finance asked infrastructure spend to come down by roughly a third within a month, and the GPU cluster was the single largest line item.

Task

I owned finding a way to hit the cost target without breaking the recommendation API's latency SLO, and without simply asking for more budget, which had already been ruled out.

Action

I mapped out where the GPU hours were actually going: most of it was nightly retraining, a smaller but latency-critical slice was live inference. I modeled two options: delay the retraining cadence from nightly to every 3 days, freeing up the bulk of the GPU time immediately with a quantifiable, boundable quality cost (a model a few days staler), or reduce inference concurrency to shrink the serving fleet, which directly risked breaching the SLO during peak traffic. I built a small before/after estimate for each option showing dollar impact and the specific risk it introduced, then brought both options plus my recommendation (delay retraining, leave inference concurrency untouched) to the team lead and the product stakeholder who owned the recommendation feature, explaining the trade-off in terms they could evaluate: staleness versus latency risk, not just abstract cost numbers.

Result

The team agreed to the retraining-cadence change. It hit the cost target within the month, the recommendation API's SLO was never at risk since concurrency was untouched, and we set a follow-up check a few weeks later to confirm model quality had not measurably degraded from the staler retraining cadence, which it had not. The concrete artifact that made the decision easy to sign off on was the side-by-side cost/risk comparison rather than a single recommendation stated without alternatives.

How to adapt this to your own story: pick a real trade-off you owned, be specific about the numbers and the options you actually considered (not just the one you picked), name who you communicated it to and how, and be honest about the outcome, including anything you would do differently next time.

Framing

Placing multiple full copies of a large model on GPU nodes for A/B testing is expensive if done naively, each variant reserving its own full memory allocation. The goal is to share what's safe to share (frozen, identical weights) while keeping what must differ isolated, without either variant's load starving the other's latency and quietly invalidating the comparison.

Placement and memory-sharing strategy

When variants are close cousins, for example two fine-tunes of the same base model or two checkpoints differing only in a small adapter (a LoRA-style adapter is a small set of additional weights trained on top of a frozen base model, so two variants can share the base and differ only in that small delta), co-locate them on the same GPU node and load the shared base weights into GPU memory once, duplicating only the differing layers per variant. This is the largest memory win available, since adapter weights are typically a small fraction of the base model's size.

When variants are not close cousins, fully independent fine-tunes with no shared frozen base, fall back to hardware-level isolation instead of loading two full copies onto one unpartitioned GPU: MIG (Multi-Instance GPU, a hardware feature on some GPUs that splits one physical GPU into several fully isolated virtual GPUs with dedicated memory) gives real memory and performance isolation rather than best-effort sharing.

Bin-pack (fit each variant onto the node where it wastes the least leftover capacity, rather than spreading them out) by memory footprint first, a variant that barely fits solo shouldn't be co-located, then balance expected request rate across nodes so no single GPU carries a disproportionate share of either variant's traffic.

Avoiding interference

Prefer hardware partitioning (MIG, or dedicated GPUs per variant) over software-only sharing whenever traffic volume justifies the extra memory cost, because compute contention (two models competing for the same streaming multiprocessors, the GPU's core compute units, and for the scheduler that decides whose kernels run on them and when) is harder to bound than memory sharing alone fixes. If sharing a GPU without hardware partitioning, set explicit per-process compute and memory quotas, for example via the GPU driver's MPS (Multi-Process Service, which lets multiple processes share a GPU with some scheduling isolation), rather than relying on best-effort scheduling, and monitor each variant's GPU utilization and queueing delay separately so interference becomes a measurable, alertable signal instead of a mystery latency difference.

Ensuring fair comparison metrics

Route traffic to variants through consistent, randomized assignment, for example hashing the user or session ID, so both variants see a statistically similar traffic mix regardless of placement decisions. Because placement can still introduce a confound, if variant A happens to share a node with a noisier neighbor than variant B, explicitly monitor and report each variant's own per-node contention level alongside its outcome metrics, so a reviewer can rule out placement as the explanation before trusting the result. Keep placement stable for the full duration of the experiment; letting the scheduler silently re-place a variant mid-test for bin-packing efficiency introduces exactly the kind of confound the experiment is trying to avoid.

Framing

Bin-packing here means deciding which pods land on which nodes so capacity is used efficiently. GPU fragmentation means the cluster's total free GPU count looks fine but no single node has enough free GPUs together for a job that needs them, because usage is scattered thinly across many nodes. The goal is to keep CPU-only work from ever contending with GPU nodes, and to keep GPU usage itself packed tightly rather than spread thin.

Kubernetes primitives to enforce placement

• Separate node pools. GPU nodes live in their own pool, CPU-only nodes in another.
• Taints and tolerations. Taint every GPU node (for example nvidia.com/gpu=present:NoSchedule). Only pods that explicitly tolerate the taint, and request the GPU resource, can land there. This is a hard guarantee: a CPU-only pod physically cannot schedule onto a GPU node by accident.
• Node affinity / nodeSelector. GPU-requiring pods additionally target the GPU pool by label (for example accelerator: nvidia-a100), so they don't just tolerate GPU nodes, they actively prefer them.
• The NVIDIA device plugin exposes GPUs as the extended resource nvidia.com/gpu, which the scheduler treats as a whole, non-shareable integer per pod by default (no fractional sharing unless you add MIG (Multi-Instance GPU, an NVIDIA feature that splits one physical GPU into several smaller, hardware-isolated GPUs) partitioning or a fractional-GPU scheduler on top).
• PriorityClass on GPU workloads so, under contention, the scheduler preempts lower-priority pods rather than leaving a high-value training job pending.

Bin-packing approach

Kubernetes' default scheduler score spreads load across nodes (LeastAllocated-style), which is the opposite of what you want for GPUs: spreading one GPU pod per node fragments availability, so a job needing 4 GPUs on one node can't find a home even though 4 GPUs are free cluster-wide. Configure the scheduler's NodeResourcesFit plugin with a MostAllocated strategy for the GPU node pool, so the scheduler actively fills already-partially-used nodes before opening a new one. Pair this with node pools sized to the actual job shapes you run, for example dedicated 8-GPU nodes for large training jobs and separate 1-GPU nodes for single-GPU inference, so a multi-GPU job's placement constraint doesn't get blocked by nodes that were never going to fit it.

Handling fragmentation and hot-standby

Fragmentation shows up when many small jobs each grab one GPU across many nodes, leaving no node with enough free GPUs for a bigger job. The MostAllocated scorer directly fights this by consolidating small jobs onto fewer nodes. For hot-standby, run a low, negative-priority placeholder ("pause") pod on GPU nodes that reserves capacity without doing real work; when a real GPU job arrives, the scheduler preempts the placeholder, and its eviction is exactly the signal the cluster autoscaler or Karpenter needs to provision a replacement node ahead of demand. That gets you pre-warmed capacity for bursty GPU demand without paying for genuinely idle reserved nodes indefinitely.