Infrastructure Scaling, Capacity Planning, and High Availability Questions

Requirements to clarify:

• Number of variants, QPS, latency SLA, model size, GPU memory per node, allowed extra CPU/GPU memory, and metric windows for A/B.

Placement & memory-sharing strategy:

• Co-locate variants on the same GPU node but avoid full duplication of weights:
  • Share a single read-only weight buffer in GPU memory and map variant-specific parameters (e.g., heads or fine-tuned layers) as small delta tensors. Use copy-on-write semantics so shared parameters use one physical buffer; only deltas allocate extra memory.
  • Use memory-mapped files (CUDA IPC / cuFile or unified virtual addressing) for the base model so different processes/containers can mmap the same physical GPU pages.
  • For very large models, use model sharding: split layers across multiple GPUs and reuse shared shards across variants; only shard-resident deltas differ.
  • Offload activations/optimizer state to host or NVMe using pinned memory or a device-side swap for infrequent paths to reduce GPU footprint.

Runtime placement & scheduling:

• Serve each variant as a lightweight process/thread that references shared weight memory. Use CUDA streams per variant to avoid kernel serialization and enable concurrent inference.
• Enforce per-variant CUDA stream and GPU resource quotas (NCCL streams, SM limits if supported) to prevent one variant hogging compute.

Avoiding interference:

• Isolate per-variant queues and thread pools; use OS/cgroups for CPU isolation.
• Warm-up each variant to populate caches; pin memory pages to avoid page faults during measurement windows.
• Prevent memory thrashing by limiting simultaneous active variants per GPU based on measured working set.
• Monitor and enforce GPU temperature/clock stability; run tests at stable power/clock governors.

Ensuring fair comparison metrics:

• Use randomized request assignment (stratified if needed) at the same edge/router to control traffic mix.
• Run A/B for the same time windows and under identical load profiles; use traffic shadowing for baseline checks.
• Collect per-request telemetry: latency percentiles, GPU utilization, memory usage, and model outputs. Tag traces with variant id.
• Use statistical tests (e.g., confidence intervals, t-test or bootstrapping) and warm-up exclusion window to avoid transient bias.
• Repeat experiments across multiple nodes and time slices to average out hardware or temporal noise.

Example practical stack:

• Base model in a shared CUDA IPC buffer; variant server processes (FastAPI + TorchServe) map shared tensors; per-variant small delta tensors loaded into GPU.
• Use Prometheus + OpenTelemetry for metrics; a router (Envoy) does randomized routing.

Trade-offs:

• Sharing reduces memory but complicates deployment and increases engineering complexity. If isolation is paramount, accept higher memory cost and fully duplicate models per GPU.

Steps and assumptions

• Assume all parameters and optimizer states stored in float32 (4 bytes per value). Gradients are stored in same dtype. No mixed precision or parameter sharding.
• Model size: 2 billion parameters.
• Optimizer: Adam keeps two states (m, v) same size as params.
• Activations: estimate peak live activations during forward/backward ≈ batch * seq * hidden_size * 4 bytes. Account for extra activation buffers (FFN intermediate, temporary grads) by a small multiplier.
• Choose a representative hidden_size: many large LLMs with ~2B params have hidden_size ≈ 4096 (rough approximation).

Numeric estimation

1. Parameters:

• 2e9 params * 4 B = 8.0e9 B ≈ 8.0 GB

2. Gradients:

• ~same size as params = 8.0 GB

3. Adam states (m and v):

• 2 * 8.0 GB = 16.0 GB

Subtotal (params + grads + optimizer): 8 + 8 + 16 = 32.0 GB

4. Activations (peak):

• batch 8 * seq 512 * hidden 4096 * 4 B = 8 * 512 * 4096 * 4 = 67,108,864 B ≈ 64 MB
• Add headroom for FFN/intermediate/temporary buffers ≈ ×3 → ~192 MB ≈ 0.2 GB

Total approximate GPU memory

• 32.0 GB + 0.2 GB ≈ 32.2 GB

Notes / alternatives

• If using mixed precision (FP16 weights + FP32 Adam masters) total can drop significantly: FP16 params (4 GB) + FP32 master params (8 GB) + grads (4 GB) + m/v (16 GB) → different composition but often lower peak.
• Frameworks may keep additional optimizer metadata, CUDA workspace, and activation checkpointing, so add ~2–4 GB overhead. Realistic production requirement ≈ 34–36 GB to be safe.

Requirements & constraints:

• Minimize cost while meeting SLOs (latency/throughput) for GPU/CPU AI workloads.
• Use spot/preemptible instances when safe; fall back to on-demand when needed.
• Enforce a monthly budget cap with pacing to avoid early burn.
• Handle spot preemption events gracefully.

High-level design:

• Kubernetes cluster with two node pools: spot (cheaper, preemptible) and on-demand.
• A Cost-Aware Autoscaler (CAA) as a Kubernetes controller that watches metrics, node pools, spot market signals, and budget state.
• Components: Metrics collector (utilization, queue length, pod priorities), Spot predictor (ML model or heuristics), Decision engine (risk scoring + optimization), Budget manager, Actuator (scale node pools), Preemption handler (eviction/draining policies, checkpointing).

Decision logic:

1. Forecast short-term load and spot availability (minutes-hours) using time-series model (ARIMA/LSTM) trained on historical usage and cloud spot reclaim rates.
2. Compute marginal cost per useful capacity (cost_per_vGPU * expected_uptime_fraction).
3. Assign workloads by priority:
   • Best-effort (training experiments): prefer spot if expected uptime > threshold T1.
   • Latency-sensitive/critical inference: prefer on-demand or spot with mirrored fallback.
4. Risk score = (1 - expected_uptime) * workload_criticality. If risk_score > threshold -> provision on-demand.
5. Maintain a safety buffer: keep X% of capacity on on-demand or warm pool to absorb sudden load or mass spot loss.
6. Budget pacing: convert remaining monthly budget to a burn rate; limiter reduces spot/on-demand scale when projected spend would exceed monthly cap (e.g., scale down low-priority jobs, pause noncritical experiments).

Handling preemption:

• Use checkpointing, model sharding, or k8s PodDisruptionBudgets to minimize lost work.
• On Spot preempt event, CAA immediately:
  • Recompute needed capacity, spin on-demand nodes for critical pods.
  • Reschedule best-effort jobs to remaining spot if projected uptime allows.

Implementation notes:

• Integrate with Kubernetes Cluster Autoscaler or Karpenter; implement CAA as a controller that influences provisioner requests.
• Use cloud APIs for spot pool health and pricing; persist budget state in a DB.
• Expose policy knobs: thresholds T1/T2, buffer %, priority classes.
• Monitoring & alerts for budget burn, spot churn, SLO violations.

Trade-offs:

• Aggressive spot use lowers cost but increases complexity for checkpointing and higher risk of SLO impact.
• Predictive models improve decisions but need retraining and can mispredict during market shocks — hence keep conservative safety buffer.

This design balances cost and reliability by combining forecasting, risk-based decisioning, priority-aware placement, and strict budget pacing.

Definition / key differences:

• On-demand instances: standard cloud VMs reserved for the duration you request; predictable availability, higher cost, suitable for latency-sensitive or long-running jobs where interruption is unacceptable.
• Spot / preemptible instances: spare capacity offered at steep discounts but with the risk of sudden termination by the provider (short notice, e.g., 30–120s). Best for cost-sensitive, fault-tolerant workloads.

Which ML workloads suit spot instances:

• Large batch training that can be checkpointed regularly (distributed training with periodic checkpointing to durable storage).
• Hyperparameter sweeps, population-based training, or many independent trials (each trial is short/isolated).
• Data preprocessing, feature extraction, and offline inference for non-latency-critical pipelines (e.g., nightly batch scoring).
• Ensemble/replica work where individual node loss is tolerable.

Unsuitable for spot:

• Single long-running training without checkpointing, real-time/low-latency inference, stateful services, or jobs with strict SLOs.

Failure modes:

• Sudden VM termination causing job failure, lost local state, corrupted distributed training state, slower overall progress if many restarts.

Mitigation strategies:

• Checkpoint frequently to durable object storage (S3/GCS) and resume from last checkpoint.
• Use fault-tolerant training frameworks: TorchElastic, Horovod with fault tolerance, TF's checkpoint recovery.
• Hybrid clusters: mix on-demand for masters/parameter servers and spot for workers; keep critical components on stable instances.
• Autoscaling + job queueing: requeue interrupted jobs automatically; use spot-aware schedulers (Kubernetes with node pools, Batch/Spot fleets).
• Redundancy: run redundant short trials or maintain ensemble diversity so single failures don’t block results.
• Diversify instance types and zones to reduce correlated revocations.
• Monitor spot interruption signals and drain gracefully (save state, notify orchestrator).

Trade-off summary:

• Use spot for cost-effective, fault-tolerant, and parallelizable ML tasks; keep on-demand for critical, stateful, or low-latency workloads.

Requirements and constraints:

• Capture model-level signals (loss, accuracy, perplexity, calibration, feature drift) and infra signals (GPU/CPU utilization, memory, PCIe/NVLink I/O, disk, network, node temps).
• Support multi-tenant clusters and experiments; low-latency for live training and batch analysis.
• Cost controls: sampling, aggregation, TTLs, cold-tiering.
• Ownership: tag by team, workspace, experiment, model, version, run-id.

High-level architecture:

• Instrumentation → Ingest (gateway) → Stream processor → Hot store + Cold store → Correlation/feature store → Alerting/Notebooks/Visualization.

What to collect:

• Metrics: per-step/per-epoch scalars (loss, lr, gradients norm), GPU metrics (util, power, mem), host metrics, data loader throughput, queue wait times. (high-cardinality labels limited)
• Traces: sampled end-to-end traces for data pipeline, training loop, checkpointing, model serving requests with spans for data load, forward, backward, sync, AIO.
• Logs: structured logs from trainers, schedulers, pre/post-processing, errors, stack traces, hyperparams, config snapshots.
• Artifacts: model checkpoints, evaluation datasets, feature distributions.

Sampling & retention:

• High-frequency infra metrics: aggregate to 1s-10s and keep raw 1s for 24–72h, 10s-1m for 30d, downsampled to 5m for 1y (cold storage).
• Model scalars: record every step during debug runs; for long runs, sample every N steps (adaptive sampling based on anomaly score) + always keep per-epoch summaries.
• Traces: probabilistic sampling (e.g., 1–5%) + deterministic sampling of error traces and traces correlated to drift/alerts.
• Logs: index errors/warnings fully; sample INFO/DEBUG based on run tag (retain full for important experiments).
• TTLs: auto-tier to cheap object storage after hot window, compress numeric timeseries.

Labeling/tagging:

• Mandatory labels: team, project, workspace, model_name, model_version, run_id, job_type (train/validate/serve), tenant_id.
• Ownership mapping service to route alerts and access control.
• Enforce via instrumentation libraries and CI checks that reject unlabeled metrics.

Correlation & analysis:

• Use a unified event time store to join model metrics with infra metrics by time window and run_id.
• Compute rolling correlations and causality proxies (Granger test, lag analysis) between loss spikes and infra signals (GPU throttling, OOMs, high PCIe latency). Enrich with traces to identify span causing slowdown (e.g., data loader blocking).
• Feature-store snapshots: track input feature distributions; use drift detectors (KL, PSI) and correlate feature drift with accuracy degradation.
• Build an observability ML pipeline that produces root-cause candidates: feature drift, resource saturation, I/O bottleneck, software regressions.

Alerting & playbooks:

• Multi-threshold alerts: symptom (loss increases), cause (GPU mem pressure), and combined severity rules. Route to owner via labels.
• Auto-trigger runs: when alert fires, capture high-fidelity trace+logs for next N minutes.

Tech choices & scaling:

• Ingest: Kafka or Pulsar; processing: Flink or Spark Streaming for enrichment/aggregation.
• Hot metrics store: Prometheus/Thanos/M3DB for infra; Cortex/Promscale for multi-tenant. Long-term: ClickHouse or Parquet on S3.
• Traces: Jaeger/Tempo with indexed error traces.
• Logs: Loki/ELK with JSON structured logs; archive to object store.
• Feature/metric correlation: ClickHouse or vector DB + Jupyter dashboards; Grafana + custom apps for model metrics.

Cost trade-offs:

• Prioritize full fidelity for short hot windows; use adaptive sampling driven by anomaly detectors to retain detail only when interesting.
• Use label-driven retention (experiments marked "prod" get longer retention).
• Enforce cardinality limits and synthetic aggregation to prevent explosion.

Operational practices:

• Standard instrumentation library for frameworks (PyTorch/TensorFlow) that enforces tags and efficient client-side aggregation.
• Regular audits for high-cardinality metrics, cost reports per team.
• Blameless postmortems instrumented into the system to close the loop and improve playbooks.