Model Deployment and Inference Optimization Questions

Situation: You run mixed CPU/GPU nodes and multiple ML teams compete for scarce GPUs. You need predictable placement, avoid GPU fragmentation, allow preemption for high-priority jobs, and enforce fair sharing.

Strategy (high level):

• Treat GPUs as a first-class, scarce resource and combine node metadata, kube device-plugins, a smarter scheduler (or scheduler plugin), and namespace/quota controls.

Node labeling, taints & affinities:

• Label nodes by accelerator and capability: e.g., accelerator=nvidia-a100, gpu.mig=true, gpu-count=8. This lets pods request specific hardware types.
• Taint GPU nodes (nvidia.com/gpu=present:NoSchedule) and add tolerations only to GPU workloads to avoid accidental placement.
• Use nodeAffinity/antiAffinity for explicit placement controls (e.g., team-specific pools).

Device plugins & GPU partitioning:

• Use the Kubernetes device-plugin (NVIDIA) to expose GPUs. Where supported, use MIG (Multi-Instance GPU) or MPS (Multi-Process Service) to partition GPUs for fractional sharing; otherwise allocate whole GPUs.
• Ensure kubelet exposes topology information (NUMA (non-uniform memory access)/socket) so scheduler can avoid remote memory penalties.

Avoiding fragmentation (binpacking vs spreading):

• Default to binpacking for GPU workloads: pack pods to fill GPUs on as few nodes as possible so you leave other nodes entirely free. This reduces leftover fractional GPUs and increases utilization.
• Use spreading only when you need fault tolerance or thermal/ power distribution: PodTopologySpreadConstraints can spread replicas across nodes/zones.
• Combine: prefer binpacking within a labeled GPU pool, and spread replicas across pools or AZs for resilience.

Gang scheduling and preemption:

• Use gang scheduling (Volcano, kube-batch, or Kubernetes Scheduler Framework + plugin) for multi-pod training jobs so all required GPUs are allocated atomically - avoid partial starts that block resources.
• Define PriorityClasses for job criticality. Enable preemption so high-priority jobs can evict lower-priority pods; couple with graceful termination and checkpoint/resume in ML code.
• Use PodDisruptionBudgets carefully to avoid blocking necessary preemption.

Fair sharing across teams:

• Put teams in namespaces and enforce quotas (ResourceQuota) for GPUs per namespace.
• Use a higher-level scheduler with fair-share queuing (Volcano with fair-share, YuniKorn, or K8s Scheduler + scheduler-plugins fairness) so queues are weighted by team shares, not pure FIFO.
• Complement with limit ranges and admission controller to prevent over-requesting.

Operational controls & safety:

• Enforce GPU requests/limits (requests must equal limits for GPUs). Validate via admission webhook.
• Monitor utilization (Prometheus + DCGM exporter) and tune binpacking thresholds.
• Provide preemption safeguards: grace period, checkpointing, or use checkpoint-capable frameworks (MPIOperator, Ray).
• Document SLAs and provide a priority/queue UI (e.g., Volcano's queue) so teams understand trade-offs.

Trade-offs summary:

• Binpacking maximizes utilization but increases blast radius on node failure; spreading increases resilience at cost of lower utilization.
• Whole-GPU allocation is simplest and less noisy; MIG lets more fine-grained sharing but needs hardware support and scheduler-awareness.
• Preemption plus gang scheduling requires orchestration tooling (Volcano or scheduler plugins) and disciplined checkpointing from teams.

Concrete stack recommendation:

• Node labeling + NVIDIA device-plugin (MIG where available)
• Taints/tolerations for GPU pools
• Volcano (or YuniKorn) for gang scheduling, fair-share queues and preemption control
• Namespace ResourceQuota + PriorityClasses + admission webhook enforcing gpu requests
• Monitoring with DCGM/Prometheus and autoscaling policies to add GPU nodes as needed

This combination avoids fragmentation, supports preemption and atomic allocations, and enforces fair sharing while giving teams predictable SLAs.

Liveness vs Readiness (brief):

• Liveness probe: answers “Is the container alive or should Kubernetes restart it?” Use to detect deadlocked or crashed processes.
• Readiness probe: answers “Is this pod ready to serve traffic?” Used by Service/Endpoints to add/remove pod from load balancing.

Concrete example for an ML inference container that loads a large model at startup:

• Readiness: should be false until model fully loaded (so traffic isn’t sent while warm-up happens).
• Liveness: should allow longer startup time to avoid killing the pod while it’s loading; once running, ensure the process hasn’t hung.

Example YAML snippets:

# readiness: check model-loaded endpoint
readinessProbe:
  httpGet:
    path: /health/ready   # returns 200 only after model loaded
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 5
  failureThreshold: 1
  timeoutSeconds: 2

# liveness: simpler, less aggressive, allow long startup
livenessProbe:
  httpGet:
    path: /health/live    # returns 200 if process responsive
    port: 8080
  initialDelaySeconds: 120   # allow model load time
  periodSeconds: 30
  failureThreshold: 3
  timeoutSeconds: 2

What happens if misconfigured:

• Readiness too quick/incorrect: pod receives traffic before model ready → request failures, high latency, user errors.
• Readiness too strict: pod never becomes ready → traffic never reaches it, reduced capacity.
• Liveness too aggressive (short initialDelay): kubelet restarts pod during model load → crash loop, degraded availability.
• Liveness too lax: dead process not detected → stale/failed pods remain serving bad responses.

Best practices: expose explicit endpoints for ready vs live, tune initialDelaySeconds to realistic model load times, implement a small warm-up health check that confirms model inference works (e.g., a cheap dummy prediction).

Kubernetes provides several autoscaling tools useful for ML inference; choosing and combining them depends on workload patterns (latency-sensitive vs batch).

Key autoscalers:

• Horizontal Pod Autoscaler (HPA): scales replicas based on CPU/memory or custom metrics (QPS (queries per second), latency, GPU utilization). Good for handling increases in concurrent requests. Works well with stateless model servers.
• Vertical Pod Autoscaler (VPA): adjusts container requests/limits (CPU/memory) to match observed usage. Helpful when a single replica needs more resources (heavy single-threaded model), but causes restarts when resizing - so less ideal for strict SLAs.
• Cluster Autoscaler (CA): adds/removes nodes when pods are unschedulable or nodes are underutilized. Essential when HPA increases replicas beyond node capacity (e.g., GPU nodes).
• Custom metrics-based autoscaling: feed application-specific signals (request latency P95, queue length, GPU memory pressure, model cold-start counts) into HPA via Metrics API or KEDA for event-driven scaling (Kafka, Pub/Sub).

Trade-offs:

• Latency-sensitive (real-time inference): prioritize pre-warmed capacity and conservative scaling thresholds to avoid cold starts. Use HPA with low scale-up latency, keep a minimum replica count, combine with VPA (for baseline sizing) and CA (to ensure node capacity). Prefer fast autoscaling paths (scale-up by replicas rather than resizing).
• Batch/jobs: tolerate higher startup latency; scale aggressively to minimize job completion time, rely more on CA to add nodes and HPA/KEDA to spawn workers. VPA can optimize per-pod resource efficiency.

Handling cold starts and warm pools:

• Keep a minimum replica floor (HPA minReplicas) tuned to baseline traffic.
• Warm pools / prewarmed pods: maintain a small set of idle but ready containers (provisioned with model loaded) or use pod “standby” pools. Implement readiness probes that mark pod ready only after model init.
• Use cached model artifacts (local SSD, memory-mapped weights) and lightweight model variants (smaller quantized models) for fast startup.
• Use fast node types (avoid spot/preemptible for critical latency) and use CA warm node pools (keep a few nodes always available).
• Leverage request buffering/queue with autoscaler on queue length (KEDA) to smooth bursts.

Best practices:

• Measure end-to-end P99/P95 latency and use those as autoscaling signals.
• Combine HPA (reactive), VPA (right-sizing offline or during low-traffic windows), and CA (ensure capacity) while enforcing min replicas/nodes for SLAs.
• Load-test scaling behaviors and tune cooldowns, stabilization windows, and scale increments to balance cost vs latency.

Also covers (folded from merged near-duplicates): bb122f13 folds the horizontal-vs-vertical-scaling framing and cost/cold-start trade-off; 87f81511 folds the DS-voice GPU-bound-pod autoscaling specifics (custom metrics, scheduler considerations).

Requirements (clarify):

• Functional: multi-model hosting, per-model isolation, A/B testing, autoscaling.
• Non‑functional: 500 concurrent requests across 100 models (~5 concurrent/model avg), p99 <= 200ms, support large models (cold-start risk).

High-level approach:

• Use Kubernetes with a model-serving layer (KServe or a custom serving mesh) + service mesh (Istio/Envoy) for traffic routing. Models are artifacts in an immutable model registry (OCI or S3) and pulled into serving pods. Autoscaling via KEDA/HPA with custom metrics.

Key Kubernetes objects and responsibilities:

• Namespace per team or tenant; label per model for policy.
• Deployment (or KServe InferenceService) per logical model version for strict isolation or multi-model Deployment for many small models. For large models prefer one-model-per-pod.
• HorizontalPodAutoscaler (HPA) driven by custom metrics (requests/sec, latency) or KEDA scaling on queue length / Kafka / Redis streams.
• VerticalPodAutoscaler (VPA) in recommend mode for right-sizing CPU/memory.
• PodDisruptionBudget and ResourceQuota per namespace for isolation.
• NodePools (nodeSelector / taints & tolerations) for different resource profiles (CPU, high-memory, GPU).
• ConfigMap/Secret for model config and credentials.
• PersistentVolume/CSI drivers or initContainers to fetch model artifacts for large models into hostPath/cache.

Artifact management:

• Model registry: push artifacts (model weights + metadata) as OCI images or versioned blobs in S3/MinIO. Include manifest with input/output schema, dependencies, and a warmup script.
• CI/CD: GitOps pipeline (ArgoCD) that updates InferenceService manifests on new model releases.
• Image-based models for fast distribution (container image with model baked in) for latency-critical large models; otherwise on-demand fetch with checksum verification.

Traffic routing and A/B testing:

• Use Istio VirtualService / DestinationRule or KServe traffic-splitting features.
• For A/B tests: deploy model A and B as separate services (or versions in one InferenceService) and configure weighted routing (e.g., 80/20). Use header-based routing for user-level bucketing (cookie or x-user-id hash) to ensure sticky sessions.
• Collect metrics per model/version via Prometheus and export traces via Jaeger. Evaluate accuracy/perf and automate weight shifts based on metrics (canary promotion pipeline).

Autoscaling strategy & resource sizing:

• With 500 concurrent across 100 models => average 5 concurrency/model. For p99 200ms target, provision minimal replicas per model = ceil(expected_concurrency / max_concurrency_per_pod). If a pod handles 10 concurrent requests at p99, start with 1 replica per model and HPA scale on observed concurrency.
• Use KEDA to scale down to zero for rarely used small models, but avoid zero for large models - use a minimum replica (warm pool) for large models.

Mitigating cold starts for large models:

• Warm pool: maintain a small number of pre-warmed pods per nodepool (NodePool autoscaling + DaemonSet or Deployment with minReplicas) that have large models loaded.
• Model preloading/warmup hooks: run a warmup init inside pod after model download to populate caches and mmap weights.
• Snapshot serialization: persist warmed model state (memory-mapped checkpoint) on local SSD or PV so restart is faster.
• Lazy load + async load: accept incoming requests with queuing and return "warming" responses for non-critical paths; or route first requests to warm replicas.
• Use GPU/CPU resource pools: share expensive GPU nodes with multi-tenant inference frameworks (e.g., Triton) to reduce per-model loading overhead.
• Bake very large models into images for low startup latency where update frequency is low.

Observability & safety:

• Per-model Prometheus metrics (latency percentiles, error rates), request logs, model drift metrics.
• Circuit breakers in Envoy to prevent cascading latency spikes.
• Automated rollback if p99 latency or error rate crosses thresholds.

Trade-offs:

• One-model-per-pod gives best isolation and simple scaling but higher resource usage. Multi-model pods are denser but increase cold-start complexity and noisy‑neighbor risk.
• Baking models into images reduces cold-start but increases CI/CD overhead and image size.

This design balances isolation, autoscaling, and experiment control while using warm pools, model baking and smart routing to meet a 200ms p99 latency target for typical model sizes.

Requirements & constraints:

• Zero-downtime rolling updates for a model server that needs warm-up time and must drain in-flight requests.
• Kubernetes Deployment with probes and lifecycle hooks to avoid dropped requests.

Manifest (key parts):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: model-server
spec:
  replicas: 3
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0     # never take replicas down -> avoid capacity loss
      maxSurge: 1           # allow one extra pod while updating
  minReadySeconds: 30     # ensure pod stays ready for warm-up period
  template:
    metadata:
      labels: { app: model-server }
    spec:
      terminationGracePeriodSeconds: 120
      containers:
      - name: model
        image: myregistry/model-server:latest
        ports: [{ containerPort: 8501 }]
        readinessProbe:
          httpGet:
            path: /ready        # returns 200 only after model warm-up & loaded
            port: 8501
          initialDelaySeconds: 5
          periodSeconds: 5
          failureThreshold: 6   # allows ~30s of transient failures
          timeoutSeconds: 2
        livenessProbe:
          httpGet:
            path: /healthy      # quick health check for process responsiveness
            port: 8501
          initialDelaySeconds: 30
          periodSeconds: 10
          failureThreshold: 3
        lifecycle:
          preStop:
            exec:
              command:
              - /bin/sh
              - -c
              - |
                # Ask server to stop accepting new requests and drain
                curl -sS -X POST http://127.0.0.1:8501/drain || true
                # give app time to finish in-flight requests
                sleep 60

Why these choices and how they avoid dropped requests:

• strategy.maxUnavailable: 0 + maxSurge:1 ensures new pods start before old ones are removed, preserving capacity during rollout.
• minReadySeconds enforces that a pod stays ready long enough for warm-up (e.g., model loading) before being considered available to the Service.
• readinessProbe points to an endpoint that returns success only after the model is fully loaded (warm-up). Kubernetes will not send traffic to a pod until this probe passes, preventing requests to a cold model.
• livenessProbe catches crashed/hung processes but has a longer initialDelay to avoid false positives during load time.
• terminationGracePeriodSeconds gives the container time to finish draining; preStop invokes an application-level drain endpoint to stop accepting new requests and then sleeps to allow in-flight requests to complete. The app should handle SIGTERM to stop accepting new connections immediately and complete ongoing work.
• Combined, these settings ensure new pods become READY before they get traffic and old pods stop receiving new traffic and finish in-flight requests before termination, preventing dropped requests during rolling updates.

Operational notes:

• Implement /ready, /healthy, /drain and proper SIGTERM handling in the model server (stop accepting new connections, finish active requests, flush metrics).
• Tune timeouts (minReadySeconds, sleep, failureThreshold) according to model load times and tail-latency SLAs.

Also covers (folded from merged near-duplicates): Also folds 81583691 (Helm chart templating structure for the same deployment) - no format loss, both design/system_design.