Situation: We're validating that our inference cluster (Kubernetes + GPU nodes + model-serving stack) degrades gracefully and recovers when GPUs fail, services get slow, a replica has a corrupted model artifact, or network partitions occur.
Experiment design (high-level):
- Tools: LitmusChaos/Chaos Mesh, kubectl, tc/iptables, Prometheus/Grafana, SLO/alerting, model-artifact checksums, deployment rollouts, service-mesh (Istio) for traffic control.
- Scope: Namespace-scoped chaos on a canary inference deployment behind an ingress and traffic-split (Istio/VirtualService).
Fault injections:
- GPU node failure: cordon + drain selected GPU node; or simulate via litmuschaos aws-terminate/chaosd or kill kubelet on that node.
- Inter-service latency: inject network delay (tc netem) between ingress ↔ model-service and model-service ↔ feature-store for Xs at 95th/99th percentile.
- Corrupted model artifact: replace model file with invalid checksum on one replica (or change symlink) to simulate load-time error.
- Network partition: iptables rules to simulate split between subset of pods or between AZs for 5–15 minutes.
Blast-radius controls:
- Namespace-limited; target only canary deployment; max 10–20% of replicas/nodes; run during business off-peak window; automatic timeout (max 15 minutes) per experiment; requires two-person manual approval to expand.
KPIs to monitor:
- Latency: p50/p95/p99 inference latency
- Success rate / error rate (4xx/5xx)
- Throughput (req/s)
- Queue/backlog length (async queues)
- GPU utilization and memory errors (dmesg)
- Pod restarts and crashloop counts
- Model-serving health probes (readiness/liveness)
- Model correctness: sample-request accuracy or checksum mismatch alerts
- End-to-end SLA: % requests within SLO
Safe abort conditions (auto-abort if any triggers):
- Error rate increase > 5% absolute and sustained >2 min
- p99 latency > 2x baseline and sustained >2 min
- Total cluster GPU capacity drop below threshold (e.g., remaining GPUs < 50% of baseline)
- Any production-critical downstream service reports P0
- Unexpected node/pod terminations beyond target scope
Staging vs limited production differences:
- Staging: full-scale scenarios allowed; synthetic traffic only; no customer impact; longer durations; focus on root-cause data collection.
- Limited production (canary): strict blast radius (<=20% traffic), automated rollback path, run during defined window, pre-notified on-call and stakeholders, shorter faults (1–5 min), close monitoring and warm standby capacity.
Expected recovery & rollback procedures:
- Automatic:
- Chaos experiment has timeout and will remove iptables/tc rules; LitmusChaos triggers remediation (uncordon node, restart pod, replace corrupted artifact).
- Service mesh shifts traffic off failing canary via virtual service weights.
- HorizontalPodAutoscaler / KEDA spin up warm replicas if configured.
- Manual (if auto fails or safe-abort triggered):
- Shift 100% traffic to stable revision (istio route or feature-flag)
- kubectl rollout undo deployment/model-server@revision
- Recreate corrupted replica from artifact repository (validate checksum), restart pods, verify readiness probes
- If node hardware failure: cordon + drain remaining workloads, provision replacement GPU node (cloud/infra), or shift to other AZ.
- Post-incident: run model validation job on restored replica, check model accuracy on sample dataset, ensure metrics back to baseline before reintroducing traffic.
Observability & postmortem:
- Record traces, request ids, GPU logs, kernel OOM/dmesg, container logs, and artifact hashes.
- Run automated runbook checklist for operator to follow.
- Post-run: compare KPIs vs baseline, document failure modes, update SLOs/remediation automation, and decide whether to harden probes, add circuit breakers, backpressure, or increase replica count.
Why this approach:
- Controlled, incremental blast radius prevents customer impact.
- Clear KPIs and auto-abort reduce risk.
- Combining simulation (latency/partition) with data-corruption tests both infrastructure and model-ops robustness.
- Staging vs limited-prod policies ensure learnings are validated safely before wider rollout.
