Start by defining clear goals: which resource exhaustion modes you’ll simulate (memory, file descriptors, CPU, disk) and what “acceptable” behavior is (graceful degradation, correct error codes, safe shutdown, no data corruption). For each mode, implement reproducible experiments, assertions, and CI integration.
- Reproduce in isolation
- Containers/cgroups: run service in Docker/K8s with resource limits.
- Memory: docker run --memory=256m ... ; or k8s pod spec resources.limits.memory.
- FDs: setulimit -n 256 inside container or use docker run --ulimit nofile=256:256.
- CPU starvation: docker run --cpus=0.5 or cgroups cpu.quota/cpu.period; use stress-ng --cpu N to consume.
- Disk full: create small tmpfs or loopback file and mount; or pre-fill with fallocate: fallocate -l 99% /data/disk.img then mount.
- Tools: stress-ng (cpu/mem/io), dd/fallocate, ulimit, lsof to monitor FDs.
- Test scenarios and scripts
- Memory-pressure test: spawn background allocators (stress-ng --vm) in same cgroup until OOM; assert process either continues with degraded features or exits cleanly with logs indicating OOM handled.
- FD-exhaustion test: open many sockets/files from a helper process until accept()/open() returns EMFILE; assert the server returns 503 or throttles new connections and logs clear error; ensure existing connections continue.
- CPU-starvation test: saturate CPUs and run timed requests; assert latency percentiles (p50/p95/p99) remain within SLOs or that the circuit-breaker trips and returns controlled errors rather than timeouts or corrupted responses.
- Disk-full test: fill disk and attempt writes (uploads, DB writes); assert app surfaces clear 507/5xx with durable retry/backoff logic, no partial writes, and data integrity verified.
- Assertions to add
- Functional: health-check endpoint responds (200 or degraded 200) or returns expected degraded status (e.g., 200 with {"status":"degraded"}).
- API-level: for new requests assert expected error codes (503, 507) and stable JSON error schema.
- Correctness: no partial/garbage files; verify checksums of persisted data before/after.
- Observability: logs contain specific messages (OOM handler, FD limit reached), metrics emit degradation flags, and alerts would fire (metric thresholds).
- Resource invariants: process didn’t leak FDs (open FDs count stabilized), memory usage bounded, no unbounded spawning.
- Verify graceful shutdown/safe paths
- Send SIGTERM under exhaustion; assert shutdown completes within timeout, drains in-flight requests or returns 503, flushes state, and exits with zero/expected code.
- Verify restart behavior: after resource restored (increase disk space, free FDs), app recovers automatically or with restart policy.
- Integration test: simulate long-running write interrupted by disk full — assert rollback/transactional consistency.
- CI practices
- Keep heavy stress tests in nightly/weekly pipelines; add lightweight smoke tests for common failure modes on PRs.
- Use reproducible fixtures (Dockerfiles, scripts) and test harness that manipulates cgroups/ulimits inside CI agents or runs in a privileged job that can mount loopback images.
- Fail CI only on assertions about correctness/data loss; degrade noncritical SLO checks to warnings to avoid flakiness.
- Capture artifacts: logs, metrics snapshots, core dumps (if any) for postmortem.
- Example commands (quick)
- Limit FDs: docker run --ulimit nofile=256:256 myapp
- Memory stress: docker exec $C pid=$(pgrep myapp); docker exec $C stress-ng --vm 2 --vm-bytes 200M --timeout 30s
- Fill disk: fallocate -l 5G /mnt/loop.img; mkfs.ext4 /mnt/loop.img; mount -o loop /mnt/loop.img /data; dd if=/dev/zero of=/data/fill bs=1M
Finally, automate verification: scripts that run scenario -> wait -> run assertions (HTTP checks, DB integrity queries, open-FD counts via /proc/$pid/fd) and report pass/fail. Document expected behaviors and thresholds so engineers and SREs know whether behavior is acceptable or a bug.
