Non-Functional Testing Questions

Requirements & constraints:

• Detect memory/goroutine leaks before/after deploy at scale; low false positives; automated rollback for canaries; actionable post-deploy analysis.

High-level strategy:

1. Static checks (pre-merge)

• Run govet, staticcheck, gosec, revive in CI. Flag common leak patterns: forgotten ticker.Stop(), unclosed channels, incorrect context use.
• Enforce code review checklist for goroutine lifecycle and context cancellation.

2. Unit / integration tests with pprof (CI)

• Add deterministic unit tests that create goroutines and assert cleanup (use testing/quick and leaktest-like helpers).
• Integration/load tests: spin multiple instances locally/containerized with realistic workloads; take baseline and steady-state profiles (heap, goroutine, thread, cpu).
• Capture profiles periodically via net/http/pprof or runtime/pprof and record:
  • heap (alloc/total/objects)
  • goroutine count & stacks
  • GC pause / next GC
• Example commands to capture profiles:

# capture 30s heap profile
curl http://canary:6060/debug/pprof/heap > heap.prof
go tool pprof -png heap.prof > heap.png
curl http://canary:6060/debug/pprof/goroutine?debug=2 > goroutines.txt

3. CI thresholds & gating

• Baseline profiles from a stable commit recorded in CI artifact storage.
• Define automated regressions checks comparing new profile vs baseline:
  • HeapAlloc increase > 15% AND absolute > 50MB → fail
  • Live object count increase > 20% → fail
  • Goroutine count increase > 10% and > 50 goroutines absolute → fail
• Use pprof diff scripts (pprof --proto + custom analyzer) to detect new long-lived stacks.

4. Canary deployment & monitoring

• Deploy to canary group (1–5% traffic, or fixed small pool). Collect metrics at 30s granularity:
  • RSS (resident memory)
  • HeapAlloc, HeapSys
  • Goroutine count
  • GC pause 95th percentile
  • Request latency (P99), error rate
  • CPU
• Automated detection (sliding windows):
  • Rapid growth: RSS slope > 5 MB/min for 10 minutes → warn
  • Sustained growth: RSS increase > 15% over 30 minutes → rollback
  • Goroutine slope > 20 goroutines/min for 10 minutes OR absolute goroutine count > baseline + 200 → rollback
  • GC pause increase (P95) > 2x baseline and correlates with memory growth → rollback
  • Any P99 latency regression > 30% or error rate spike > 1% absolute → rollback
• Use orchestration hooks to auto-roll back on trigger; mark deployment as failed and escalate.

5. Alerting & escalation

• Alerts tiered: warning (investigate) vs critical (automated rollback + paged).
• Attach recent pprof snapshots and metric time-series to alert.

6. Post-deploy analysis / SRE runbook

• If rollback triggered or warning persists:
  • Collect 3 snapshots (t0, t+10m, t+30m) of heap/goroutine profiles and pmap/ps.
  • Use go tool pprof to compare top allocations and top goroutine stacks. Look for newly introduced stack traces holding objects or blocked goroutines.
  • Generate flamegraphs and object growth over time.
  • Identify root cause: leaked channel/goroutine waiting on external resource, large cache growth, unintended retention (e.g., global slices, map keys).
  • Mitigation: patch to close/cancel, add context timeouts, bound caches with LRU + eviction, add finalizers only when necessary.
  • Postmortem: quantify impact (memory delta, latency, users affected), timeline, and preventive measures.

7. Continuous improvement

• Add synthetic long-run canary job that runs daily for 24–48 hours to catch slow leaks.
• Maintain a benchmark baseline per release and retain profile history.
• Regularly review configured thresholds based on observed cluster variability.

Why these numbers:

• Thresholds balance sensitivity and noise: % and absolute checks prevent failing on small services with tiny baselines or on large services where percentage is misleading.
• Use slopes (MB/min or goroutines/min) to detect ongoing leaks vs transient spikes from cold-start GC.

This strategy automates early detection (static + CI), catches regressions in controlled canaries, triggers safe rollbacks, and provides concrete post-deploy steps to root-cause and fix leaks.