Clarify requirements & goals
- Continuous CI-friendly load tests that measure latency percentiles (p50/p95/p99), error rates, throughput and correlate with infra metrics (CPU, memory, network).
- Automatic deployment blocking when SLA breached.
- Control cloud cost of load generators.
High-level architecture
- CI job (GitHub Actions/Jenkins/GitLab) triggers test orchestrator.
- Orchestrator spins ephemeral load-generators (k6 or JMeter in container) in cloud autoscaling group or Kubernetes Job.
- Application and infra export metrics to Prometheus; load test emits metrics via StatsD/Prometheus or k6 cloud.
- Central Prometheus + long-term TSDB (Thanos/Cortex) and Grafana for dashboards and alerts.
- Results aggregator (small service) reads Prometheus + test summary, computes SLAs and stores artifacts in object storage.
Test flow
- CI builds image → deploy to a short-lived test environment (canary).
- Run k6 script (or distributed JMeter) with scenario ramps; k6 pushes metrics to Prometheus Pushgateway.
- Prometheus scrapes both app and load-generator metrics. Aggregator queries Prometheus API for percentile histograms, error rates, throughput and infra metrics for same time window, correlates spikes (example: p99 latency vs CPU).
- Aggregator posts pass/fail to CI.
Automatic blocking
- Define SLAs (e.g., p95 < 200ms, error rate < 0.1%). Aggregator evaluates after test window.
- If SLA breached: aggregator returns non-zero exit code to CI and triggers rollback or blocks merge via pipeline status check and sends alert to Slack/Teams + ticket.
- Use GitOps checks: admission controller or branch protection prevents promotion when pipeline status is failing.
Cost control for load generators
- Use small-to-medium instance types with autoscaling and horizontal distribution (K8s HPA).
- Reuse burst capacity (spot/preemptible VMs) with fallback to on-demand.
- Limit test duration; use progressive ramp-ups and traffic shaping to minimize total vCPU-hours.
- Cache and reuse container images; tear down immediately on completion.
- Set quotas per pipeline and quota enforcement in orchestrator.
Observability & correlation
- Instrument application with histograms (Prometheus client) for accurate percentiles. Use exemplars linking traces (OpenTelemetry) to metrics for deep correlation.
- Dashboards show latency percentiles alongside CPU/memory/network and request rate; provide automated anomaly detection (Prometheus recording rules + alertmanager) for unusual patterns.
Trade-offs
- k6 is lightweight and CI-friendly; JMeter better for complex protocols. Distributed tests add orchestration complexity and cost. Spot instances reduce cost but increase flakiness—use only for non-blocking exploratory runs.
Metrics & validation
- Store raw test artifacts (k6 JSON, JMeter JTL) and Prometheus queries for audit. Run smoke load-tests on every PR, full scale in nightly/regression gates.
This design delivers end-to-end automated load testing integrated into CI, enforces SLAs automatically, correlates app and infra metrics for root-cause, and applies practical cost controls for load generators.
