Performance and Load Testing Questions

Load testing

• Goal: Verify system meets expected concurrent user/session throughput and response-time SLOs under normal-to-peak traffic.
• Scenario: Simulate steady 5,000 concurrent shoppers browsing category pages, searching, and checking out during a seasonal promotion.
• Acceptance criterion: 95% of page requests complete within 800 ms and error rate ≤ 0.5% during a 1-hour steady-state run.

Stress testing

• Goal: Find breaking point and failure modes by pushing beyond capacity; validate graceful degradation and recovery.
• Scenario: Ramp concurrent users from 5k to 20k over 30 minutes to observe when checkout or DB connections fail.
• Acceptance criterion: System alerts trigger before total outage; failures are limited to non-critical services and mean time to recover (MTTR) ≤ 5 minutes after load drops.

Soak / Endurance testing

• Goal: Detect memory leaks, resource exhaustion, or performance degradation over long duration under nominal load.
• Scenario: Run 2,000 concurrent users performing mixed transactions continuously for 48 hours.
• Acceptance criterion: No >10% degradation in throughput or >30% increase in median response time compared to initial hour; no out-of-memory or resource leak incidents.

Spike testing

• Goal: Validate behavior under sudden, large traffic spikes and how autoscaling/queueing reacts.
• Scenario: Instant jump from 500 to 10,000 users for 5 minutes (flash sale) then back down.
• Acceptance criterion: Autoscaling adds instances and 90% of requests are served within 2× baseline latency within 3 minutes of the spike; error rate ≤ 2% during spike.

Each test includes monitoring (CPU, memory, DB connections, latency, error rates) and automated scripts (JMeter/Locust) plus post-run logs to identify root cause.

Brief answer / trade-offs

Reproducing full production traffic gives highest fidelity — catches scaling, contention, networking, and rare cascades — but is costly, time-consuming, and risky (environment setup, data privacy, potential service impact). Scaled-down or synthetic workloads are cheaper, faster, safer, and useful for frequent regression tests, but can miss non-linear behaviors and emergent bottlenecks.

How I decide which to run

• Use cheaper synthetic/scale-down for: CI smoke/perf regressions, early-stage tuning, verifying fixes, and components with linear scaling.
• Use full-scale tests for: major releases, architectural changes, capacity planning, SLA validation, or after fixes that previously only manifested at scale.
• Decision factors: risk (customer impact), change scope, historical incidents, cost/time budget, and compliance.

Validating approximation fidelity

• Calibrate synthetic models against production telemetry: request mix, session length, think-times, error patterns, geo distribution.
• Run intermediate “ramp” tests: progressively increase scale to observe non-linearities.
• Compare key KPIs between prod and test: p95/p99 latency, throughput, error rates, CPU/memory contention, DB locks.
• Use statistical similarity metrics (e.g., KL divergence on request distributions) and root-cause checks for any diverging metrics.
• Document assumptions and acceptance thresholds; if divergence exceeds threshold, schedule full-scale test.

This approach balances cost with risk while ensuring tests remain meaningful for releases.

Approach (QA perspective)
I would treat this as a reproducible performance test + observability task: collect GC logs and metrics, analyze key indicators, propose JVM tuning with rationale, then validate changes via controlled load tests.

Collect GC logs & metrics

• Enable logging on CI/dev environment:

# Java 11+ G1 example
-XX:+UseG1GC -Xlog:gc*:file=/var/log/app/gc.log:time,uptime,level,tags
-XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDetails

• Collect JVM metrics (jmap/jstat/JFR) and app latency (APM, Prometheus histograms).
• Record full-heap dumps when OOMs occur.

Interpret key GC metrics

• Pause times: look at max/p50/p99 stop-the-world durations; correlate with user-perceived latency spikes.
• Frequency: many short young GCs vs infrequent long full GCs—frequent short GCs may be OK; long old-gen collections cause spikes.
• Young/old gen ratios: high promotion rate → old-gen growth and eventual full-GC.
• Allocation rate and live-set size (from jstat/jcmd GC.heap_info).

Tuning recommendations

• If large stop-the-world full GCs: tune G1
  • Set -Xms = -Xmx to avoid resizing during load.
  • Increase heap (-Xmx) if live-set approaches max.
  • G1 flags:

-XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:InitiatingHeapOccupancyPercent=35 -XX:G1ReservePercent=10
-XX:G1HeapRegionSize=8m  # if heap > 8GB, choose 8m/16m to reduce bookkeeping

• If promotion rate high: increase young generation fraction (via G1NewSizePercent/G1MaxNewSizePercent) to reduce promotions.
• For throughput-sensitive cases consider CMS/ZGC/Epsilon depending on JDK and constraints.

Load-test design to validate changes

• Reproduce production-like allocation/throughput using tools (Gatling, JMeter, k6) and workload generators that exercise specific code paths.
• Scenarios:
  • Baseline steady-state at normal RPS
  • Spike test (sudden +3x RPS)
  • Long soak test (24–72 hours) to catch fragmentation/leaks
• Measure: request latency percentiles (p50/p95/p99), GC pause histograms, GC frequency, CPU, heap usage, safepoint times.
• Run A/B tests by swapping JVM flags and compare metrics; require statistical significance on p99 latency and max pause time.

QA responsibilities

• Automate GC log collection and parsing in CI (use gcviewer/gcparser or convert to JSON).
• Create test assertions (e.g., p99 latency < SLO, max pause < threshold).
• Document findings and recommended safe flags for dev to implement.

Clarify requirements & constraints

• Target: automatic detection of performance regressions per PR/build for microservices.
• Goals: fast feedback (PR-level), broad coverage (nightly), low false positives, clear triage info for devs.

High-level architecture

• Instrumentation: Prometheus for metrics, OpenTelemetry for traces/logs, private APM (Jaeger/Zipkin) and service-side SDKs emitting spans and service metrics.
• Collection pipeline: exporters → metrics DB (Prometheus/Thanos) + traces store (Jaeger/Elasticsearch) → feature store for regression engine.

Baselining & test selection

• Baselines: rolling-window baselines per metric + environment tag (canary, staging), compute median, p50/p95, variance over N successful builds (e.g., 30).
• Per-PR vs nightly: lightweight synthetic scenarios (smoke, critical user flows) run per-PR with small load / deterministic data; heavier load/perf scenarios (stress, soak, end-to-end multi-service) scheduled nightly or on demand.

Detection techniques

• Statistical tests: use paired non-parametric tests (Wilcoxon) comparing current build samples to baseline; apply effect-size (Cliff’s delta) and confidence intervals.
• Time-series anomaly detection: EWMA + seasonality decomposition; change-point detection (CUSUM or Bayesian online) for sudden shifts.
• Combine with trace-based amplification: if metric regresses, correlate to traces to find service/span-level latency and error spike.

Noise reduction

• Environment normalization: compare only same-canary/stage labels, normalize for traffic and concurrency.
• Outlier trimming: use robust metrics (median, trimmed-mean), remove warmup windows.
• Multiple hypothesis correction: Benjamini-Hochberg to limit false positives across many metrics.

Actionable reporting

• Triage payload: concise card per regression with metric, delta %, confidence, affected services/spans, example trace IDs, top contributing endpoints and SQL/DB calls, reproducible test scenario & command, build/commit links.
• Prioritization: severity scoring (user-impact, regression magnitude, frequency) and suggested owner.
• Delivery: PR comments + Slack/Teams alert for blocking regressions; nightly dashboard with trendlines, diffs, and drill-down to traces and raw samples.

Operational concerns

• Storage: downsample older metrics, keep raw samples for recent builds.
• Feedback loop: allow devs to mark false positives; retrain/update baselines.
• Metrics to track for the system: detection precision/recall, time-to-alert, alert rate.

This design balances fast PR-level feedback with deeper nightly analysis, uses robust statistics plus trace correlation to make findings actionable for developers.

Situation & goal
As QA I’ll validate and quantify which option reduces p95 latency and sustains throughput with acceptable risk and time-to-value.

Benchmark approach

• Baseline: record p95, throughput, per-instance CPU, error rates, load patterns (peak and steady).
• Create reproducible load test (same scenarios, seed data) that reflects production traffic.
• For A (scale): incrementally add instances ( +1, +2) and run tests to measure p95, throughput, CPU per instance and cost = n * X/month.
• For B (opt): instrument hotspots, run profiling-guided optimizations in a feature branch; run before/after tests to measure delta. Track engineering time Y person-weeks.

Estimate performance gains

• Use observed deltas from tests:

gain_percent = (baseline_p95 - new_p95) / baseline_p95 * 100

• Example: scaling 2x instances reduces p95 from 800ms to 450ms → gain 43.75%.

ROI over 12 months

• Costs:

cost_scale_12m = n_instances * X * 12
cost_opt_12m = (Y_person_weeks * weekly_salary) + negligible infra delta

• Benefits: avoided operational cost of failures, SLA penalties, improved conversions (quantify if available).
• Net ROI = (monetary_benefit - cost) / cost

Explain the numbers with a sensitivity table (pessimistic/expected/optimistic).

Recommendation & risks

• If B yields ≥ target gain and Y <= 4 weeks, recommend optimize first (one-time, lower long-term cost), QA-owned acceptance criteria and regression suite required.
• If quick time-to-value needed or B uncertain, choose A for immediate relief while parallelizing B (short-term + long-term plan).
• Risks: optimization may introduce regressions (mitigate via comprehensive regression tests, canary deploys). Scaling risks: cost creep, autoscaling flapping (mitigate with autoscale rules, alerts).

Deliverables for stakeholders

• Benchmark report (metrics, test scripts), ROI table, recommended path with timeline (e.g., immediate +1 instance, 6-week optimization sprint), risk mitigations, and go/no-go criteria (p95 target, error rate threshold).