Covers design and execution of tests that measure how software behaves under varying levels of user concurrency and resource demand, including load testing, stress testing, soak testing, and spike testing. Includes key performance metrics such as response time, throughput, latency, error rates, and resource utilization and how to collect and interpret these signals. Explains common tooling and approaches for load generation and results analysis, for example JMeter, Gatling, and LoadRunner, and how to instrument systems for monitoring and tracing. Addresses testing at scale, including distributed load generation, test environment configuration, test data management, and identifying and diagnosing performance bottlenecks across application, database, and infrastructure layers. Describes how to integrate performance testing into the development lifecycle and continuous integration and continuous delivery pipelines, how to report findings and performance regressions to stakeholders, and how functional correctness concerns interact with performance objectives.
EasyTechnical
63 practiced
Explain key performance metrics used in load testing: response time (including percentiles such as p95/p99), throughput, latency, error rate, and resource utilization (CPU, memory, disk I/O, network). For each metric describe what it measures, one limitation of using it alone, and one practical tool or agent you would use to collect it in a distributed microservice environment.
Sample Answer
**Overview (QA perspective)** As a QA Engineer I use these metrics to validate service SLA and identify bottlenecks during load tests.**Response time (including p95/p99)** - What: Time from request start to response completion; percentiles (p95/p99) show the value below which 95%/99% of requests fall. - Limitation: Mean can hide long tails; even p95 misses the worst 1–5% issues. - Tool: JMeter or Gatling for generating load + Grafana/Prometheus to record percentiles.**Throughput** - What: Requests processed per second (RPS) or transactions/sec — system capacity under load. - Limitation: High throughput may coincide with poor per-request latency; doesn’t reflect user experience. - Tool: Locust for load generation; Prometheus for metric collection.**Latency** - What: Network or processing delay for a single hop (often measured at client or service entry). - Limitation: Network latency and server processing are conflated if not instrumented separately. - Tool: OpenTelemetry agents (tracing) or Jaeger to separate and measure spans.**Error rate** - What: Percentage of failed requests (HTTP 5xx, timeouts, exceptions). - Limitation: Low error rate can hide degraded performance (slow but successful responses). - Tool: Application logs + Prometheus/Grafana alerts; Sentry for aggregated errors.**Resource utilization (CPU, memory, disk I/O, network)** - What: Host/container-level metrics showing resource consumption and saturation. - Limitation: High utilization doesn’t prove causation—needs correlation with latency/throughput. - Tool: node_exporter / cAdvisor + Prometheus, visualized in Grafana; Kubernetes metrics-server for containerized environments.When reporting, I correlate these metrics (latency percentiles vs throughput vs CPU) to pinpoint root causes and recommend fixes.
HardSystem Design
63 practiced
Design a performance test to validate Kubernetes HPA/VPA behavior for a stateless service. Define the test steps to trigger scaling using CPU or a custom metric, list the metrics to collect (pod count, CPU utilization, pod startup/readiness times), explain handling of scaling cooldowns, and describe how to verify that traffic is balanced across new pods and that race conditions during scale-up/down are detected.
Sample Answer
**Overview (QA perspective)** I would create an automated performance test that exercises both HPA (CPU/custom metric) and VPA behavior for a stateless service, validates timing, and detects race conditions during scale events.**Test prerequisites & tools** - Kubernetes cluster with Metrics Server + Prometheus + kube-state-metrics - Load generator (k6) and Prometheus alerts/queries - Test harness (pytest or bash) to run scenarios and collect artifacts (kubectl, /metrics endpoints)**Test steps** 1. Deploy the stateless service with readinessProbe and livenessProbe; record initial replica count. 2. Baseline: run low load for 2min; capture pod_count, pod_cpu, pod_mem, pod_ready. 3. Trigger CPU-based scale-up: run sustained CPU load (k6 HTTP requests with expensive endpoint) until HPA target reached; keep load until new replicas stabilize. 4. Trigger custom-metric scale-up: push metric via custom exporter (e.g., request_per_second) and run matching load. 5. Scale-down: reduce load abruptly to test cooldown and VPA interactions. 6. Flapping test: alternate high/low load every 30s to expose race conditions and flapping. 7. Observe VPA recommendations and avoid conflicting controllers (set VPA to “recommendation” mode if testing HPA).**Metrics to collect** - pod_count over time (kubectl get hpa, kube-state-metrics) - per-pod CPU utilization (Prometheus: container_cpu_usage_seconds_total) - pod startup time = timestamp(pod scheduled) -> timestamp(container Ready) - pod readiness probe success latency and failure counts - per-pod request count / latency (nginx prom endpoint or app metrics) - HPA events and stabilizationWindow logs (kubectl describe hpa) - Node resource pressure/evictions**Handling scaling cooldowns & timing** - Read HPA’s stabilizationWindow and cooldown settings; adjust test durations to exceed them so scale decisions occur naturally. - For testing immediate reactions, temporarily set short stabilizationWindow in a controlled namespace. - Capture timestamps of HPA decision vs. actual replica change to measure propagation/cooldown.**Verifying traffic balancing** - Instrument service to export per-pod request counters; validate request distribution is near-uniform after new pods Ready. - Use Prometheus queries to compare per-pod request_rate and latency. - Verify readiness: only send significant traffic to pods that are Ready; check absence of 5xxs during startup window.**Detecting race conditions during scale-up/down** - Correlate events: HPA decision times, pod creation, readiness, and incoming requests. Look for: - Requests routed to not-yet-ready pods (increased 5xx/connection errors) - Overprovisioning loops (HPA triggers scale-up while pods are still starting) - Rapid scale oscillation tied to metric sampling — detected by frequent replica churn and HPA events. - Automated assertions: fail if >X% requests hit pods with Ready==False or if replica changes >Y times/minute. - Capture full event logs and Prometheus histograms for post-mortem.**Expected outcomes & pass criteria** - HPA/VPA scale decisions align with configured targets within acceptable latency (e.g., scale-up within stabilizationWindow + startup_time) - Traffic distribution across Ready pods is balanced (per-pod request variance <25%) - No request spike of errors attributable to pods not Ready; no excessive oscillation.This test plan is automatable and repeatable; results feed bug reports with timestamps, Prometheus queries, and pod event dumps for developers to reproduce and fix.
EasyTechnical
111 practiced
In Apache JMeter, what is a Thread Group and how does it affect load generation? Describe how you'd configure a Thread Group to simulate 200 concurrent users executing a login → browse → checkout scenario for 15 minutes, specifying number of threads, ramp-up, loop count (or scheduler), and timers. Explain why ramp-up and timers are important for realistic tests.
Sample Answer
**What is a Thread Group & effect on load**A Thread Group in JMeter defines the virtual users (threads), how quickly they start, and how many iterations they run. It directly controls concurrency and request arrival patterns — more threads = more simultaneous requests; ramp-up and timers shape realism and avoid artificial spikes.**Configuration to simulate 200 concurrent users for 15 minutes**- Number of Threads (users): 200 - Ramp-Up Period: 120 seconds (start ~1.67 users/sec) — smooth start avoids sudden load spike on SUT - Scheduler: enabled; Duration = 900 seconds (15 minutes) — ensures test runs exactly 15 min - Loop Count: set to Forever (or large) and rely on Scheduler to stop, OR calculate loops per thread if deterministic behavior needed**Timers / Think-time**- Add a Uniform Random Timer or Constant Timer between samplers: - Login → think 2–4s (simulate credential entry) - Browse → think 1–6s between page requests - Checkout → think 3–8s for form filling/confirmations- Optionally add a Constant Throughput Timer if you want to cap global requests/sec.**Why ramp-up and timers matter**- Ramp-up prevents unrealistic instantaneous peak load that can misrepresent system behavior or cause cascading failures unrelated to typical usage. - Timers emulate human “think-time” and create realistic inter-arrival times; without them you generate unrealistically high request rates and skew response-time metrics.This setup yields 200 concurrent realistic users for 15 minutes while avoiding artificial spikes and producing actionable performance data.
HardTechnical
74 practiced
The security team plans to enable WAF rules, TLS termination at the load balancer, and stronger at-rest encryption. Explain how each of these changes could impact performance, how you would design tests to quantify the impact (before/after comparisons), and propose mitigations if performance degrades (e.g., TLS offload, selective WAF rule tuning, caching).
Sample Answer
**Situation & summary**As a QA engineer I’d evaluate three planned security changes: WAF rules, TLS termination at the ALB, and stronger at‑rest encryption. Each can add CPU/latency/IO overhead; my goal is controlled before/after measurements and actionable mitigations.**Performance impacts**- WAF rules: extra request inspection -> increased per-request latency and CPU on the WAF/ALB; complex rules (regex, rate-limits) amplify cost.- TLS termination at load balancer: TLS handshake and encryption/decryption shift to ALB -> increased ALB CPU but reduced app server CPU and potentially fewer long-lived TLS handshakes if session resumption used.- Stronger at‑rest encryption: higher CPU for encryption/decryption during reads/writes; increased IO latency and throughput drops for DBs or object stores.**Test design (quantify before/after)**- Metrics: p95/p99 latency, median latency, throughput (req/s), CPU, memory, network, disk IOPS, error rate.- Controlled load tests: baseline load (e.g., steady 500 rps + spike tests) using JMeter / k6; run identical scenarios before and after changes.- Synthetic real‑user traces: replay production traffic with a traffic-capture tool to measure realistic behavior.- Microbenchmarks: measure TLS handshake times, WAF rule processing time, DB read/write latency with and without encryption.- Environment: use staging identical to prod (ALB, WAF, DB), run multiple iterations, capture metrics via Prometheus/Grafana and ALB/WAF logs.- Analysis: compute delta in p95/p99 and CPU utilization; use statistical significance (t-test or confidence intervals).**Mitigations if degraded**- TLS: enable TLS offload to dedicated ALB instances or use hardware TLS accelerators / dedicated proxy; enable session resumption and OCSP stapling; tune cipher suites for faster symmetric ciphers.- WAF: selectively disable or rework expensive rules, move some detections to async processing, use rate-limiting rules to reduce load, or tier rules (enable strict rules only for suspicious paths).- At‑rest encryption: use envelope encryption (KMS) to reduce per‑operation crypto; cache decrypted data safely in memory with strict TTL; provision more CPU or use DB instances with crypto acceleration.- General: autoscaling policies, connection pooling, CDN caching for static content, and layered monitoring/alerting.**Outcome & collaboration**I’d present measured deltas, recommend targeted mitigations, and validate fixes with repeat tests. I’d collaborate with security/infra to balance security and performance with data from our tests.
HardTechnical
68 practiced
How would you design tests to measure the impact of increased network latency and packet loss between services (for example, between web and DB tiers) on application performance? List tools to inject network conditions, key metrics to observe, likely application behaviors you'll see, and mitigation strategies you would test (e.g., retry/backoff, caching, protocol tuning).
Sample Answer
**Approach summary**Design controlled experiments that inject latency/packet loss at the network layer between tiers, run representative workloads (functional + load), and compare baseline vs degraded runs. Measure user-impact and system-level metrics, then validate mitigations.**Tools to inject conditions**- tc / netem (Linux) — latency, loss, duplication, reordering- Pumba / Chaos Mesh / LitmusChaos — container-level chaos for k8s- Toxiproxy — proxy-based fault injection for services- Istio Fault Injection — service-mesh scenarios- WANem / Chaos Monkey for network — lab/VM appliances**Test design**- Baseline run → incremental latency steps (10ms, 50ms, 200ms, 500ms) and loss rates (0.1%, 1%, 5%, 10%)- Workloads: synthetic API calls, realistic traffic replay (JMeter, k6), DB-heavy transactions, background jobs- Repeat runs, isolate single-variable changes, test peak and steady-state**Key metrics to observe**- End-to-end latency (P50/P95/P99), throughput (req/s)- Error rates (HTTP 5xx, DB timeouts), retry counts- Resource metrics: DB connection pool usage, thread/goroutine queueing, CPU, memory- Latency breakdown (app vs network vs DB), tail latencies, SLA violations**Likely behaviors**- Increased tail latency and more timeouts- Connection pool exhaustion and cascading failures- Higher retry storms amplifying load- Circuit breaker trips and degraded functionality**Mitigation strategies to test**- Client-side: exponential backoff with jitter, idempotent retries limits- Caching: read-through caches, TTL tuning, cache warmup- Protocol tuning: lower TCP timeouts, keepalives, increased DB connection pool size- Resilience patterns: bulkheads, circuit breakers, request hedging- Load shedding and graceful degradation tests**Validation**- Verify mitigations reduce SLA breaches and resource saturation under same injected conditions and measure trade-offs (stale cache, added complexity).
Unlock Full Question Bank
Get access to hundreds of Performance and Load Testing interview questions and detailed answers.
