Performance and Load Testing Questions

Approach: use asyncio to schedule worker coroutine(s) that send requests at a target QPS, control concurrency with an asyncio.Semaphore, stop after N seconds, and record per-request latency and status. For percentile aggregation at the end, keep a thread-safe list (or use reservoir sampling / t-digest for streaming large runs). Below is a concise, correct core design.

import asyncio
import time
import aiohttp
from collections import Counter
import statistics

class LoadGenerator:
    def __init__(self, endpoint, qps, duration_s, concurrency):
        self.endpoint = endpoint
        self.qps = qps
        self.duration = duration_s
        self.semaphore = asyncio.Semaphore(concurrency)
        self.latencies = []          # list of floats (ms)
        self.status_counts = Counter()
        self._stop = asyncio.Event()

    async def worker(self, session):
        """Send a single request while respecting concurrency control."""
        async with self.semaphore:
            start = time.perf_counter()
            try:
                async with session.post(self.endpoint, json={"input":"data"}) as resp:
                    await resp.read()  # ensure body consumed
                    status = resp.status
            except Exception:
                status = 'error'
            elapsed_ms = (time.perf_counter() - start) * 1000
            self.latencies.append(elapsed_ms)
            self.status_counts[status] += 1

    async def run(self):
        """Drive workers to achieve approximate QPS for duration."""
        interval = 1.0 / self.qps  # seconds between requests
        start_time = time.perf_counter()
        async with aiohttp.ClientSession() as session:
            next_send = start_time
            while time.perf_counter() - start_time < self.duration:
                # schedule worker
                asyncio.create_task(self.worker(session))
                next_send += interval
                # sleep until next send time (allow bursts if behind)
                await asyncio.sleep(max(0, next_send - time.perf_counter()))
        # wait for inflight tasks to finish
        await asyncio.sleep(0)  # yield
        # optionally wait until semaphore fully released:
        while self.semaphore._value != self.semaphore._bound_value:
            await asyncio.sleep(0.01)

    def summary(self):
        if not self.latencies:
            return {}
        lat_sorted = sorted(self.latencies)
        return {
            "count": len(lat_sorted),
            "p50_ms": statistics.median(lat_sorted),
            "p90_ms": lat_sorted[int(0.9*len(lat_sorted))-1],
            "p99_ms": lat_sorted[int(0.99*len(lat_sorted))-1] if len(lat_sorted)>=100 else lat_sorted[-1],
            "mean_ms": statistics.mean(lat_sorted),
            "status_counts": dict(self.status_counts),
            "error_rate": (self.status_counts.get('error',0) + sum(v for k,v in self.status_counts.items() if isinstance(k,int) and k>=500)) / len(lat_sorted)
        }

# usage
# lg = LoadGenerator("http://model/infer", qps=50, duration_s=30, concurrency=100)
# asyncio.run(lg.run()); print(lg.summary())

Key points:

• QPS pacing by spacing requests (interval sleep) gives steady load.
• Semaphore bounds concurrency to limit sockets/CPU.
• Use asyncio.create_task to let tasks run concurrently.
• For large runs, replace in-memory list with t-digest or HDR histogram to bound memory and compute percentiles streaming.
• Edge cases: very high QPS where interval < event loop resolution (use token-bucket), long tail causing backlog, correct shutdown waiting for inflight tasks.

Requirements & constraints:

• End-to-end: client → API serving recommendations → online feature store (strong consistency) → model scoring → cache layer → metric collection
• Realistic user sessions, feature freshness SLAs (e.g., <100ms staleness), delayed label arrival for offline eval, warm/cold cache, and controlled failures.

High-level test architecture:

• Traffic generator (k6 or Locust) simulating sessions + think-times → API gateway → rate limiter → cache (Redis/Memcached) → recommendation service (reads online feature store via synchronous read-after-write) → model/renderer → response. Observability via Prometheus, Jaeger, and a metrics bus that records feature timestamps and labels arrival for offline metrics.

Traffic profiles & session simulation:

• Personas: Browsers (short sessions, high QPS), Power users (long sessions, sequential interactions), New users (cold-starts).
• Session generator: maintain per-session state, inter-request think-times sampled from realistic distributions (e.g., log-normal), action mixes (view/click/feedback) to trigger feature updates.
• Percent split example: 70% browsers, 20% power users, 10% cold-start/new users.
• Inject user churn and bursts (e.g., 5% of sessions generate rapid-fire events).

Ramp patterns:

• Gradual ramp: 5→50→250→1k RPS over 30–60 min with 10–15 min holds. Stress spike tests: sudden 2x–5x surge for 10 minutes to test autoscaling and rate limits. Soak test: sustained production-level load (e.g., 80% of expected peak) for 6–24 hours to expose memory leaks and stateful degradation.

Feature freshness & staleness validation:

• Tag every feature read with feature_timestamp and server_receive_timestamp. Compute staleness = server_receive_timestamp − feature_timestamp. Assert SLA percentiles (p50/p95/p99) < target (e.g., 50ms/100ms/200ms).
• Simulate writes from event stream with controlled propagation delays to emulate real-time feature updates; verify that reads after write are strongly consistent (read-after-write checks) under normal and degraded network conditions.

Warm vs cold cache patterns:

• Cold-cache tests: clear caches then replay realistic warm-up traffic to measure cache miss rate, latency spike, and backend load during warm-up.
• Warm-cache tests: pre-populate cache with hot keys (top 5% by access) and run steady-state traffic to measure sustained P99 latency and hit ratio.
• Mixed: partial warm/cold where new users cause cold misses while power users hit warm entries.

Delayed label arrival and offline metrics:

• Simulate label pipeline: events emitted during sessions are ingested to an offline store with configurable delays (e.g., 1h, 6h, 24h). Use synthetic ground-truth generator to produce labels later. Correlate predictions with label arrival to compute offline metrics (AUC, NDCG) with time-windowed backfills. Track label-latency distribution and its effect on evaluation windows.

Failure injection points:

• Network latency/jitter between service and feature store (introduce 10–500ms delays)
• Transient feature store errors (HTTP 5xx, timeouts) at configurable rates (e.g., 0.1% → 5%)
• Cache failures (evict, flush, or induce high miss rate)
• Throttling / rate-limit induced 429 responses from upstream
• Model-serving degradation (slow responses, return baseline recommendations)
• Chaos: kill pods, simulate disk pressure, saturate CPU on model nodes
  For each failure, run A/B style test: control vs. injected group and observe degradation tolerance, graceful fallbacks, and SLA violations.

Metrics to capture:

• Latency percentiles (p50/p90/p95/p99) end-to-end and per-component (API gateway, feature-store read, model score, cache lookup)
• Error rates: 4xx, 5xx, timeouts, 429s
• Throughput: requests/sec, model QPS, feature-store QPS
• Feature-staleness distribution (p50/p95/p99) and violation rate (percentage above SLA)
• Cache metrics: hit ratio, miss latency, eviction rate
• Backend resource metrics: CPU, mem, threads, GC, connection pools
• Availability & success-rate per persona
• Offline metrics: label arrival latency distribution, delay-adjusted model metrics (AUC, precision@k) as labels arrive
• Business KPIs: click-through-rate (simulated), conversion proxy

Validation & acceptance criteria:

• End-to-end p95 latency < target (e.g., 200ms) and p99 < 500ms under production-like load
• Error rate < 0.1% (or agreed SLO)
• Feature-staleness SLA met for 99% of reads
• Degradation mode: when feature-store latency increases by X, system gracefully falls back to cached features or safe defaults with bounded QoS loss
• Offline metrics converge as labels arrive within expected windows

Tools & automation:

• k6/Locust for session-based load with JS/Python hooks to emit events and tags
• Prometheus + Grafana for metrics and dashboards
• Jaeger for distributed traces, connecting feature-store read traces to latency
• Mock/event generator to simulate delayed label ingestion
• Chaos toolkit / Litmus / Gremlin for failure injection
• CI job to run canary load tests on PRs and scheduled long-duration soak tests in staging

Example k6 test plan highlights:

• Use k6 VUs to represent concurrent sessions; each VU maintains session cookie and event emission; add custom metrics for feature_timestamp and label_id to correlate later.
• Phases: ramping stages, spike stage, soak stage, chaos stage where an external script toggles failure injection.

Why this design:

• Simulates realistic user behavior (sessions & think-times) to exercise feature updates, cache dynamics, and strong-consistency reads.
• Measures feature freshness explicitly and ties offline evaluation to delayed labels, ensuring both online performance and measurement validity.
• Failure injection tests resilience and fallback paths, revealing cascading failure modes that simple load tests miss.