Latency Analysis & Optimization Questions

Understanding and reducing response time across the request path, including tail latency, latency budgets, and critical-path analysis. Covers where latency accumulates (compute, I/O, serialization, network hops, queuing), percentile-based reasoning (p50/p95/p99), and targeted techniques to shave the dominant contributors. Focuses on end-to-end latency as an engineered property rather than an incidental one.

MediumTechnical

25 practiced

An endpoint's p95 latency is 800ms and the product goal is 400ms. Propose a measurable plan to reduce latency by 50%: describe how you'd profile the endpoint, identify likely hotspots, propose at least three optimizations (with trade-offs), and explain how you'd present the options and estimated impact to product and SREs.

Sample Answer

Approach — profile first, then fix highest-impact hotspots, validate with metrics and controlled rollouts.

Profiling plan- Reproduce load that yields p95=800ms in a staging perf test.- Enable distributed tracing (e.g., OpenTelemetry) to break down latency by service/span.- Collect flamegraphs / CPU profiles, heap/GC stats, and slow SQL traces (EXPLAIN + query timings).- Measure network RTTs, payload sizes, and downstream service latencies.- Produce a breakdown: e.g., 40% DB, 25% serialization, 20% remote call, 15% app CPU.

Likely hotspots- Slow DB queries or N+1 access- Blocking sync calls / remote service waits- Heavy serialization/deserialization or large payloads- Suboptimal concurrency (thread pools, connection pools), GC pauses

Three optimizations (with estimates & trade-offs)1) Cache expensive reads (edge or app cache) - What: add Redis/memcached or HTTP cache for idempotent GETs; TTL tuned. - Estimated impact: reduces DB portion — p95 down ~25–40%. - Trade-offs: cache invalidation complexity, stale data window, operational cost.

2) Optimize DB queries / add indexes / denormalize - What: EXPLAIN+rewrite slow queries, add composite indexes or precomputed columns. - Estimated impact: reduces DB latency — p95 down ~20–35%. - Trade-offs: increased write cost, storage, migration effort, possible locking during deploy.

3) Make flows asynchronous / batch work - What: move non-critical work to background jobs (queue), batch remote calls, use async I/O. - Estimated impact: cuts synchronous waits — p95 down ~15–30%. - Trade-offs: increased system complexity, eventual consistency, more components to monitor.

Presentation to Product & SREs- Deliver a short decision memo with: - Current breakdown chart (tracing) and target gap. - Options table: change, estimated p95 improvement, dev effort (story points), risk, rollback plan, SRE impact (operational cost, new alerts). - Recommended phased plan: immediate low-risk wins (cache small scope), medium-term DB fixes, longer-term async refactor. - Rollout: canary → monitoring (p95, error rate, cache hit, DB QPS) → full deploy. Define measurable success criteria (p95<=400ms, no increase in errors).- Coordinate SRE for capacity planning, runbooks, dashboards, and automated alarms before each deployment.

Validation & follow-up- After each change run A/B or canary perf tests, compare p95 and span breakdown, and iterate until goal achieved.

MediumTechnical

29 practiced

A profile API shows increased 99.9th percentile latency under load. Outline an approach to diagnose and mitigate high tail latency: instrumentation you would add, experiments to reproduce the tail, prioritization of fixes, and candidate mitigations (caching, circuit-breakers, prioritized queues, and load-shedding).

Sample Answer

Start by clarifying scope and SLOs: confirm the 99.9th percentile target, current SLO breach window, and which endpoints/regions/users show the degradation.

Instrumentation to add- Distributed tracing (e.g., OpenTelemetry): trace requests end-to-end, capture spans for network, DB, cache, auth, downstream calls.- Per-call timing histograms and percentiles per component (service, DB query, third-party) and per route.- Contextual logs with request IDs, payload size, user tier, and resource tags.- Resource metrics: CPU, GC/pause times, heap, thread pool queue lengths, socket queues, file descriptors.- Error and concurrency counters (timeouts, retries, circuit-breaker opens).- Sampling focused on tail requests (log full traces for 1 in N tail samples).

Experiments to reproduce tail- Load-test with realistic traffic shape (ramp, spikes) and use production-like data sizes.- Chaos experiments: inject latency/failure in downstream services, DB, and network to reveal cascading effects.- Profile under load: heap/CPU flame graphs, thread dumps during tail events.- A/B tests toggling caching, concurrency limits, and client-side retries.

Prioritization of fixesRank by impact × ease:1) Fix high-impact, low-effort mitigations (e.g., add caching for heavy-read endpoints, increase index on DB query).2) Next, address systemic issues (thread pool exhaustion, blocking IO).3) Larger changes (architectural refactor, sharding) last.

Candidate mitigations (when & why)- Caching: apply at edge or service level for idempotent profile reads; reduces backend load and tail variance. Use short TTLs and cache invalidation on writes.- Circuit-breakers: protect callers from cascading failures; open fast on rising error/latency rates and fallback to stale cache or graceful degradation.- Prioritized queues / request prioritization: ensure interactive requests (profile view) get priority over heavy background work; implement separate thread pools with bounded queues to avoid head-of-line blocking.- Load-shedding: reject or downgrade non-critical requests when system saturates (return 503 with Retry-After or lightweight fallback). Implement at ingress/load-balancer or service boundary.- Async / bulkhead patterns: move slow calls to async flows or use per-dependency bulkheads to isolate failures.

Validation- After each mitigation, re-run load tests and track 99.9th percentile, error rates, and business KPIs. Roll incremental fixes with telemetry to verify improvement before broader rollout.

Trade-offs- Caching increases staleness risk; circuit-breakers trade availability for consistency; shedding impacts user experience. Choose based on SLA and business priorities.

MediumTechnical

26 practiced

A service increased its p95 latency for a key endpoint. Outline a step-by-step investigation plan: what metrics, logs, and traces you would examine; experiments to run in staging or production to isolate the cause; and quick mitigations you would apply to reduce p95 while you work on a permanent fix.

EasyTechnical

30 practiced

Implement an LRU cache class in Python with get(key) and put(key, value) operations supporting a fixed capacity and O(1) semantics. The implementation can be single-threaded; explain how your design achieves O(1) and outline tests to validate correctness under heavy load.

Sample Answer

Approach: use a hash map for O(1) key lookup and a doubly-linked list to maintain recency order (head = most recent, tail = least). get moves node to head; put inserts or updates and evicts tail when capacity exceeded. Both operations are O(1).

python

class Node:
    def __init__(self, key, val):
        self.key = key
        self.val = val
        self.prev = None
        self.next = None

class LRUCache:
    def __init__(self, capacity: int):
        self.cap = capacity
        self.map = {}  # key -> Node
        # dummy head/tail to simplify edge cases
        self.head = Node(None, None)
        self.tail = Node(None, None)
        self.head.next = self.tail
        self.tail.prev = self.head

    def _add_to_head(self, node):
        node.next = self.head.next
        node.prev = self.head
        self.head.next.prev = node
        self.head.next = node

    def _remove(self, node):
        node.prev.next = node.next
        node.next.prev = node.prev

    def get(self, key: int) -> int:
        node = self.map.get(key)
        if not node:
            return -1
        # move to head (most recent)
        self._remove(node)
        self._add_to_head(node)
        return node.val

    def put(self, key: int, value: int) -> None:
        node = self.map.get(key)
        if node:
            node.val = value
            self._remove(node)
            self._add_to_head(node)
        else:
            if len(self.map) >= self.cap:
                # evict LRU (tail.prev)
                lru = self.tail.prev
                self._remove(lru)
                del self.map[lru.key]
            newn = Node(key, value)
            self.map[key] = newn
            self._add_to_head(newn)

Why O(1): hashmap gives O(1) lookup; linked-list node removal/insertion is O(1) with direct pointers. No traversal is required.

Tests to validate correctness and performance:- Unit tests: basic get/put, overwrite existing key, eviction order (sequence-driven).- Edge cases: capacity = 0, single capacity, repeated same key updates, negative/None keys if allowed.- Heavy-load functional test: run 1M mixed random get/put ops with known oracle (smaller model) to assert correctness periodically.- Performance/stress: measure ops/sec for large capacity (e.g., 100k) and ensure memory growth matches expected O(n).- Fuzz tests: random sequences and cross-check against Python collections.OrderedDict-based reference implementation.

This design is simple, memory O(capacity), and suitable for single-threaded use.

EasyBehavioral

34 practiced

Tell me about a time when you improved performance of a service. Describe the initial problem, how you measured a baseline, the specific changes you implemented, how you validated improvements with metrics, and the measurable business impact after deployment.

Unlock Full Question Bank

Get access to hundreds of Latency Analysis & Optimization interview questions and detailed answers.

Join thousands of developers preparing for their dream job.