Systematic Debugging and Root Cause Analysis Questions

Testing whether a specific schema/index change caused a latency regression, rather than a coincidence, needs a controlled comparison, not just "it happened right after."

Experiment and monitoring design

Run the change as a canary/control split where possible: apply the index change to a subset of traffic or a subset of read replicas while a control set keeps the previous index configuration, holding query pattern and load as similar as possible between the two groups. Track the key metric (query latency, specifically the affected endpoint) plus guardrails (overall DB load, lock contention) over a duration long enough to average out normal noise (hours, not minutes, if traffic has daily patterns).

Quick statistical checks

Compare the latency distributions before/after (or control/variant) using percentiles (p50/p95/p99), not just the mean, since an index change often affects the tail disproportionately (some query shapes newly need a full scan while others speed up); a simple check like "did p95 latency shift outside its pre-existing normal range by more than typical day-to-day variance" is usually sufficient without needing a full formal significance test under time pressure. Concretely: if the endpoint's p95 had been a stable 40ms give or take 5ms for the prior week, and after the index change it now sits at 120ms for three straight days, that is well outside the normal band and is enough evidence to act on without waiting for a formal significance test.

Confirming causation, not just correlation

If a true control/variant split isn't feasible, the next-best check is examining the query's execution plan directly before and after the index change: does the plan show it stopped or started using the removed/added index for this specific query, which is a mechanistic, not just correlational, explanation for the latency shift.

Trade-offs and pitfalls

An index change can improve some queries while degrading others (a new index adds write overhead, or an old index's removal forces a different query plan for some code paths); the investigation needs to check the specific affected endpoint's execution plan, not just an aggregate database-wide metric, or a targeted regression can be diluted into invisibility in an aggregate view.

An eventually-consistent cache with a code-change-introduced invalidation race is a two-part problem: stop the ongoing corruption, then reconcile what's already wrong.

Detecting the race in production

Add audit logging around every cache invalidation and every write: log the key, the writer identity, and a monotonic version/timestamp for both the cache invalidation event and the primary-DB write. A race shows up as a write and an invalidation for the same key whose logged order doesn't match the order the application intended (e.g., an invalidation logged before a write it should have followed).

Immediate mitigation

Stop-the-bleeding options, roughly in order of cost: temporarily disable the specific cache-invalidation path that changed (fail toward re-reading from the primary DB, accepting a latency cost, not a correctness cost); or widen the invalidation to be more conservative (invalidate broader keys/TTL, trading cache hit rate for correctness) while the real fix ships.

Durable fix and reconciliation

The durable fix is almost always to make the write path itself the single source of truth for invalidation ordering: attach a version number to cached values and reject/ignore a stale invalidation or write that arrives with an older version than what's already cached (compare-and-set semantics), rather than trusting event arrival order. Concretely: cache key order:88 is currently cached at version 5 after a recent write; a delayed invalidation message tagged version 3 (from an earlier, slower write path) arrives after that newer write already landed. Because 3 is older than the cached 5, the invalidation is compared against the version and dropped instead of blindly wiping out the newer, correct value. For already-corrupted data: run a reconciliation job that recomputes canonical values from the primary DB and dedupes/overwrites any diverged cache/DB entries, working in small verified batches rather than one big blind rewrite, since some corrupted writes may have already been read and acted on downstream.

Trade-offs and pitfalls

Widening invalidation scope as a stop-gap trades cache efficiency for correctness, which is usually the right trade during an active incident; leaving that "temporary" wide invalidation in place permanently is the common follow-up mistake once the pressure is off.

A strong candidate starts from a fixed loop, not a guess: gather signal, form a hypothesis, run the cheapest test that could disprove it, and only then act.

The loop

1. Collect first, don't touch anything. Pull the alert/report, recent deploys and config changes, and the three signal types: metrics (what changed and when), logs (what the service says happened), traces (where time went across components).
2. Triage the layer before the cause, using a fixed set of checks per layer (see below): does it correlate with one host, one region, one dependency, or all of them? A failure isolated to one instance points at that instance; a failure across all instances after a deploy points at the deploy; a failure correlated with a dependency's own error rate points downstream.
3. Form one falsifiable hypothesis at a time ("the new deploy is the cause") and pick a test that would disprove it cheaply (check whether the errors started at the exact deploy timestamp, or whether rolling back one canary host clears them). Avoid changing five things at once.
4. Prioritize under time pressure: mitigate first (rollback, scale out, fail over) if user impact is ongoing, investigate the true cause in parallel or after.

Commands and tools per layer, and how to triage between them

Run these roughly in parallel across a couple of affected hosts, not sequentially one host at a time:

• Service/application layer: kubectl get pods -o wide and kubectl describe pod <pod> (or systemctl status <service> on VMs) to check restart counts and recent events; kubectl logs -f <pod> --previous or journalctl -u <service> -f for the exact error at the moment of the alert; top/htop or ps aux --sort=-%cpu,-%mem for CPU/memory pressure on the process itself; for managed runtimes, jstack <pid> or jcmd <pid> Thread.print to check for stuck threads. Signal that this is the layer: errors and restarts correlate with specific pods/hosts or with the deploy timestamp, not with a single dependency or the network path.
• Network layer: curl -v against the exact endpoint from an affected host to separate DNS/TLS/connect time from application response time; dig/nslookup for DNS resolution issues; traceroute/mtr for path/latency between hops; ss -s or netstat -antp for socket/connection-state saturation (too many TIME_WAIT, exhausted ephemeral ports); tcpdump on a specific host if a particular hop is suspected. Signal: curl -v's connect/TLS phase is slow while the app's own processing time (visible in traces) is normal, or the problem tracks a specific region/CDN edge rather than a specific service version.
• Database layer: active/slow query lists (SHOW PROCESSLIST on MySQL, pg_stat_activity on Postgres), lock waits (SHOW ENGINE INNODB STATUS, pg_locks), and the app's own connection-pool metrics (checked-out connections near the pool limit). Signal: request latency traces show most of the time inside the DB span, and DB-side query/lock metrics show a corresponding spike at the same timestamp.
• Infrastructure layer: kubectl describe node / kubectl top node for node-level CPU/memory/disk pressure, dmesg or journalctl -k for OOM-killer or kernel-level events, and the cloud provider's status page or recent autoscaling/capacity events. Signal: the failure correlates with a specific node, availability zone, or a capacity/autoscaling event rather than with a code deploy or a single dependency.

The layer whose checks show a signal exactly aligned with the alert's onset time is the one to dig into first; the others should still be glanced at briefly to rule out a compounding factor, but don't get equal depth until the primary layer is ruled out.

Worked example

A service starts returning 500s at 14:02. A deploy went out at 14:00. Metrics show error rate flat on hosts still running the old build and elevated only on hosts running the new one. That single comparison (same traffic, different build, different outcome) is strong evidence for the deploy as root cause, tested in under a minute using data you already have, before touching any code.

Trade-offs and pitfalls

The most common mistake is skipping straight to "it's probably the database" because that's where the last incident was, without checking whether this failure actually correlates with DB latency. A hypothesis not tested against data is a guess wearing an RCA costume. The other common failure is fixing the first plausible thing that appears in the logs, when it's a symptom of an earlier upstream cause; correlating the alert time against the deploy/change timeline first avoids that trap.

A correlation ID is a single identifier generated at the edge of a request and propagated through every hop, so logs, metrics, and traces from otherwise-independent services can be joined into one end-to-end picture.

Design

1. Generate once, at the edge (the first service to see the request), and reject or ignore any incoming ID from an untrusted external source unless you specifically want to correlate with an external partner's identifier.
2. Propagate via headers (e.g. X-Request-Id or the W3C traceparent format) on every downstream call, and via message baggage for async work (queue messages, background jobs) so the ID survives a hop that isn't a synchronous HTTP call.
3. Attach it to every log line and trace span emitted while handling that request, not just at the top-level handler.
4. For third-party/legacy components you cannot instrument, enrich logs at your own ingress/egress boundary around that component (the request going in, the response coming out) so you can at least bound how much time was spent inside the black box, even without visibility inside it.
5. Query time: given an ID, a single query across the log/trace store reconstructs the full path, ordered by timestamp, across every service that touched the request.

Trade-offs and pitfalls

High-cardinality IDs are cheap to log but expensive to index if you try to make every field high-cardinality; keep the correlation ID itself indexed and unique, but avoid also indexing every other high-cardinality field (like raw user input) at the same tier, or storage cost balloons. Sampling (only tracing 1% of requests in detail) is fine for aggregate analysis but means a specific customer's specific failing request may not have full trace data; the mitigation is to force full capture whenever an error is detected on that request, regardless of the sampling decision made at the start.

This LRU cache's single-threaded eviction logic is correct in isolation, but "evicts the wrong key... under high throughput" points at concurrent access without any locking, which lets the check-then-evict sequence interleave across threads.

class LRU:
    def __init__(self, k):
        self.k = k
        self.order = []
        self.map = {}
    def get(self, x):
        if x in self.map:
            self.order.remove(x); self.order.append(x); return self.map[x]
        return -1
    def put(self, x, v):
        if x in self.map:
            self.order.remove(x)
        elif len(self.order) >= self.k:
            old = self.order.pop(0); del self.map[old]
        self.order.append(x); self.map[x] = v

Verified demonstration of the race

Setting capacity to 1 and forcing a deliberate interleaving (thread A decides to evict based on order=['a'], then sleeps before actually popping; thread B, seeing the same full cache, evicts 'a' immediately and inserts 'd') shows thread A then wakes and pops from the now-different list, evicting 'd' instead of what it originally observed. Executed result: 'd' is silently evicted moments after being inserted by the other thread, even though the final cache size stays at capacity, a genuine wrong-eviction bug caused purely by the unsynchronized check-then-act sequence, not by any single-threaded logic error.

Step by step, with actual values: at capacity 1, the cache holds order=['a'] (map={'a': 1}). Thread A calls put('c', 3): it sees len(self.order) >= self.k is true, so it decides to evict, reads old = self.order[0] conceptually as 'a', but then the OS pauses thread A right before self.order.pop(0) actually runs. Thread B now calls put('d', 4) and runs to completion uninterrupted: it also evicts 'a' (order becomes ['d'], map={'d': 4}). Thread A resumes, calls self.order.pop(0) against the now-current order=['d'], which pops and evicts 'd' instead of the 'a' it originally decided to evict, then proceeds to insert its own 'c'. The result: 'd', which thread B just inserted, is evicted almost immediately, and the final state is order=['c'] when a correct execution should have let both 'c' and 'd' compete fairly for the single slot, not silently discard 'd'.

A robust fix

Wrap every get/put in a single lock so the whole check-then-evict-then-insert sequence is atomic:

import threading
class LRU:
    def __init__(self, k):
        self.k = k; self.order = []; self.map = {}; self.lock = threading.Lock()
    def get(self, x):
        with self.lock:
            if x in self.map:
                self.order.remove(x); self.order.append(x); return self.map[x]
            return -1
    def put(self, x, v):
        with self.lock:
            if x in self.map:
                self.order.remove(x)
            elif len(self.order) >= self.k:
                old = self.order.pop(0); del self.map[old]
            self.order.append(x); self.map[x] = v

Complexity

list.remove/pop(0) are O(n); for a hot, high-throughput cache, an OrderedDict (O(1) move-to-end and popitem(last=False)) behind the same lock is the standard production-grade replacement, same locking discipline, better constant factors.

Edge cases

Capacity of 0 or 1, repeated put of the same key (must update recency without double-inserting), and concurrent get/put on the same key from different threads (must not interleave a partial remove/append), all specifically exercised by a concurrent stress test, not just a serial unit test.