Production Incident Diagnosis and Distributed Systems Troubleshooting Questions

Direct answer. With two independent systems down at once, the immediate priority is stopping further damage from EACH failure separately, since they likely require different mitigations, before attempting any cross-system reconciliation, which only makes sense once both systems are stable again.

Structured elaboration.

1. Immediate mitigations, one per system. For the message broker: pause or buffer producers if possible (rather than letting them fail loudly or silently drop messages) while the broker is unavailable, and identify a fallback path if one exists (a secondary broker, or a degraded synchronous mode for the most critical messages only). For the cache: fail over to a source-of-truth read path (the database directly) for critical reads, accepting higher latency and load on the database temporarily rather than serving nothing or serving from a broken cache.
2. Steps to restore each service, prioritized by which is more critical to core functionality and which is faster to restore; restoring them doesn't need to happen simultaneously, and restoring the more critical one first, even if the other stays degraded a bit longer, is a legitimate prioritization call.
3. Reconcile state between systems, once both are back. The message broker's outage likely means some messages were never delivered (or were queued and are about to be delivered late, out of their original order relative to other system state); the cache's outage means the database was serving all reads directly during the outage, so the cache, once restored, needs to be treated as fully stale and either invalidated entirely or warmed fresh rather than trusted to reflect current state.
4. Deduplicate side effects. Anything that happened during the hour needs auditing for duplicate processing: did the broker's recovery cause any messages to be redelivered that had already been processed via a fallback path during the outage, and did any process see stale cached data and take an action based on it that's now inconsistent with the database's actual current state.
5. Validation checks post-recovery. Confirm the broker's queue depth and consumer lag return to normal, confirm the cache's hit rate and served-data freshness look correct (not still serving anything stale from before the outage), and specifically audit for duplicate or inconsistent side effects from the dual-failure window before declaring the incident fully resolved, not just 'services are up'.

Worked example. Suppose during the hour-long outage, a fallback path processed a subset of critical messages synchronously (bypassing the broker entirely) to avoid losing them, and separately, reads fell back to the database directly once the cache was known to be down. Once the broker recovers, it may still have QUEUED versions of those same messages that were already handled via the synchronous fallback, which would cause duplicate processing unless each message carries an idempotency key that the consumer checks before acting; if that idempotency key exists and is checked, redelivery is safe and requires no special reconciliation. If it does NOT exist, an explicit reconciliation pass, cross-referencing what the fallback path already processed against what the broker is now redelivering, is required before letting the broker's backlog drain automatically, to avoid, for example, double-charging a customer or double-sending a notification.

Trade-offs and pitfalls. Restoring both systems and immediately resuming full, unrestricted normal operation without a reconciliation pass risks exactly the duplicate-processing and stale-data problems this scenario sets up; the temptation to declare victory the moment both systems report healthy again is real, especially after an hour of firefighting, but the validation and reconciliation step is what actually confirms the incident is over rather than just the immediate symptoms disappearing. It's also worth treating this dual-failure as a prompt to check whether the two systems have any DEPENDENCY on each other's health that wasn't obvious before (did the cache failure make the broker situation worse, or vice versa) since understanding that interaction matters for preventing a similar dual-failure in the future.

Direct answer. When data goes missing silently, with no errors and no alerts firing, the first job is not root-causing yet: it's establishing exactly how much data is affected and for how long, because that scope determines both your mitigation urgency and what a correct fix even needs to repair.

Structured elaboration.

1. Establish scope before cause. Compare an authoritative count or checksum at the start of the pipeline against the same count or checksum at the end, for a range of recent time windows, to find exactly when the loss started and how much has been lost. Without this, you can't tell stakeholders how bad it is, and you risk fixing the code but missing that a specific batch or partition also needs to be reprocessed.
2. Contain first, then investigate. If the pipeline is still running and still losing data, the immediate priority is stopping further loss: this might mean pausing the pipeline stage that's dropping events, or routing new events to a location where they're safe (a raw, unprocessed store) even if you haven't fixed the processing yet.
3. Walk the pipeline stage by stage. With no obvious error, the loss is likely happening in a place that fails silently: a filter or transform step that drops records that don't match an expected shape without logging them, a deduplication step that's over-aggressive and treats distinct events as duplicates, an at-most-once delivery mechanism (the message is sent once and never retried, so if it's lost in transit it's simply gone, unlike at-least-once delivery which keeps retrying until acknowledged) that occasionally drops a message under backpressure (a signal from an overwhelmed downstream stage telling upstream to slow down or drop work rather than queue it indefinitely), or a downstream write that silently no-ops on a conflict instead of erroring. Check each stage's input count against its output count to localize which stage is where records disappear.
4. Reconstruct the two-day case specifically. For an incident that ran silently for an extended period, check whether any monitoring existed at all for input-versus-output counts at each stage; if not, that absence of monitoring is itself part of the root cause, since a real bug that WOULD have been caught in minutes with the right check instead ran undetected for days.
5. Recover and prevent recurrence. Once the losing stage is found, recovery usually means reprocessing the affected window from a raw or replayable source if one exists; if there is no way to recover the missing data, that gap needs to be communicated honestly to whoever consumes this data downstream. Prevention has three complementary parts: instrumentation (emitting an explicit count or checksum metric at every stage boundary, not just logging on error, so a silent drop shows up on a dashboard instead of requiring someone to notice); data contracts (an explicit, enforced agreement on what shape and volume of data each stage should produce, so a stage that silently changes behavior violates a checkable contract rather than drifting unnoticed); and scheduled reconciliation jobs that periodically re-verify end-to-end counts independently of the pipeline's own reporting, catching a class of bug where the pipeline's own instrumentation is itself the thing that's wrong.

Worked example. Say a checksum comparison finds the pipeline's ingestion stage received 100,000 events per hour throughout the affected window, but a stage-by-stage count shows only 94,000 events per hour reaching the final sink, a stable roughly 6% loss rather than a spike. Checking each intermediate stage's input-versus-output count in turn shows the drop happens entirely at a deduplication step, whose input and output counts differ by exactly the missing 6%. Looking at that step's logic reveals it dedupes on a composite key that, for a specific event type, isn't actually unique across different real events, so it's discarding legitimate events it mistakes for duplicates. Because the loss was steady rather than a sudden failure, it evaded any anomaly-style alerting that watches for sudden drops, which is exactly why an ongoing count-reconciliation check (rather than only alerting on sudden changes) is the prevention that would have caught it on day one instead of day two.

Trade-offs and pitfalls. The biggest pitfall is jumping straight to a fix before establishing scope: without the before/after counts, you don't know if you're looking at a total outage of one stage or a steady small leak, and those need very different urgency and different recovery plans. The other common mistake is fixing the code bug and declaring victory without checking whether the ALREADY-LOST data during the incident window can be recovered or backfilled from a raw source; a correct fix going forward doesn't repair the historical gap on its own.

Direct answer. A threefold, GLOBAL latency spike across multiple services points away from a single code bug (which would rarely hit every region and every affected service simultaneously) and toward something shared: a common piece of infrastructure, a global configuration or routing change, or a dependency every affected service happens to share.

Structured elaboration.

1. Collect first, before forming a hypothesis. Pull metrics (which services and regions are affected, and by how much, to see if the impact is genuinely uniform or has structure), logs (any error patterns common across the affected services), traces (to see if a common downstream call shows up across services), and system state (recent deploys, config changes, or infrastructure events globally, not just for one service).
2. Look for global infrastructure first, since 'global' and 'multiple services' both point that direction. DNS, a shared load balancer or CDN layer, a service mesh control plane, a shared authentication or authorization service, or a cloud provider's own regional or global infrastructure issue are the most common causes of a genuinely global, multi-service latency event.
3. Isolate network from infrastructure from application. If traces show elevated time specifically in inter-service network hops (not inside any service's own processing), that points at network. If a specific shared service (auth, a service-mesh sidecar, a shared cache) shows the same latency increase across every trace that touches it, that points at that shared infrastructure component specifically. If, after checking both, no shared component or network layer explains it, consider whether multiple SEPARATE application-layer issues coincidentally started at the same time, which does happen (for example a scheduled batch job or a marketing campaign driving a simultaneous traffic surge across many services).
4. Mitigate proportionally to confidence. If you're confident in a specific shared cause, a targeted mitigation (failing over that component, rolling back a global config change) is fastest. If you're still uncertain and the impact is severe, broader containment (like shedding non-critical traffic globally) buys time without betting on an unconfirmed hypothesis.

Worked example. Suppose traces across multiple unrelated services all show a new, roughly 150 to 200ms span that wasn't there before, corresponding to a call to a shared service-mesh sidecar for authorization checks, and a check of the mesh's own control-plane logs shows a configuration push went out globally about the same time the spike started. That converges cleanly: the config push likely changed something about how the sidecar handles authorization checks (a new policy evaluation that's more expensive, for example), and every service using the mesh inherited the cost simultaneously, which explains both the multi-service AND the global nature of the spike in one mechanism. The fix is rolling back that specific config push and validating that the added span disappears from traces across the previously affected services.

Trade-offs and pitfalls. The instinct under a severe, global incident is to investigate each affected service individually and in parallel, which can work but risks duplicated effort and conflicting theories across responders; explicitly looking for the SHARED cause first, and assigning one person to own that thread, tends to converge faster. It's also worth being disciplined about NOT assuming coincidence (multiple unrelated services breaking at once by chance) until you've genuinely ruled out a shared cause, since shared-infrastructure causes are far more common than true coincidence at this scale.

Direct answer. The moment the cache node fails, every client that would have gotten a cache hit instead gets a miss at roughly the same time, and if nothing throttles that, the database receives a spike close to the FULL read traffic the cache was absorbing, all at once, which is usually far more than the database is provisioned to handle directly.

Structured elaboration.

1. Immediate mitigation: stop the flood, not the symptom. Rate-limit or shed load at the edge closest to the database (an API gateway, a proxy, or the database's own connection limiter) so the database doesn't get overwhelmed while you work the real fix. Request coalescing (also called singleflight) is the more targeted version of this: if 500 clients all miss on the same key at once, coalescing ensures only ONE of those 500 actually queries the database, and the other 499 wait for and reuse that single result.
2. Restore cache capacity. Bring the failed node back or fail over to a healthy replica; if the cache cluster uses consistent hashing, losing one node ideally only invalidates the keys that node owned, not the whole cache, so confirm that's actually how your cache is configured (a poorly configured cache can lose everything on one node failure instead of a slice).
3. Long-term: prevent the all-at-once-miss pattern itself. Singleflight/coalescing as a standing pattern (not just an incident response) caps how much duplicate load a single hot key's cache miss can generate. Staggered TTLs (adding jitter to expiration times) prevent a large batch of keys from expiring at exactly the same moment, which is the same failure pattern even without a literal node failure. Cache warming (pre-populating a new or recovering node before it takes traffic) avoids the cold-cache-meets-full-traffic combination entirely. Bulkheads between the cache-miss path and other database traffic keep a cache-related spike from starving unrelated queries.
4. Detection and prevention plan. Alert specifically on cache hit-rate dropping and on database connection/CPU saturation together, since that combination is the signature of this failure mode; a hit-rate alert alone might not fire fast enough, and a database-load alert alone doesn't tell you WHY.

Worked example. Say the cache was serving 50,000 reads per second at a 98% hit rate, meaning only about 1,000 reads per second normally reach the database. The cache node fails, hit rate drops toward 0% for the affected keys, and now roughly 49,000 additional reads per second hit the database at once, a roughly 50x jump from its normal 1,000 rps baseline; a database provisioned for 1,000 to maybe 5,000 rps of direct traffic is overwhelmed almost instantly. With request coalescing in place, if those 49,000 requests are actually requests for a much smaller number of distinct hot keys (say 200 distinct keys being requested repeatedly), coalescing would collapse that down to roughly 200 concurrent database queries instead of 49,000, a reduction of about 245x, which the database can plausibly absorb while the cache recovers.

Trade-offs and pitfalls. Rate-limiting at the database protects the database but means some user requests get degraded or fail during the recovery window; that's usually the right trade because an unprotected database going down would make EVERYTHING fail, not just the cache-miss path. Staggered TTLs and singleflight need to be built in before the incident, not during it, which is why 'detection and prevention' matters as much as the immediate response: a system with no coalescing and no jitter will hit this exact failure mode again the next time a cache node fails or a batch of keys happens to expire together.

Direct answer. Sporadic high kernel (sys) time, as opposed to user-space CPU time, means the process is spending real CPU cycles inside the operating system itself, most commonly from syscalls, interrupt handling, or contention the application code doesn't directly control, and eBPF-based tools let you see exactly which kernel functions are consuming that time without modifying or restarting the running process.

Structured elaboration.

1. Confirm it really is kernel time, and get a first-pass breakdown. A tool like bpftrace's built-in profiling (or perf) can sample stack traces system-wide during the sporadic spike and show you a flame-graph-style breakdown of which kernel functions are hottest; this is the fastest way to go from 'sys time is high' to 'here are the specific functions responsible' without guessing.
2. If the profile points at network interrupts, a high volume of small packets, an interrupt storm, or the network card's interrupts landing disproportionately on one CPU core, rather than being spread across cores via receive-side scaling (a network-card feature that distributes incoming network interrupts across multiple CPU cores instead of pinning them all to one), are common causes; bpftrace can trace softirq (a deferred, lower-priority form of interrupt handling the Linux kernel uses to finish processing a network packet after the initial hardware interrupt) and hardware-interrupt events specifically to confirm.
3. If the profile points at futex (a common Linux syscall behind most user-space lock implementations), that usually means the APPLICATION is contending on a lock heavily enough that the kernel-level futex wait/wake path itself becomes a meaningful cost, which links this investigation back to the same kind of lock-contention pattern as an earlier CPU-spike question here, just visible at a lower level.
4. If the profile points at epoll_wait or similar, that's often actually benign (a process efficiently waiting for I/O readiness looks like kernel time but isn't a problem on its own), so distinguishing 'a lot of time in epoll_wait because the process is idle and waiting, which is fine' from 'a lot of time in epoll_wait because of excessive wakeups from a misbehaving event loop' matters, and tracing the RATE of wakeups (not just time spent) helps tell them apart.
5. Example commands. A one-liner like bpftrace -e 'profile:hz:99 { @[kstack] = count(); }' samples kernel stacks system-wide at 99Hz and counts occurrences, giving you a ranked list of the hottest kernel call paths over your sampling window; a targeted one like bpftrace -e 'kprobe:futex_wait { @[comm] = count(); }' counts futex-wait entries specifically, broken down by process name, which would confirm or rule out lock contention as the mechanism directly. bcc's funclatency or profile tools provide similar capability with less hand-written tracing code, if available on the host.

Worked example. Suppose the bpftrace kernel-stack profile during a spike shows roughly 70% of sampled kernel time inside futex_wait and related paths, concentrated on this service's own process. That points squarely at application-level lock contention manifesting as kernel time (since the underlying futex mechanism involves a kernel-level wait/wake handshake once contention is high enough), not a network or I/O issue. Cross-checking with a user-space thread dump taken at the same moment, showing many threads blocked waiting on the same lock object, corroborates it independently. The fix, at this point, is the same as any lock-contention problem: reduce the critical section, shard the lock, or reduce concurrency contending on it, not anything at the kernel or network level, even though the SYMPTOM (high sys time) initially pointed at the kernel.

Trade-offs and pitfalls. It's a common and understandable mistake to see 'high kernel time' and assume the problem is infrastructure-level (network, disk, the OS itself) when, as this example shows, it's frequently a downstream SIGNATURE of an application-level problem like lock contention; the kernel is just where that contention becomes visible as CPU time. eBPF tracing is low-overhead and safe to run on production without restarting anything, which is exactly why it's the right first tool here rather than something riskier like attaching a debugger, but sampling frequency (like the 99Hz above) is a real trade-off between profiling resolution and the (small but nonzero) overhead of tracing itself.