Debugging and Systematic Troubleshooting Questions

Direct answer

Application-level debugging asks "why did this code produce the wrong result," and its evidence lives inside your process: variables, call stacks, exceptions, business logic. Infrastructure-level debugging asks "why couldn't the correct code even run correctly," and its evidence lives outside your process: network reachability, disk and memory pressure, DNS, container scheduling, resource limits. The two failure modes can look identical from the outside (a request fails, a page times out) while requiring completely different evidence to distinguish.

Structured elaboration

The practical test: if you could run the exact same code on a machine with unlimited, perfectly-behaving resources and a clean network, would the bug still happen? If yes, it's application-level (a logic error, a bad state transition, an unhandled edge case). If no, it's infrastructure-level (the code was never wrong, but its environment failed to give it what it needed).

Two concrete examples that look alike but diverge in root cause:

1. "Requests are timing out." Application-level cause: a newly introduced N+1 query pattern (fetching a list, then running one extra database query per item in that list instead of one combined query, so 100 items means 101 round trips instead of 1) makes one endpoint genuinely slow under load, so it exceeds a client timeout. Infrastructure-level cause: the container's memory limit is too tight, so the process is spending most of its time in garbage collection (or worse, getting OOM-killed (the operating system forcibly terminating the process because it used more memory than it was allowed to) and restarted), and the timeout is really a symptom of resource starvation that has nothing to do with the query logic. Both present as "P99 latency spiked" in a dashboard.
2. "Intermittent 500 errors on one specific endpoint." Application-level cause: a race condition in that endpoint's handler corrupts shared state under concurrent access. Infrastructure-level cause: that endpoint happens to be the one that calls out to a dependency whose DNS resolution is flaky, so the failures are really network-layer and would happen on ANY endpoint that made the same external call.

How to distinguish them in practice: check infrastructure-level signals FIRST because they're cheap and rule out a whole class of causes at once: container restarts/OOM kills, CPU throttling, disk pressure, DNS/network error rates, host-level resource graphs. If those are clean, the evidence points inward to application logic, and now stack traces, code paths, and business-logic invariants become the productive place to look. Going the other direction (deep-diving application logs before checking whether the host was even healthy) wastes time chasing red herrings in code that was never the problem.

Worked example

Given "checkout intermittently fails for large carts": check host-level memory/CPU graphs for the checkout service during the failure window (infra-level, five minutes). If those are flat and unremarkable, the next cheapest infra check is whether the failures correlate with a specific downstream dependency's error rate (still infra-adjacent, still cheap). Only once both come back clean do you profile the actual cart-processing code path, because now the evidence has ruled out "the environment failed the code" and pointed at "the code itself has a large-cart-specific bug," likely a data-size-dependent logic error, not a data-size-dependent resource limit.

Trade-offs and pitfalls

The trap is assuming the layer based on which tools you personally know best rather than what the symptom actually implies. A backend engineer with no infra background will often burn an hour reading application logs for a problem that a five-second look at a container-restart graph would have explained. The fix is cheap and disciplined: always take the five-minute infra-layer look first, even if you expect it to come back clean, because ruling it out is what makes the deeper application-level dig trustworthy.

Direct answer

Array.prototype.forEach does not await its callback: it fires each async callback and moves on immediately, so console.log('done') runs before any of the await processor(item) calls have resolved. The fix is to replace forEach with a construct that actually waits, either a for...of loop with await inside it (sequential) or Promise.all over items.map(...) (concurrent).

Structured elaboration

forEach was designed before async/await existed and its callback's return value, including a Promise, is simply discarded. Passing an async function as the callback doesn't change this: each invocation still returns a Promise that forEach never looks at, so forEach itself completes synchronously (having merely started every item, not finished any of them) the instant it has called the callback once per array element. The console.log('done') line after the forEach call then runs immediately, while the async work is still in flight in the background.

Worked example

async function sleep(ms) { return new Promise(r => setTimeout(r, ms)); }

// Fixed, distinct per-item delays (not Math.random()) so the interleaving below
// is deterministic and reproduces identically on every run, not just illustrative.
const delays = { 1: 20, 2: 10, 3: 30 };

// BUGGY
async function processItemBuggy(x) { await sleep(delays[x]); console.log('  processed', x, '(buggy)'); }
async function processBuggy(items) {
  items.forEach(async item => { await processItemBuggy(item); });
  console.log('done (buggy)');
}

// FIXED: sequential
async function processItemFixed(x) { await sleep(delays[x]); console.log('  processed', x, '(fixed)'); }
async function processFixed(items) {
  for (const item of items) { await processItemFixed(item); }
  console.log('done (fixed)');
}

(async () => {
  console.log('--- buggy run ---');
  await processBuggy([1, 2, 3]);
  await sleep(100); // let the buggy run's background work finish before the next section starts
  console.log('--- fixed run (sequential for-of) ---');
  await processFixed([1, 2, 3]);
})();

Executed output (buggy items finish asynchronously in the background; with the fixed per-item delays above, item 2 always resolves first, then item 1, then item 3, after done has already printed):

--- buggy run ---
done (buggy)
  processed 2 (buggy)
  processed 1 (buggy)
  processed 3 (buggy)
--- fixed run (sequential for-of) ---
  processed 1 (fixed)
  processed 2 (fixed)
  processed 3 (fixed)
done (fixed)

done (buggy) prints immediately, before any processed line: forEach fired all three async callbacks and moved on without waiting for any of them. The processed (buggy) lines do eventually print, once their fixed delays resolve, in delay order (2, then 1, then 3, not the input order), rather than before done. The for...of version correctly prints all three processed lines, strictly in input order, before done.

Trade-offs and pitfalls

There are two valid fixes with different semantics, and picking the wrong one is itself a common mistake: a for...of loop processes items strictly one at a time (useful when order matters or you must not overwhelm a downstream dependency with concurrent calls), while await Promise.all(items.map(item => processor(item))) processes all items concurrently and finishes as soon as the slowest one does (faster, but only safe if the operations are independent and the downstream system can handle concurrent load). Silently reaching for forEach out of habit, rather than deliberately choosing sequential versus concurrent semantics, is exactly how this bug class gets reintroduced even by developers who already know the rule.

Direct answer

Prioritize concurrent bugs and incidents using an explicit rubric weighing severity, user impact, reproducibility, rollback cost, and business KPIs together, not any single factor alone, since a high-severity-sounding bug with low actual user impact and an expensive rollback can rank BELOW a "smaller" bug that's actively costing revenue and has a trivial fix available.

Structured elaboration

The rubric, and how each factor is weighed:

• Severity: how bad is the failure mode itself (data loss/corruption ranks above a cosmetic issue, a security-relevant bug ranks above a performance blip), independent of how many users are currently affected.
• User impact: how many users/requests are affected right now, and is that number growing, stable, or shrinking; a severe bug affecting a handful of users may rank below a moderate bug affecting a large fraction of traffic.
• Reproducibility: a reliably reproducible bug can be diagnosed and fixed faster (lower time-to-resolution for the same engineering effort) than an intermittent one, which affects how quickly EACH candidate bug can actually be resolved if picked next, not just how bad it is.
• Rollback cost: if a bug traces to a specific recent change, how cheap and safe is reverting that change right now; a bug with a trivial, low-risk rollback available should often be handled immediately regardless of its rank on other factors, since the fix is nearly free.
• Business KPIs: which specific business metric is being affected (revenue, a contractual SLA, a compliance requirement) and how directly; a bug affecting a metric with hard, immediate business consequences (a broken payment flow) generally outranks one affecting a softer, longer-horizon metric even at similar technical severity.

Applying the rubric during a busy incident window: first, quickly triage EVERY open issue against the rubric (a few minutes per issue, not a deep investigation), to get a relative ranking rather than working issues in the order they arrived; second, look specifically for any issue with BOTH meaningful impact AND a cheap available mitigation (like a trivial rollback), since these should jump the queue regardless of their raw severity ranking, because the cost of addressing them is so low relative to the benefit; third, re-triage periodically as the window continues, since impact and severity can both change (a bug's blast radius growing, or a rollback becoming available partway through investigation of one issue), rather than treating the initial ranking as fixed for the whole incident window.

Worked example

Three concurrent issues during a busy window: (A) a high-severity-sounding data-consistency bug affecting a small, specific edge case (low current user impact, no clear quick fix, moderate rollback risk since the change is tangled with other recent work); (B) a moderate-severity bug causing a checkout-flow error for roughly 5% of transactions (clear, growing user impact, directly hitting a revenue KPI, and traced quickly to a specific recent config change with a trivial, low-risk rollback available); (C) a low-severity cosmetic UI bug reported by a few users (minimal impact, no urgency). Applying the rubric: (B) is prioritized FIRST despite being technically "less severe" than (A) in the abstract, specifically because it combines real, growing, revenue-affecting impact with an almost-free rollback fix, making it both the highest-leverage and fastest issue to resolve. (A) is prioritized second, staffed for a proper investigation given its rollback isn't cheap and its actual mechanism needs to be understood before a safe fix can be applied. (C) is deprioritized entirely for the duration of the busy window, revisited once the window calms down.

Trade-offs and pitfalls

A common mistake is prioritizing purely by SEVERITY LABEL (treating "critical" as automatically first regardless of current impact or fix cost), which can leave a rapidly-growing, revenue-affecting, cheaply-fixable issue waiting behind a technically-severe-but-currently-narrow, expensive-to-fix one; the rubric's explicit multi-factor weighing exists specifically to avoid that trap, and re-triaging periodically (rather than committing to the initial ranking for the whole window) accounts for the fact that these factors genuinely change as an incident window progresses.

Direct answer

High CPU with low user-visible latency means the CPU work isn't on the critical path the user is waiting on: it's either background work (batch jobs, garbage collection, async processing), wasted work (a busy-loop, redundant recomputation, an inefficient algorithm that happens to still finish fast enough), or a measurement artifact (CPU metrics counting time the process spends idle-but-scheduled, or double-counting across containers sharing a host). The investigation is about separating these three, not assuming the first plausible one.

Structured elaboration

1. Check whether the CPU usage correlates with request volume or is constant/background. If CPU stays high even during low-traffic periods, it's very unlikely to be request-handling work; that points toward background jobs, scheduled tasks, or a stuck loop.
2. Profile what's actually consuming CPU, using a sampling profiler (perf, py-spy, async-profiler depending on the runtime) rather than guessing from code review. Distinguish CPU time attributed to request handlers versus background threads, garbage collection, or monitoring/logging agents.
3. Check for wasted work specifically: a cache that isn't actually being hit (so every request redoes expensive work that should have been cached), a retry loop that's spinning faster than intended, or an algorithm with much worse complexity than necessary that still completes within the user's timeout because the dataset happens to be small today.
4. Rule out measurement artifacts. On containerized/shared hosts, CPU metrics can reflect cgroup accounting quirks (cgroups are the Linux kernel mechanism that caps and meters how much CPU/memory a container is allowed to use, and its usage counters can misreport in edge cases), CPU throttling counted as "usage," or a host-level metric that aggregates multiple co-located processes. Confirm the metric is scoped to the process you think it is.
5. Distinguish "wasted but harmless today" from "a ticking time bomb." Work that's wasted but currently fits comfortably within capacity may not need urgent action; the same wasted work will become a real incident the moment traffic grows or the host loses spare capacity, so it's worth flagging even if latency looks fine right now.

Worked example

A service with 70% average CPU but P99 latency (the response time slower than 99% of requests, a common way to track worst-case rather than average experience) well within its SLA (service-level agreement, the target it's contractually or operationally expected to meet): profiling shows 40% of that CPU is spent in a background reconciliation job that runs every 30 seconds regardless of load, unrelated to request handling entirely. That explains the discrepancy directly: the CPU number reflects background work, not the request path the user experiences. The remediation is about the background job's efficiency and scheduling (should it run every 30s, or can it be event-driven instead), not about the request-handling code the on-call engineer might otherwise be tempted to optimize first.

Trade-offs and pitfalls

The main trap is treating "CPU is high" and "users are impacted" as the same finding when they can be completely decoupled, as in this example. Optimizing the wrong thing (request-handler code, when the real cost is a background job) wastes effort and leaves the actual capacity risk unaddressed. The remediation priority should follow from WHERE the profiler says the time goes, not from where it would be most convenient to look.

Direct answer

First establish WHO owns the fault, before trying to fix anything: check whether the 502s originate from your own service (a bug in how you call the provider), the network path between you and them, your configuration (wrong endpoint, expired credentials, timeout misconfigured), or the provider itself. A 502 specifically means an upstream server returned an invalid response to a gateway, which already narrows the search: it usually means something between you and the provider's actual application server broke, not that your request was malformed (that would more often be a 4xx).

Structured elaboration

1. Check your own logs for the exact request/response pair. Capture the full request you sent (headers, body, timing) and the full 502 response, including any body the provider returned. A 502 with a body from the provider's own error page suggests an issue near their edge, not deep in their systems.
2. Check timing and correlation. Is the failure rate correlated with your request VOLUME (suggests rate limiting or a capacity issue on their side), with TIME OF DAY (suggests their scheduled maintenance or your own traffic pattern), or with a SPECIFIC request shape (suggests your own payload triggers an edge case in their processing)?
3. Rule out your own network and configuration. Confirm DNS resolves to the expected endpoint, TLS handshakes succeed, and you're not accidentally hitting a sandbox/staging URL in production. Check whether a recent config or credential change on your side coincides with when the 502s started.
4. Check the provider's status page and support channels. Many payment providers publish real-time incident status; if others report the same symptom at the same time, that's strong evidence the fault is upstream, not yours.
5. Determine if it's truly intermittent or has a pattern. A steady low background rate of 502s can be normal for any third-party dependency at scale; a sudden step-change is what actually indicates an incident, on either side.

Worked example

A checkout service seeing 502s from a payment provider: logs show the failures are NOT correlated with request volume (rules out simple rate limiting) but ARE correlated with a specific payment method (Apple Pay tokens specifically), while card payments succeed at the normal rate. That pattern points at the provider's Apple-Pay-specific processing path, not a general outage or a problem in the checkout service's own code, since the same service, same network path, and same general request shape succeed for card payments. Confirmed by checking the provider's status page, which shows a partial incident affecting exactly that payment method.

Trade-offs and pitfalls

Short-term mitigation while you investigate: implement a retry with backoff for 502s specifically (a 502 is often transient), and if you can identify a stable pattern like the Apple-Pay-specific example, temporarily route that payment method to a fallback or clearly surface the failure to the user rather than silently retrying a request that will keep failing the same way. The trap is assuming "third-party" automatically means "not my problem to investigate further": even a genuine provider-side incident is worth root-causing on your end, both to build an accurate mitigation and because sometimes what looks like a provider outage is actually your own malformed request that only fails for a specific payload shape.