Debugging and Systematic Troubleshooting Questions

Direct answer

Prioritize concurrent bugs and incidents using an explicit rubric weighing severity, user impact, reproducibility, rollback cost, and business KPIs together, not any single factor alone, since a high-severity-sounding bug with low actual user impact and an expensive rollback can rank BELOW a "smaller" bug that's actively costing revenue and has a trivial fix available.

Structured elaboration

The rubric, and how each factor is weighed:

• Severity: how bad is the failure mode itself (data loss/corruption ranks above a cosmetic issue, a security-relevant bug ranks above a performance blip), independent of how many users are currently affected.
• User impact: how many users/requests are affected right now, and is that number growing, stable, or shrinking; a severe bug affecting a handful of users may rank below a moderate bug affecting a large fraction of traffic.
• Reproducibility: a reliably reproducible bug can be diagnosed and fixed faster (lower time-to-resolution for the same engineering effort) than an intermittent one, which affects how quickly EACH candidate bug can actually be resolved if picked next, not just how bad it is.
• Rollback cost: if a bug traces to a specific recent change, how cheap and safe is reverting that change right now; a bug with a trivial, low-risk rollback available should often be handled immediately regardless of its rank on other factors, since the fix is nearly free.
• Business KPIs: which specific business metric is being affected (revenue, a contractual SLA, a compliance requirement) and how directly; a bug affecting a metric with hard, immediate business consequences (a broken payment flow) generally outranks one affecting a softer, longer-horizon metric even at similar technical severity.

Applying the rubric during a busy incident window: first, quickly triage EVERY open issue against the rubric (a few minutes per issue, not a deep investigation), to get a relative ranking rather than working issues in the order they arrived; second, look specifically for any issue with BOTH meaningful impact AND a cheap available mitigation (like a trivial rollback), since these should jump the queue regardless of their raw severity ranking, because the cost of addressing them is so low relative to the benefit; third, re-triage periodically as the window continues, since impact and severity can both change (a bug's blast radius growing, or a rollback becoming available partway through investigation of one issue), rather than treating the initial ranking as fixed for the whole incident window.

Worked example

Three concurrent issues during a busy window: (A) a high-severity-sounding data-consistency bug affecting a small, specific edge case (low current user impact, no clear quick fix, moderate rollback risk since the change is tangled with other recent work); (B) a moderate-severity bug causing a checkout-flow error for roughly 5% of transactions (clear, growing user impact, directly hitting a revenue KPI, and traced quickly to a specific recent config change with a trivial, low-risk rollback available); (C) a low-severity cosmetic UI bug reported by a few users (minimal impact, no urgency). Applying the rubric: (B) is prioritized FIRST despite being technically "less severe" than (A) in the abstract, specifically because it combines real, growing, revenue-affecting impact with an almost-free rollback fix, making it both the highest-leverage and fastest issue to resolve. (A) is prioritized second, staffed for a proper investigation given its rollback isn't cheap and its actual mechanism needs to be understood before a safe fix can be applied. (C) is deprioritized entirely for the duration of the busy window, revisited once the window calms down.

Trade-offs and pitfalls

A common mistake is prioritizing purely by SEVERITY LABEL (treating "critical" as automatically first regardless of current impact or fix cost), which can leave a rapidly-growing, revenue-affecting, cheaply-fixable issue waiting behind a technically-severe-but-currently-narrow, expensive-to-fix one; the rubric's explicit multi-factor weighing exists specifically to avoid that trap, and re-triaging periodically (rather than committing to the initial ranking for the whole window) accounts for the fact that these factors genuinely change as an incident window progresses.

Direct answer

Troubleshooting an unfamiliar problem is a loop, not a single step: observe the symptom precisely, form a small set of testable hypotheses ranked by likelihood and cost to check, gather evidence that discriminates between them, isolate the true cause, implement and validate a fix, and decide whether the incident needs a rollback, an escalation, or a written postmortem. The loop repeats: each piece of evidence should narrow the hypothesis set, not just confirm what you already believed.

Structured elaboration

1. Observe the symptom precisely. Write down exactly what is wrong, in falsifiable terms: not "the API is slow" but "p99 latency (the response time slower than 99% of requests, i.e. how bad the worst cases are, not just the average) on POST /orders rose from 80ms to 900ms starting at 14:32 UTC, affecting roughly 3% of requests." Vague symptoms produce vague hypotheses.
2. Form hypotheses before you start digging. List the plausible causes given what changed recently (deploys, config, traffic pattern, dependency versions) and what the symptom rules out. A hypothesis you cannot state is a hypothesis you cannot test.
3. Prioritize by expected information gain divided by cost. A five-minute log grep that could confirm or kill three hypotheses at once beats a one-hour deep profiling session that only speaks to one.
4. Gather evidence that discriminates. Logs tell you what happened at a point; metrics tell you the shape of the problem over time; traces tell you where time went inside one request. Pick the instrument that actually distinguishes your live hypotheses, not the one you're most comfortable with.
5. Isolate the root cause, not just a correlated symptom. A dropped hypothesis should be dropped because evidence contradicts it, not because you got bored of it.
6. Implement and validate the fix against the same evidence that revealed the problem. If you diagnosed via a specific metric, watch that metric recover before declaring victory.
7. Decide what happens next. If customer impact is ongoing and the fix is unproven, roll back first and diagnose second. If a similar failure could recur, or the incident had real impact, write it up so the org doesn't relearn the same lesson.

Worked example

A "the checkout page is slow" report, applied through the loop: symptom precisely stated as "median load time is normal, but a subset of loads takes 8-12 seconds, starting after this morning's deploy." Hypotheses: (a) the new deploy added a blocking call, (b) a downstream dependency degraded independently, (c) the slow subset shares a common attribute (e.g., a specific region or a large cart). A single log query grouping slow requests by attribute would discriminate between (c) and the other two in minutes, before touching a profiler. Suppose it shows the slow requests all hit a newly added inventory-check call to a dependency with no timeout: that both confirms (a) and rules out (b)/(c) as primary causes. Fix: add a timeout and a fallback; validate by watching the p99 metric drop back to baseline over the next hour, not just the fix compiling.

Trade-offs and pitfalls

The biggest failure mode is skipping hypothesis formation and going straight to your favorite tool (attaching a profiler because you're comfortable with it, even when a five-minute log check would have ruled out two hypotheses first). The second is treating the first correlated signal as the cause without checking whether it's actually causal. Under time pressure it's tempting to fix the first plausible thing you see; that's fine as a mitigation, but the loop isn't complete until you've confirmed the metric recovered and understood why, or you will be back debugging the same symptom next week.

Direct answer

First establish WHO owns the fault, before trying to fix anything: check whether the 502s originate from your own service (a bug in how you call the provider), the network path between you and them, your configuration (wrong endpoint, expired credentials, timeout misconfigured), or the provider itself. A 502 specifically means an upstream server returned an invalid response to a gateway, which already narrows the search: it usually means something between you and the provider's actual application server broke, not that your request was malformed (that would more often be a 4xx).

Structured elaboration

1. Check your own logs for the exact request/response pair. Capture the full request you sent (headers, body, timing) and the full 502 response, including any body the provider returned. A 502 with a body from the provider's own error page suggests an issue near their edge, not deep in their systems.
2. Check timing and correlation. Is the failure rate correlated with your request VOLUME (suggests rate limiting or a capacity issue on their side), with TIME OF DAY (suggests their scheduled maintenance or your own traffic pattern), or with a SPECIFIC request shape (suggests your own payload triggers an edge case in their processing)?
3. Rule out your own network and configuration. Confirm DNS resolves to the expected endpoint, TLS handshakes succeed, and you're not accidentally hitting a sandbox/staging URL in production. Check whether a recent config or credential change on your side coincides with when the 502s started.
4. Check the provider's status page and support channels. Many payment providers publish real-time incident status; if others report the same symptom at the same time, that's strong evidence the fault is upstream, not yours.
5. Determine if it's truly intermittent or has a pattern. A steady low background rate of 502s can be normal for any third-party dependency at scale; a sudden step-change is what actually indicates an incident, on either side.

Worked example

A checkout service seeing 502s from a payment provider: logs show the failures are NOT correlated with request volume (rules out simple rate limiting) but ARE correlated with a specific payment method (Apple Pay tokens specifically), while card payments succeed at the normal rate. That pattern points at the provider's Apple-Pay-specific processing path, not a general outage or a problem in the checkout service's own code, since the same service, same network path, and same general request shape succeed for card payments. Confirmed by checking the provider's status page, which shows a partial incident affecting exactly that payment method.

Trade-offs and pitfalls

Short-term mitigation while you investigate: implement a retry with backoff for 502s specifically (a 502 is often transient), and if you can identify a stable pattern like the Apple-Pay-specific example, temporarily route that payment method to a fallback or clearly surface the failure to the user rather than silently retrying a request that will keep failing the same way. The trap is assuming "third-party" automatically means "not my problem to investigate further": even a genuine provider-side incident is worth root-causing on your end, both to build an accurate mitigation and because sometimes what looks like a provider outage is actually your own malformed request that only fails for a specific payload shape.

Direct answer

Array.prototype.forEach does not await its callback: it fires each async callback and moves on immediately, so console.log('done') runs before any of the await processor(item) calls have resolved. The fix is to replace forEach with a construct that actually waits, either a for...of loop with await inside it (sequential) or Promise.all over items.map(...) (concurrent).

Structured elaboration

forEach was designed before async/await existed and its callback's return value, including a Promise, is simply discarded. Passing an async function as the callback doesn't change this: each invocation still returns a Promise that forEach never looks at, so forEach itself completes synchronously (having merely started every item, not finished any of them) the instant it has called the callback once per array element. The console.log('done') line after the forEach call then runs immediately, while the async work is still in flight in the background.

Worked example

async function sleep(ms) { return new Promise(r => setTimeout(r, ms)); }

// Fixed, distinct per-item delays (not Math.random()) so the interleaving below
// is deterministic and reproduces identically on every run, not just illustrative.
const delays = { 1: 20, 2: 10, 3: 30 };

// BUGGY
async function processItemBuggy(x) { await sleep(delays[x]); console.log('  processed', x, '(buggy)'); }
async function processBuggy(items) {
  items.forEach(async item => { await processItemBuggy(item); });
  console.log('done (buggy)');
}

// FIXED: sequential
async function processItemFixed(x) { await sleep(delays[x]); console.log('  processed', x, '(fixed)'); }
async function processFixed(items) {
  for (const item of items) { await processItemFixed(item); }
  console.log('done (fixed)');
}

(async () => {
  console.log('--- buggy run ---');
  await processBuggy([1, 2, 3]);
  await sleep(100); // let the buggy run's background work finish before the next section starts
  console.log('--- fixed run (sequential for-of) ---');
  await processFixed([1, 2, 3]);
})();

Executed output (buggy items finish asynchronously in the background; with the fixed per-item delays above, item 2 always resolves first, then item 1, then item 3, after done has already printed):

--- buggy run ---
done (buggy)
  processed 2 (buggy)
  processed 1 (buggy)
  processed 3 (buggy)
--- fixed run (sequential for-of) ---
  processed 1 (fixed)
  processed 2 (fixed)
  processed 3 (fixed)
done (fixed)

done (buggy) prints immediately, before any processed line: forEach fired all three async callbacks and moved on without waiting for any of them. The processed (buggy) lines do eventually print, once their fixed delays resolve, in delay order (2, then 1, then 3, not the input order), rather than before done. The for...of version correctly prints all three processed lines, strictly in input order, before done.

Trade-offs and pitfalls

There are two valid fixes with different semantics, and picking the wrong one is itself a common mistake: a for...of loop processes items strictly one at a time (useful when order matters or you must not overwhelm a downstream dependency with concurrent calls), while await Promise.all(items.map(item => processor(item))) processes all items concurrently and finishes as soon as the slowest one does (faster, but only safe if the operations are independent and the downstream system can handle concurrent load). Silently reaching for forEach out of habit, rather than deliberately choosing sequential versus concurrent semantics, is exactly how this bug class gets reintroduced even by developers who already know the rule.

Direct answer

Application-level debugging asks "why did this code produce the wrong result," and its evidence lives inside your process: variables, call stacks, exceptions, business logic. Infrastructure-level debugging asks "why couldn't the correct code even run correctly," and its evidence lives outside your process: network reachability, disk and memory pressure, DNS, container scheduling, resource limits. The two failure modes can look identical from the outside (a request fails, a page times out) while requiring completely different evidence to distinguish.

Structured elaboration

The practical test: if you could run the exact same code on a machine with unlimited, perfectly-behaving resources and a clean network, would the bug still happen? If yes, it's application-level (a logic error, a bad state transition, an unhandled edge case). If no, it's infrastructure-level (the code was never wrong, but its environment failed to give it what it needed).

Two concrete examples that look alike but diverge in root cause:

1. "Requests are timing out." Application-level cause: a newly introduced N+1 query pattern (fetching a list, then running one extra database query per item in that list instead of one combined query, so 100 items means 101 round trips instead of 1) makes one endpoint genuinely slow under load, so it exceeds a client timeout. Infrastructure-level cause: the container's memory limit is too tight, so the process is spending most of its time in garbage collection (or worse, getting OOM-killed (the operating system forcibly terminating the process because it used more memory than it was allowed to) and restarted), and the timeout is really a symptom of resource starvation that has nothing to do with the query logic. Both present as "P99 latency spiked" in a dashboard.
2. "Intermittent 500 errors on one specific endpoint." Application-level cause: a race condition in that endpoint's handler corrupts shared state under concurrent access. Infrastructure-level cause: that endpoint happens to be the one that calls out to a dependency whose DNS resolution is flaky, so the failures are really network-layer and would happen on ANY endpoint that made the same external call.

How to distinguish them in practice: check infrastructure-level signals FIRST because they're cheap and rule out a whole class of causes at once: container restarts/OOM kills, CPU throttling, disk pressure, DNS/network error rates, host-level resource graphs. If those are clean, the evidence points inward to application logic, and now stack traces, code paths, and business-logic invariants become the productive place to look. Going the other direction (deep-diving application logs before checking whether the host was even healthy) wastes time chasing red herrings in code that was never the problem.

Worked example

Given "checkout intermittently fails for large carts": check host-level memory/CPU graphs for the checkout service during the failure window (infra-level, five minutes). If those are flat and unremarkable, the next cheapest infra check is whether the failures correlate with a specific downstream dependency's error rate (still infra-adjacent, still cheap). Only once both come back clean do you profile the actual cart-processing code path, because now the evidence has ruled out "the environment failed the code" and pointed at "the code itself has a large-cart-specific bug," likely a data-size-dependent logic error, not a data-size-dependent resource limit.

Trade-offs and pitfalls

The trap is assuming the layer based on which tools you personally know best rather than what the symptom actually implies. A backend engineer with no infra background will often burn an hour reading application logs for a problem that a five-second look at a container-restart graph would have explained. The fix is cheap and disciplined: always take the five-minute infra-layer look first, even if you expect it to come back clean, because ruling it out is what makes the deeper application-level dig trustworthy.