Debugging and Systematic Troubleshooting Questions

Direct answer

Application-level debugging asks "why did this code produce the wrong result," and its evidence lives inside your process: variables, call stacks, exceptions, business logic. Infrastructure-level debugging asks "why couldn't the correct code even run correctly," and its evidence lives outside your process: network reachability, disk and memory pressure, DNS, container scheduling, resource limits. The two failure modes can look identical from the outside (a request fails, a page times out) while requiring completely different evidence to distinguish.

Structured elaboration

The practical test: if you could run the exact same code on a machine with unlimited, perfectly-behaving resources and a clean network, would the bug still happen? If yes, it's application-level (a logic error, a bad state transition, an unhandled edge case). If no, it's infrastructure-level (the code was never wrong, but its environment failed to give it what it needed).

Two concrete examples that look alike but diverge in root cause:

1. "Requests are timing out." Application-level cause: a newly introduced N+1 query pattern (fetching a list, then running one extra database query per item in that list instead of one combined query, so 100 items means 101 round trips instead of 1) makes one endpoint genuinely slow under load, so it exceeds a client timeout. Infrastructure-level cause: the container's memory limit is too tight, so the process is spending most of its time in garbage collection (or worse, getting OOM-killed (the operating system forcibly terminating the process because it used more memory than it was allowed to) and restarted), and the timeout is really a symptom of resource starvation that has nothing to do with the query logic. Both present as "P99 latency spiked" in a dashboard.
2. "Intermittent 500 errors on one specific endpoint." Application-level cause: a race condition in that endpoint's handler corrupts shared state under concurrent access. Infrastructure-level cause: that endpoint happens to be the one that calls out to a dependency whose DNS resolution is flaky, so the failures are really network-layer and would happen on ANY endpoint that made the same external call.

How to distinguish them in practice: check infrastructure-level signals FIRST because they're cheap and rule out a whole class of causes at once: container restarts/OOM kills, CPU throttling, disk pressure, DNS/network error rates, host-level resource graphs. If those are clean, the evidence points inward to application logic, and now stack traces, code paths, and business-logic invariants become the productive place to look. Going the other direction (deep-diving application logs before checking whether the host was even healthy) wastes time chasing red herrings in code that was never the problem.

Worked example

Given "checkout intermittently fails for large carts": check host-level memory/CPU graphs for the checkout service during the failure window (infra-level, five minutes). If those are flat and unremarkable, the next cheapest infra check is whether the failures correlate with a specific downstream dependency's error rate (still infra-adjacent, still cheap). Only once both come back clean do you profile the actual cart-processing code path, because now the evidence has ruled out "the environment failed the code" and pointed at "the code itself has a large-cart-specific bug," likely a data-size-dependent logic error, not a data-size-dependent resource limit.

Trade-offs and pitfalls

The trap is assuming the layer based on which tools you personally know best rather than what the symptom actually implies. A backend engineer with no infra background will often burn an hour reading application logs for a problem that a five-second look at a container-restart graph would have explained. The fix is cheap and disciplined: always take the five-minute infra-layer look first, even if you expect it to come back clean, because ruling it out is what makes the deeper application-level dig trustworthy.

Direct answer

A small, high-leverage change: adding correlation IDs and structured (rather than free-text) logging to a service that previously had neither, so that a failure in one place could actually be traced to its triggering request without manual log archaeology; the choice mattered because it directly attacked the SLOWEST part of the team's existing debugging process, not because it was the most sophisticated tooling available.

Structured elaboration

Why this specific change, not a different one: before proposing anything, the actual bottleneck in the team's debugging process was identified by observation: engineers were regularly spending the first 20-30 minutes of any investigation just correlating free-text log lines across services by eyeballing timestamps, because no shared identifier tied a request's activity together across service boundaries. That specific, repeatedly-observed cost is what made structured logging with a propagated correlation ID the highest-leverage change available, rather than a more sophisticated but less immediately impactful option (like a full distributed-tracing rollout, which would have taken much longer to implement and adopt).

How it was implemented:

1. Added a middleware/interceptor at each service's request entry point that either generates a new correlation ID (if the request is the origin) or propagates an incoming one (if it's already present from an upstream caller), attaching it to every log line emitted while handling that request.
2. Switched the logging format from free-text to structured (JSON) log lines with consistent field names across services, specifically so the correlation ID (and other common fields) could be queried directly rather than requiring text-pattern matching.
3. Rolled it out incrementally, starting with the two or three services most frequently involved in cross-service investigations, rather than attempting a big-bang change across the whole fleet at once, to prove the value and work out format conventions before wider adoption.

Measurable impact: tracked informally at first (asking engineers directly whether recent investigations felt faster) and then more concretely by comparing the TIME-TO-DIAGNOSIS on a sample of incidents before and after rollout for the services that had adopted it; investigations involving those services dropped from a typical 20-30 minutes of manual log correlation to a single structured query pulling the full cross-service trail for a given correlation ID in under a minute, a roughly 20x reduction specifically in the log-correlation phase of debugging (not the whole investigation, which still requires understanding and fixing the actual bug, but the mechanical, previously-wasted part of it).

Trade-offs and pitfalls

The change required upfront work from every team that adopted it (updating their logging calls, agreeing on field-name conventions) and some initial resistance from engineers comfortable with the existing free-text format; the case for adoption was made concrete and low-risk by piloting on a small number of services first and demonstrating the measured time savings, rather than mandating it broadly before it had proven value anywhere.

Direct answer

Stream the file line by line instead of loading it into memory, maintaining only a bounded Counter of error-type/status-code totals as you go, which keeps memory usage proportional to the number of DISTINCT keys seen rather than the number of lines in the file. Handling gzipped input just means picking the right file-opening function based on the extension; handling malformed lines means skipping them without crashing the whole scan.

Structured elaboration

import gzip
from collections import Counter
from typing import List, Tuple

def parse_log_counts(file_path: str, top_n: int = 5) -> List[Tuple[str, int]]:
    counts = Counter()
    opener = gzip.open if file_path.endswith('.gz') else open
    with opener(file_path, 'rt', errors='replace') as f:
        for line in f:
            line = line.rstrip('\n')
            if not line:
                continue
            status, exc = None, None
            for tok in line.split():
                if tok.startswith('status='):
                    status = tok.split('=', 1)[1]
                elif tok.startswith('exception='):
                    exc = tok.split('=', 1)[1]
            if exc:
                counts[f'exception:{exc}'] += 1
            elif status:
                counts[f'status:{status}'] += 1
            # lines with neither field are silently skipped as malformed
    return counts.most_common(top_n)

Key design points:

• Iterating for line in f reads one line at a time from the underlying file object rather than materializing the whole file in memory; this is what makes the function safe against a 10GB+ input.
• gzip.open versus open are chosen based on the file extension, so the same function transparently handles both compressed and uncompressed logs without the caller needing to know which.
• errors='replace' on the text-mode open prevents a single malformed byte sequence (a truncated write, a binary artifact mid-file) from raising a UnicodeDecodeError and aborting the entire scan.
• Lines with neither a status= nor exception= token are simply skipped rather than raising, satisfying "handle malformed lines" without crashing on them.
• Counter.most_common(top_n) does the top-N selection in O(k log n) where k is the number requested, more efficient than sorting the entire counts dictionary when only a few top entries are needed.

Worked example

A synthetic log with 50 status=200 lines, 12 exception=ValueError lines, 8 exception=TimeoutError lines, 20 status=404 lines, plus two malformed lines (one garbage line, one blank):

plain file:   [('status:200', 50), ('status:404', 20), ('exception:ValueError', 12), ('exception:TimeoutError', 8)]
gzipped file: [('status:200', 50), ('status:404', 20), ('exception:ValueError', 12), ('exception:TimeoutError', 8)]
top_n=2:      [('status:200', 50), ('status:404', 20)]

The gzipped and plain versions of the identical content produce byte-identical results, confirming the compression branch works correctly, and top_n=2 correctly truncates to the two highest counts without needing a separate code path.

Edge cases

• Empty file: the loop simply never executes, counts stays empty, and most_common(top_n) returns an empty list, no special-casing needed.
• File with only malformed lines: same result, an empty list, which is the correct behavior (nothing to report) rather than an error.
• A line with BOTH status= and exception=: the current implementation prioritizes exception: (checked first), which is a deliberate choice since an exception is generally the more specific and actionable signal; this priority should be called out explicitly rather than left as an implicit accident of code order.
• top_n larger than the number of distinct keys: most_common simply returns all available entries, no error.

Trade-offs and pitfalls

This implementation keeps ALL distinct keys in memory (bounded by the number of distinct error types/status codes, not the number of log lines, which is normally small), but if the log contained a very high-cardinality field mistakenly counted this way (say, counting by full request URL instead of status code), memory could still grow unboundedly with the number of distinct values; the safety guarantee here specifically relies on error types and status codes being a naturally small, bounded set.

Direct answer

A concrete story: a service occasionally returned stale pricing data to a subset of users, detected via a customer complaint rather than any internal alert (since the values were plausible-looking, just wrong, not obviously broken); the root cause traced to a caching layer that keyed its cache entries incorrectly, causing two logically-distinct pricing contexts to collide and overwrite each other's cached value, and the permanent fix corrected the cache key's uniqueness rather than just adjusting the cache's expiry time.

Structured elaboration

How it was detected: a customer support ticket reported seeing a price that didn't match what should have applied to their account tier, with no corresponding error or alert on the engineering side, since the returned value was a real, validly-formatted price, just the WRONG one; this is a useful detail because it illustrates a class of bug (returning plausible-but-wrong data) that's structurally invisible to error-rate-based monitoring, and only surfaces via a downstream consumer noticing a substantive discrepancy.

How it was reproduced: confirming the report wasn't a one-off required identifying the PATTERN, not just the single instance; checking whether other users on the same account tier around the same time window also received an unexpected price showed a small but real cluster, ruling out "one weird one-off" and confirming a systemic, reproducible mechanism worth a full investigation.

Debugging tools and techniques used: traced the pricing-lookup code path for the affected requests, and found it flows through an in-memory cache keyed, it turned out, on account tier ALONE rather than on the combination of account tier AND region (pricing legitimately varies by both); when two users on the same tier but different regions made requests close together in time, the second request's result could overwrite the first's cache entry under the shared, insufficiently-specific key, and a THIRD user (same tier, either region) arriving shortly after could then receive whichever region's price happened to be cached most recently, regardless of their own actual region.

The root cause: a cache key that didn't include every dimension the underlying value actually varied by, a classic caching-correctness bug: the cache was implicitly promising "this value is valid for anyone with this tier," when the real invariant needed was "this value is valid for anyone with this tier AND this region."

The permanent fix implemented: updated the cache key to include region alongside tier, restoring the correct invariant; also added a specific integration test that exercises exactly this scenario (two regions, same tier, interleaved requests) to catch a regression of this specific mechanism in the future, since the original bug had shipped without any test covering this particular combination of dimensions.

What you learned that helps you avoid similar bugs: whenever introducing a cache, explicitly enumerate every dimension the cached value can legitimately vary by, and verify the cache key includes ALL of them, not just the ones that happen to be obvious or top-of-mind at implementation time; a caching bug of this shape is especially dangerous specifically because it fails SILENTLY (no error, no crash, just occasionally-wrong data) and is invisible to typical error-rate monitoring, which argues for treating "does this cache key capture every dimension of variation" as a deliberate design-review question on any future caching work, not something to verify only after a bug report arrives.

Trade-offs and pitfalls

The tempting quick fix, once the symptom (stale/wrong cached price) was understood, would have been to simply reduce the cache's TTL (expiry time), which would have reduced the WINDOW during which a collision could produce visibly wrong data without fixing the actual collision mechanism at all; the permanent fix specifically addressed the cache KEY's correctness, not the expiry duration, since a shorter TTL would have masked the bug's visible frequency without removing its root cause.

Direct answer

Before handing a service to operations, verify: (1) every meaningful failure path logs enough context to diagnose it without a redeploy, (2) the four golden signals (latency, traffic, errors, saturation) are exposed as metrics, not just logs, (3) a request can be traced end to end when it crosses more than one service, (4) alert thresholds exist and point to an actual runbook, not just a page with no next step, and (5) someone other than the author has actually looked at the dashboards and confirmed they answer "is this healthy right now" at a glance.

Structured elaboration

Each item on the checklist exists because of a specific failure mode it prevents:

1. Actionable error logs. A log line that says "operation failed" with no request ID, no input summary, and no stack trace forces on-call to redeploy with more logging just to understand a 2am page. The bar: could someone who has never read this code diagnose the failure category from the log line alone?
2. The four golden signals as metrics, not just logs. Logs answer "what happened in this one case"; metrics answer "is this normal right now." Without dashboards for latency, traffic, error rate, and saturation (CPU/memory/queue depth/connection pool usage), operations has no way to distinguish a healthy blip from a developing outage without grepping logs under pressure.
3. Distributed tracing or correlation IDs. The moment a request crosses a service boundary, "check the logs" stops being a single grep and becomes "which of these five services' logs, and how do I know they're the same request?" A propagated request ID is the cheapest fix and the most commonly missing piece.
4. Alert thresholds tied to a runbook. An alert that fires with no documented first step trains on-call to snooze it, which is worse than no alert at all: it becomes noise that hides the next real incident.
5. A second set of eyes on the dashboards. The author of a service is the worst-positioned person to judge whether their own dashboard is readable to someone unfamiliar with the code, the same blind spot that makes self-review of documentation unreliable in general.

Worked example

A concrete pre-handoff review of a new payment-retry service: logs include payment_id, attempt_number, and the specific failure reason on every retry (satisfies #1); a Grafana panel shows retry rate, success rate, and queue depth (satisfies #2); the payment_id is propagated as a header to the downstream charge service so both services' logs can be joined (satisfies #3); the "retry queue depth > 500" alert links directly to a runbook section titled "Retry queue backing up" with three ranked likely causes (satisfies #4); a teammate who did not write the service opened the dashboard cold and correctly identified within 30 seconds whether the system was healthy (satisfies #5).

Trade-offs and pitfalls

The most common gap isn't missing instrumentation entirely, it's instrumentation that only makes sense to the person who wrote it: log lines with internal variable names instead of business-meaningful fields, dashboards with no annotations explaining what "normal" looks like, or alerts that reference a metric name with no context. The checklist is deliberately about READINESS for someone else to operate the system, not about whether instrumentation exists in principle.