Debugging and Systematic Troubleshooting Questions

Direct answer

The bug is that lst=[] creates the default list once, at function-definition time, not once per call. Every call that omits the lst argument reuses the exact same list object, so items accumulate across calls that were never meant to share state. The fix is to use None as the sentinel default and create a fresh list inside the function body when no list was passed.

Structured elaboration

In Python, default argument values are evaluated exactly once, when the def statement executes, and the resulting object is bound to the function itself. A mutable default (a list, dict, or set) is therefore the SAME object across every call that relies on the default, so mutating it in one call is visible in the next. This is a well-known Python pitfall precisely because the code reads as if a fresh list is created per call, which is the intuitive (and correct) behavior for immutable defaults like None, 0, or "", but not for mutable ones.

Worked example

def append_item_buggy(item, lst=[]):
    lst.append(item)
    return lst

def append_item_fixed(item, lst=None):
    if lst is None:
        lst = []
    lst.append(item)
    return lst

print("BUGGY:")
print(append_item_buggy(1))
print(append_item_buggy(2))

print("FIXED:")
print(append_item_fixed(1))
print(append_item_fixed(2))

Executed output:

BUGGY:
[1]
[1, 2]
FIXED:
[1]
[2]

The buggy version's second call returns [1, 2] instead of the expected [2], because it's still appending to the exact same list object created back when the function was defined. The fixed version creates a brand-new list inside the function body on every call where the caller didn't explicitly pass one, so each call starts clean.

Trade-offs and pitfalls

This bug is especially dangerous because it's silent and cumulative: a function like this can appear to work correctly in isolated tests (each test creates its own scenario and might not notice the shared state) and only manifest as data mysteriously growing or leaking between unrelated calls once the function is used repeatedly in a long-running process, which is exactly the kind of "works in tests, fails in production" gap this class of bug produces. The general rule: never use a mutable object (list, dict, set, or a custom mutable class instance) as a default argument value; use None and construct the mutable object inside the function body instead.

Direct answer

Roll back when the signals show the blast radius is growing or user-facing impact is severe and the fix isn't yet proven; keep debugging in place when the signals are stable or narrowing and you have a low-risk path to more evidence. The rubric is about trend and confidence, not just the current error rate: a flat 2% error rate you understand is safer to sit with briefly than a 0.5% rate that's climbing and whose cause is still unknown.

Structured elaboration

Concrete signals that push toward rollback:

• Error rate or latency is still climbing, not plateaued, meaning the blast radius is actively growing while you investigate.
• The failure touches a critical path (payments, auth, data writes) where continued exposure risks data integrity or revenue, not just degraded UX.
• You cannot yet state a specific hypothesis for the cause. If you don't know what's wrong, you can't bound how bad it might get, and every minute of uncertainty is a minute of unbounded risk.
• The most recent deploy is a plausible cause and rolling it back is cheap (a well-tested, low-risk rollback path exists). When rollback is cheap and the cost of being wrong is high, the asymmetry favors rolling back even before the cause is confirmed.

Signals that support continuing to debug in place:

• The metric is flat or already recovering on its own (e.g., a transient dependency blip that's clearing), meaning the risk is bounded and shrinking.
• You have a specific, testable hypothesis and the evidence to confirm or reject it is minutes away, not hours.
• The failure is isolated to a non-critical path or a small, known user segment, so continued exposure has a low, well-understood ceiling.
• Rolling back has its own real cost (loses a needed migration, reintroduces a different known bug, or the rollback path itself is unproven), making rollback riskier than the current state.

Two more factors that belong in the rubric alongside the raw signals: your technical confidence in a proposed quick patch, and your monitoring capability while you wait. A patch you are highly confident in, that you can deploy and verify within minutes, tips the balance toward fixing forward even at a moderate error rate; a patch you're guessing at does not, no matter how appealing "just ship the fix" feels under pressure. Separately, if your monitoring can only tell you the aggregate error rate but not WHO is affected or WHY, you are debugging half-blind, and that itself is a reason to prefer the safer, well-understood state (rollback) over continuing to poke at a system you can't fully observe.

Worked example

A rubric applied in production: after a deploy, error rate on the checkout endpoint rises from 0.1% to 1.5% and holds flat for four minutes while you check logs. The rate is not climbing further, the failures are concentrated in one specific edge case (carts with a discount code applied), and a log line points directly at a null-handling bug in the new discount logic. Decision: continue debugging, because the blast radius is bounded, isolated, and you have a specific hypothesis you can confirm in the next two minutes. Contrast: if that same 1.5% had climbed to 6% over those four minutes with no clear pattern in the failing requests, the correct call flips to immediate rollback, because the trend is the dominant signal, not the absolute number.

Trade-offs and pitfalls

The rubric fails when applied to a single snapshot instead of a trend: a rate that looks acceptable in isolation can be five minutes from becoming a major incident, or a rate that looks alarming can already be resolving itself. The discipline is to always ask "is this getting better, worse, or staying the same" before deciding, and to treat "I don't have a hypothesis yet" as itself a strong vote toward rollback, since it means you cannot bound the risk of waiting.

Direct answer

Troubleshooting an unfamiliar problem is a loop, not a single step: observe the symptom precisely, form a small set of testable hypotheses ranked by likelihood and cost to check, gather evidence that discriminates between them, isolate the true cause, implement and validate a fix, and decide whether the incident needs a rollback, an escalation, or a written postmortem. The loop repeats: each piece of evidence should narrow the hypothesis set, not just confirm what you already believed.

Structured elaboration

1. Observe the symptom precisely. Write down exactly what is wrong, in falsifiable terms: not "the API is slow" but "p99 latency (the response time slower than 99% of requests, i.e. how bad the worst cases are, not just the average) on POST /orders rose from 80ms to 900ms starting at 14:32 UTC, affecting roughly 3% of requests." Vague symptoms produce vague hypotheses.
2. Form hypotheses before you start digging. List the plausible causes given what changed recently (deploys, config, traffic pattern, dependency versions) and what the symptom rules out. A hypothesis you cannot state is a hypothesis you cannot test.
3. Prioritize by expected information gain divided by cost. A five-minute log grep that could confirm or kill three hypotheses at once beats a one-hour deep profiling session that only speaks to one.
4. Gather evidence that discriminates. Logs tell you what happened at a point; metrics tell you the shape of the problem over time; traces tell you where time went inside one request. Pick the instrument that actually distinguishes your live hypotheses, not the one you're most comfortable with.
5. Isolate the root cause, not just a correlated symptom. A dropped hypothesis should be dropped because evidence contradicts it, not because you got bored of it.
6. Implement and validate the fix against the same evidence that revealed the problem. If you diagnosed via a specific metric, watch that metric recover before declaring victory.
7. Decide what happens next. If customer impact is ongoing and the fix is unproven, roll back first and diagnose second. If a similar failure could recur, or the incident had real impact, write it up so the org doesn't relearn the same lesson.

Worked example

A "the checkout page is slow" report, applied through the loop: symptom precisely stated as "median load time is normal, but a subset of loads takes 8-12 seconds, starting after this morning's deploy." Hypotheses: (a) the new deploy added a blocking call, (b) a downstream dependency degraded independently, (c) the slow subset shares a common attribute (e.g., a specific region or a large cart). A single log query grouping slow requests by attribute would discriminate between (c) and the other two in minutes, before touching a profiler. Suppose it shows the slow requests all hit a newly added inventory-check call to a dependency with no timeout: that both confirms (a) and rules out (b)/(c) as primary causes. Fix: add a timeout and a fallback; validate by watching the p99 metric drop back to baseline over the next hour, not just the fix compiling.

Trade-offs and pitfalls

The biggest failure mode is skipping hypothesis formation and going straight to your favorite tool (attaching a profiler because you're comfortable with it, even when a five-minute log check would have ruled out two hypotheses first). The second is treating the first correlated signal as the cause without checking whether it's actually causal. Under time pressure it's tempting to fix the first plausible thing you see; that's fine as a mitigation, but the loop isn't complete until you've confirmed the metric recovered and understood why, or you will be back debugging the same symptom next week.

Direct answer

Stream the file line by line instead of loading it into memory, maintaining only a bounded Counter of error-type/status-code totals as you go, which keeps memory usage proportional to the number of DISTINCT keys seen rather than the number of lines in the file. Handling gzipped input just means picking the right file-opening function based on the extension; handling malformed lines means skipping them without crashing the whole scan.

Structured elaboration

import gzip
from collections import Counter
from typing import List, Tuple

def parse_log_counts(file_path: str, top_n: int = 5) -> List[Tuple[str, int]]:
    counts = Counter()
    opener = gzip.open if file_path.endswith('.gz') else open
    with opener(file_path, 'rt', errors='replace') as f:
        for line in f:
            line = line.rstrip('\n')
            if not line:
                continue
            status, exc = None, None
            for tok in line.split():
                if tok.startswith('status='):
                    status = tok.split('=', 1)[1]
                elif tok.startswith('exception='):
                    exc = tok.split('=', 1)[1]
            if exc:
                counts[f'exception:{exc}'] += 1
            elif status:
                counts[f'status:{status}'] += 1
            # lines with neither field are silently skipped as malformed
    return counts.most_common(top_n)

Key design points:

• Iterating for line in f reads one line at a time from the underlying file object rather than materializing the whole file in memory; this is what makes the function safe against a 10GB+ input.
• gzip.open versus open are chosen based on the file extension, so the same function transparently handles both compressed and uncompressed logs without the caller needing to know which.
• errors='replace' on the text-mode open prevents a single malformed byte sequence (a truncated write, a binary artifact mid-file) from raising a UnicodeDecodeError and aborting the entire scan.
• Lines with neither a status= nor exception= token are simply skipped rather than raising, satisfying "handle malformed lines" without crashing on them.
• Counter.most_common(top_n) does the top-N selection in O(k log n) where k is the number requested, more efficient than sorting the entire counts dictionary when only a few top entries are needed.

Worked example

A synthetic log with 50 status=200 lines, 12 exception=ValueError lines, 8 exception=TimeoutError lines, 20 status=404 lines, plus two malformed lines (one garbage line, one blank):

plain file:   [('status:200', 50), ('status:404', 20), ('exception:ValueError', 12), ('exception:TimeoutError', 8)]
gzipped file: [('status:200', 50), ('status:404', 20), ('exception:ValueError', 12), ('exception:TimeoutError', 8)]
top_n=2:      [('status:200', 50), ('status:404', 20)]

The gzipped and plain versions of the identical content produce byte-identical results, confirming the compression branch works correctly, and top_n=2 correctly truncates to the two highest counts without needing a separate code path.

Edge cases

• Empty file: the loop simply never executes, counts stays empty, and most_common(top_n) returns an empty list, no special-casing needed.
• File with only malformed lines: same result, an empty list, which is the correct behavior (nothing to report) rather than an error.
• A line with BOTH status= and exception=: the current implementation prioritizes exception: (checked first), which is a deliberate choice since an exception is generally the more specific and actionable signal; this priority should be called out explicitly rather than left as an implicit accident of code order.
• top_n larger than the number of distinct keys: most_common simply returns all available entries, no error.

Trade-offs and pitfalls

This implementation keeps ALL distinct keys in memory (bounded by the number of distinct error types/status codes, not the number of log lines, which is normally small), but if the log contained a very high-cardinality field mistakenly counted this way (say, counting by full request URL instead of status code), memory could still grow unboundedly with the number of distinct values; the safety guarantee here specifically relies on error types and status codes being a naturally small, bounded set.

Direct answer

High CPU with low user-visible latency means the CPU work isn't on the critical path the user is waiting on: it's either background work (batch jobs, garbage collection, async processing), wasted work (a busy-loop, redundant recomputation, an inefficient algorithm that happens to still finish fast enough), or a measurement artifact (CPU metrics counting time the process spends idle-but-scheduled, or double-counting across containers sharing a host). The investigation is about separating these three, not assuming the first plausible one.

Structured elaboration

1. Check whether the CPU usage correlates with request volume or is constant/background. If CPU stays high even during low-traffic periods, it's very unlikely to be request-handling work; that points toward background jobs, scheduled tasks, or a stuck loop.
2. Profile what's actually consuming CPU, using a sampling profiler (perf, py-spy, async-profiler depending on the runtime) rather than guessing from code review. Distinguish CPU time attributed to request handlers versus background threads, garbage collection, or monitoring/logging agents.
3. Check for wasted work specifically: a cache that isn't actually being hit (so every request redoes expensive work that should have been cached), a retry loop that's spinning faster than intended, or an algorithm with much worse complexity than necessary that still completes within the user's timeout because the dataset happens to be small today.
4. Rule out measurement artifacts. On containerized/shared hosts, CPU metrics can reflect cgroup accounting quirks (cgroups are the Linux kernel mechanism that caps and meters how much CPU/memory a container is allowed to use, and its usage counters can misreport in edge cases), CPU throttling counted as "usage," or a host-level metric that aggregates multiple co-located processes. Confirm the metric is scoped to the process you think it is.
5. Distinguish "wasted but harmless today" from "a ticking time bomb." Work that's wasted but currently fits comfortably within capacity may not need urgent action; the same wasted work will become a real incident the moment traffic grows or the host loses spare capacity, so it's worth flagging even if latency looks fine right now.

Worked example

A service with 70% average CPU but P99 latency (the response time slower than 99% of requests, a common way to track worst-case rather than average experience) well within its SLA (service-level agreement, the target it's contractually or operationally expected to meet): profiling shows 40% of that CPU is spent in a background reconciliation job that runs every 30 seconds regardless of load, unrelated to request handling entirely. That explains the discrepancy directly: the CPU number reflects background work, not the request path the user experiences. The remediation is about the background job's efficiency and scheduling (should it run every 30s, or can it be event-driven instead), not about the request-handling code the on-call engineer might otherwise be tempted to optimize first.

Trade-offs and pitfalls

The main trap is treating "CPU is high" and "users are impacted" as the same finding when they can be completely decoupled, as in this example. Optimizing the wrong thing (request-handler code, when the real cost is a background job) wastes effort and leaves the actual capacity risk unaddressed. The remediation priority should follow from WHERE the profiler says the time goes, not from where it would be most convenient to look.