Debugging and Systematic Troubleshooting Questions

Direct answer

You can capture a stack trace and the relevant local state without touching the library's source by instrumenting at the BOUNDARY where you call into it: wrap the call site in a try/catch (or its language equivalent) that re-raises after logging, install a global/uncaught-exception hook that fires regardless of where the exception originates, or attach a debugger/tracer that breaks on the exception type without modifying any code at all.

Structured elaboration

Three complementary approaches, from least to most invasive:

1. Wrap the call site, not the library. Since you control the code that CALLS into the library, even if you can't modify the library itself, catching the exception at your own call site and logging the full stack trace, the exception type/message, and any locally-relevant variables (the arguments you passed in, the current request context) before re-raising gives you a permanent, low-overhead capture point. In Python: try: library_call(...) except Exception: logger.exception("context: %s", relevant_vars); raise. In Java: a try/catch around the call that logs via e.printStackTrace() or a structured logger before rethrowing. In Node.js: wrapping in a try/catch for synchronous calls, or .catch() on the returned Promise for async ones.
2. Install a global uncaught-exception/unhandled-rejection hook. This catches cases where the exception surfaces somewhere you didn't anticipate wrapping: Python's sys.excepthook, Java's Thread.setDefaultUncaughtExceptionHandler, Node's process.on('uncaughtException', ...) and process.on('unhandledRejection', ...). These fire regardless of exactly where inside the library the exception originated, at the cost of being a last-resort catch-all rather than a precisely-scoped one.
3. Attach a debugger or tracer without changing code, when you need MORE than a stack trace, such as full local variable state at the moment of the throw. Most debuggers support "break on exception" (Python's pdb with python -m pdb, or import pdb; pdb.set_trace() combined with a signal, or a conditional breakpoint set on the exception type in a full IDE debugger; Java's debugger supports exception breakpoints directly in most IDEs) which pauses execution at the exact throw site, inside the library's own code, letting you inspect the full call stack and locals without ever editing the library.

Worked example

A third-party HTTP client library occasionally raises an unlabeled ConnectionError with no useful message, and you need to know what request triggered it. Wrapping the call site: try: response = third_party_client.get(url, timeout=5) except ConnectionError: logger.exception("third_party_client failed for url=%s, timeout=%s", url, 5); raise. This doesn't change a single line of the library, but every future occurrence now logs the exact URL and timeout that triggered it, alongside the library's own stack trace, immediately giving you the context needed to distinguish "this URL is consistently unreachable" from "this only fails under a specific timeout value."

Trade-offs and pitfalls

A global uncaught-exception hook is a safety net, not a substitute for a targeted wrap at the call site: it tells you SOMETHING failed somewhere, but without the call-site context (what arguments were in play, what business operation was in progress) it's often not enough to actually diagnose the issue, only to know it happened. The debugger approach is the most informative but requires either a reproducible local trigger or an interactive session, so it's best reserved for cases where the logged stack trace alone isn't enough to form a hypothesis.

Direct answer

First establish WHO owns the fault, before trying to fix anything: check whether the 502s originate from your own service (a bug in how you call the provider), the network path between you and them, your configuration (wrong endpoint, expired credentials, timeout misconfigured), or the provider itself. A 502 specifically means an upstream server returned an invalid response to a gateway, which already narrows the search: it usually means something between you and the provider's actual application server broke, not that your request was malformed (that would more often be a 4xx).

Structured elaboration

1. Check your own logs for the exact request/response pair. Capture the full request you sent (headers, body, timing) and the full 502 response, including any body the provider returned. A 502 with a body from the provider's own error page suggests an issue near their edge, not deep in their systems.
2. Check timing and correlation. Is the failure rate correlated with your request VOLUME (suggests rate limiting or a capacity issue on their side), with TIME OF DAY (suggests their scheduled maintenance or your own traffic pattern), or with a SPECIFIC request shape (suggests your own payload triggers an edge case in their processing)?
3. Rule out your own network and configuration. Confirm DNS resolves to the expected endpoint, TLS handshakes succeed, and you're not accidentally hitting a sandbox/staging URL in production. Check whether a recent config or credential change on your side coincides with when the 502s started.
4. Check the provider's status page and support channels. Many payment providers publish real-time incident status; if others report the same symptom at the same time, that's strong evidence the fault is upstream, not yours.
5. Determine if it's truly intermittent or has a pattern. A steady low background rate of 502s can be normal for any third-party dependency at scale; a sudden step-change is what actually indicates an incident, on either side.

Worked example

A checkout service seeing 502s from a payment provider: logs show the failures are NOT correlated with request volume (rules out simple rate limiting) but ARE correlated with a specific payment method (Apple Pay tokens specifically), while card payments succeed at the normal rate. That pattern points at the provider's Apple-Pay-specific processing path, not a general outage or a problem in the checkout service's own code, since the same service, same network path, and same general request shape succeed for card payments. Confirmed by checking the provider's status page, which shows a partial incident affecting exactly that payment method.

Trade-offs and pitfalls

Short-term mitigation while you investigate: implement a retry with backoff for 502s specifically (a 502 is often transient), and if you can identify a stable pattern like the Apple-Pay-specific example, temporarily route that payment method to a fallback or clearly surface the failure to the user rather than silently retrying a request that will keep failing the same way. The trap is assuming "third-party" automatically means "not my problem to investigate further": even a genuine provider-side incident is worth root-causing on your end, both to build an accurate mitigation and because sometimes what looks like a provider outage is actually your own malformed request that only fails for a specific payload shape.

Direct answer

Stream the file line by line instead of loading it into memory, maintaining only a bounded Counter of error-type/status-code totals as you go, which keeps memory usage proportional to the number of DISTINCT keys seen rather than the number of lines in the file. Handling gzipped input just means picking the right file-opening function based on the extension; handling malformed lines means skipping them without crashing the whole scan.

Structured elaboration

import gzip
from collections import Counter
from typing import List, Tuple

def parse_log_counts(file_path: str, top_n: int = 5) -> List[Tuple[str, int]]:
    counts = Counter()
    opener = gzip.open if file_path.endswith('.gz') else open
    with opener(file_path, 'rt', errors='replace') as f:
        for line in f:
            line = line.rstrip('\n')
            if not line:
                continue
            status, exc = None, None
            for tok in line.split():
                if tok.startswith('status='):
                    status = tok.split('=', 1)[1]
                elif tok.startswith('exception='):
                    exc = tok.split('=', 1)[1]
            if exc:
                counts[f'exception:{exc}'] += 1
            elif status:
                counts[f'status:{status}'] += 1
            # lines with neither field are silently skipped as malformed
    return counts.most_common(top_n)

Key design points:

• Iterating for line in f reads one line at a time from the underlying file object rather than materializing the whole file in memory; this is what makes the function safe against a 10GB+ input.
• gzip.open versus open are chosen based on the file extension, so the same function transparently handles both compressed and uncompressed logs without the caller needing to know which.
• errors='replace' on the text-mode open prevents a single malformed byte sequence (a truncated write, a binary artifact mid-file) from raising a UnicodeDecodeError and aborting the entire scan.
• Lines with neither a status= nor exception= token are simply skipped rather than raising, satisfying "handle malformed lines" without crashing on them.
• Counter.most_common(top_n) does the top-N selection in O(k log n) where k is the number requested, more efficient than sorting the entire counts dictionary when only a few top entries are needed.

Worked example

A synthetic log with 50 status=200 lines, 12 exception=ValueError lines, 8 exception=TimeoutError lines, 20 status=404 lines, plus two malformed lines (one garbage line, one blank):

plain file:   [('status:200', 50), ('status:404', 20), ('exception:ValueError', 12), ('exception:TimeoutError', 8)]
gzipped file: [('status:200', 50), ('status:404', 20), ('exception:ValueError', 12), ('exception:TimeoutError', 8)]
top_n=2:      [('status:200', 50), ('status:404', 20)]

The gzipped and plain versions of the identical content produce byte-identical results, confirming the compression branch works correctly, and top_n=2 correctly truncates to the two highest counts without needing a separate code path.

Edge cases

• Empty file: the loop simply never executes, counts stays empty, and most_common(top_n) returns an empty list, no special-casing needed.
• File with only malformed lines: same result, an empty list, which is the correct behavior (nothing to report) rather than an error.
• A line with BOTH status= and exception=: the current implementation prioritizes exception: (checked first), which is a deliberate choice since an exception is generally the more specific and actionable signal; this priority should be called out explicitly rather than left as an implicit accident of code order.
• top_n larger than the number of distinct keys: most_common simply returns all available entries, no error.

Trade-offs and pitfalls

This implementation keeps ALL distinct keys in memory (bounded by the number of distinct error types/status codes, not the number of log lines, which is normally small), but if the log contained a very high-cardinality field mistakenly counted this way (say, counting by full request URL instead of status code), memory could still grow unboundedly with the number of distinct values; the safety guarantee here specifically relies on error types and status codes being a naturally small, bounded set.

Direct answer

Before handing a service to operations, verify: (1) every meaningful failure path logs enough context to diagnose it without a redeploy, (2) the four golden signals (latency, traffic, errors, saturation) are exposed as metrics, not just logs, (3) a request can be traced end to end when it crosses more than one service, (4) alert thresholds exist and point to an actual runbook, not just a page with no next step, and (5) someone other than the author has actually looked at the dashboards and confirmed they answer "is this healthy right now" at a glance.

Structured elaboration

Each item on the checklist exists because of a specific failure mode it prevents:

1. Actionable error logs. A log line that says "operation failed" with no request ID, no input summary, and no stack trace forces on-call to redeploy with more logging just to understand a 2am page. The bar: could someone who has never read this code diagnose the failure category from the log line alone?
2. The four golden signals as metrics, not just logs. Logs answer "what happened in this one case"; metrics answer "is this normal right now." Without dashboards for latency, traffic, error rate, and saturation (CPU/memory/queue depth/connection pool usage), operations has no way to distinguish a healthy blip from a developing outage without grepping logs under pressure.
3. Distributed tracing or correlation IDs. The moment a request crosses a service boundary, "check the logs" stops being a single grep and becomes "which of these five services' logs, and how do I know they're the same request?" A propagated request ID is the cheapest fix and the most commonly missing piece.
4. Alert thresholds tied to a runbook. An alert that fires with no documented first step trains on-call to snooze it, which is worse than no alert at all: it becomes noise that hides the next real incident.
5. A second set of eyes on the dashboards. The author of a service is the worst-positioned person to judge whether their own dashboard is readable to someone unfamiliar with the code, the same blind spot that makes self-review of documentation unreliable in general.

Worked example

A concrete pre-handoff review of a new payment-retry service: logs include payment_id, attempt_number, and the specific failure reason on every retry (satisfies #1); a Grafana panel shows retry rate, success rate, and queue depth (satisfies #2); the payment_id is propagated as a header to the downstream charge service so both services' logs can be joined (satisfies #3); the "retry queue depth > 500" alert links directly to a runbook section titled "Retry queue backing up" with three ranked likely causes (satisfies #4); a teammate who did not write the service opened the dashboard cold and correctly identified within 30 seconds whether the system was healthy (satisfies #5).

Trade-offs and pitfalls

The most common gap isn't missing instrumentation entirely, it's instrumentation that only makes sense to the person who wrote it: log lines with internal variable names instead of business-meaningful fields, dashboards with no annotations explaining what "normal" looks like, or alerts that reference a metric name with no context. The checklist is deliberately about READINESS for someone else to operate the system, not about whether instrumentation exists in principle.

Direct answer

When standard observability has failed (logs rotated away, metrics missing, dashboards blank), the investigation has to fall back to whatever EVIDENCE still exists outside the normal instrumentation path: OS-level and host metrics that exist independent of the application's own logging, network captures showing what actually crossed the wire, user-visible outputs and error messages as a proxy for internal state, and any third-party system that independently recorded a copy of relevant activity.

Structured elaboration

When the usual "check the logs, check the dashboard" path is unavailable, the fallback sources fall into a few categories, each useful for a different kind of question:

1. OS-level and host metrics (/proc, a live snapshot of what the OS knows about running processes and memory; top/ps history, point-in-time process and resource usage; the systemd journal, the system's own log of service starts, stops, and events; and kernel logs via dmesg, low-level messages from the OS itself, including out-of-memory kills) exist independent of whatever the application chose to log. Of these, dmesg is usually the first one worth checking for a suspected resource-exhaustion incident, since it directly reports OOM kills, and can confirm or rule out resource exhaustion, OOM kills, or a process restart, even with zero application-level logging.
2. Network captures or connection state (existing packet captures if any were running for another purpose, or even just current connection state via ss/netstat if the investigation starts while the issue is still ongoing) can show what actually happened on the wire, useful when application logs would have recorded the SAME information but didn't.
3. User-visible outputs: error messages actually shown to users, HTTP status codes observed by a synthetic monitor or a customer's own tools, timestamps from customer reports, all reconstructable from OUTSIDE the system under investigation.
4. Any third-party or downstream system that independently recorded activity: a CDN's own access logs, a payment processor's transaction log, a client-side error-tracking tool, any system that received a copy of the interaction and kept its own record, even if it's not the system you'd normally check first.
5. Timing correlation across whatever fragments exist, reconstructing a plausible timeline even from incomplete sources: a customer-reported timestamp, a downstream system's transaction timestamp, and a host-level kernel log entry, cross-referenced, can often reconstruct enough of a sequence to form a solid hypothesis even with none of the "normal" evidence intact.

Worked example

An outage where the application's structured logs were rotated away before anyone realized an investigation would be needed, and the metrics dashboard shows a gap during the exact incident window due to an unrelated agent failure. Reconstructing from what remains: dmesg shows an OOM-killer event for the application's process at almost exactly the time customers first reported errors, which is a strong, independent signal pointing at memory exhaustion without needing a single application log line. Cross-referencing against a downstream payment processor's own transaction log (which independently timestamped every request it received from this service) shows request volume to it dropped to zero at the same moment, confirming the process was fully down, not just degraded. Together, these two independent, non-application sources reconstruct a timeline (memory exhaustion, process killed, requests stopped) that's just as actionable as a full log trail would have been, despite neither source being the application's own instrumentation.

System changes made afterward to avoid recurrence: extend log retention specifically to cover a reasonable incident-investigation window (not just cost-driven defaults); add redundant, INDEPENDENT monitoring (a host-level agent separate from the application's own metrics pipeline, so a single pipeline failure doesn't blind you to everything at once); and specifically add OOM/resource-exhaustion alerting sourced from the kernel/host level rather than solely from application-reported metrics, since this incident showed the application-level signal was exactly what went missing.

Trade-offs and pitfalls

The core skill here isn't a specific tool, it's the discipline of treating "our normal observability failed" as its own root-cause-worthy fact (why did logs rotate that fast, why did the metrics agent fail silently) in addition to whatever the underlying incident was, since fixing only the original incident without fixing the observability gap guarantees the NEXT incident will be just as hard to diagnose.