Debugging and Systematic Troubleshooting Questions

Direct answer

Prioritize concurrent bugs and incidents using an explicit rubric weighing severity, user impact, reproducibility, rollback cost, and business KPIs together, not any single factor alone, since a high-severity-sounding bug with low actual user impact and an expensive rollback can rank BELOW a "smaller" bug that's actively costing revenue and has a trivial fix available.

Structured elaboration

The rubric, and how each factor is weighed:

• Severity: how bad is the failure mode itself (data loss/corruption ranks above a cosmetic issue, a security-relevant bug ranks above a performance blip), independent of how many users are currently affected.
• User impact: how many users/requests are affected right now, and is that number growing, stable, or shrinking; a severe bug affecting a handful of users may rank below a moderate bug affecting a large fraction of traffic.
• Reproducibility: a reliably reproducible bug can be diagnosed and fixed faster (lower time-to-resolution for the same engineering effort) than an intermittent one, which affects how quickly EACH candidate bug can actually be resolved if picked next, not just how bad it is.
• Rollback cost: if a bug traces to a specific recent change, how cheap and safe is reverting that change right now; a bug with a trivial, low-risk rollback available should often be handled immediately regardless of its rank on other factors, since the fix is nearly free.
• Business KPIs: which specific business metric is being affected (revenue, a contractual SLA, a compliance requirement) and how directly; a bug affecting a metric with hard, immediate business consequences (a broken payment flow) generally outranks one affecting a softer, longer-horizon metric even at similar technical severity.

Applying the rubric during a busy incident window: first, quickly triage EVERY open issue against the rubric (a few minutes per issue, not a deep investigation), to get a relative ranking rather than working issues in the order they arrived; second, look specifically for any issue with BOTH meaningful impact AND a cheap available mitigation (like a trivial rollback), since these should jump the queue regardless of their raw severity ranking, because the cost of addressing them is so low relative to the benefit; third, re-triage periodically as the window continues, since impact and severity can both change (a bug's blast radius growing, or a rollback becoming available partway through investigation of one issue), rather than treating the initial ranking as fixed for the whole incident window.

Worked example

Three concurrent issues during a busy window: (A) a high-severity-sounding data-consistency bug affecting a small, specific edge case (low current user impact, no clear quick fix, moderate rollback risk since the change is tangled with other recent work); (B) a moderate-severity bug causing a checkout-flow error for roughly 5% of transactions (clear, growing user impact, directly hitting a revenue KPI, and traced quickly to a specific recent config change with a trivial, low-risk rollback available); (C) a low-severity cosmetic UI bug reported by a few users (minimal impact, no urgency). Applying the rubric: (B) is prioritized FIRST despite being technically "less severe" than (A) in the abstract, specifically because it combines real, growing, revenue-affecting impact with an almost-free rollback fix, making it both the highest-leverage and fastest issue to resolve. (A) is prioritized second, staffed for a proper investigation given its rollback isn't cheap and its actual mechanism needs to be understood before a safe fix can be applied. (C) is deprioritized entirely for the duration of the busy window, revisited once the window calms down.

Trade-offs and pitfalls

A common mistake is prioritizing purely by SEVERITY LABEL (treating "critical" as automatically first regardless of current impact or fix cost), which can leave a rapidly-growing, revenue-affecting, cheaply-fixable issue waiting behind a technically-severe-but-currently-narrow, expensive-to-fix one; the rubric's explicit multi-factor weighing exists specifically to avoid that trap, and re-triaging periodically (rather than committing to the initial ranking for the whole window) accounts for the fact that these factors genuinely change as an incident window progresses.

Direct answer

A small, high-leverage change: adding correlation IDs and structured (rather than free-text) logging to a service that previously had neither, so that a failure in one place could actually be traced to its triggering request without manual log archaeology; the choice mattered because it directly attacked the SLOWEST part of the team's existing debugging process, not because it was the most sophisticated tooling available.

Structured elaboration

Why this specific change, not a different one: before proposing anything, the actual bottleneck in the team's debugging process was identified by observation: engineers were regularly spending the first 20-30 minutes of any investigation just correlating free-text log lines across services by eyeballing timestamps, because no shared identifier tied a request's activity together across service boundaries. That specific, repeatedly-observed cost is what made structured logging with a propagated correlation ID the highest-leverage change available, rather than a more sophisticated but less immediately impactful option (like a full distributed-tracing rollout, which would have taken much longer to implement and adopt).

How it was implemented:

1. Added a middleware/interceptor at each service's request entry point that either generates a new correlation ID (if the request is the origin) or propagates an incoming one (if it's already present from an upstream caller), attaching it to every log line emitted while handling that request.
2. Switched the logging format from free-text to structured (JSON) log lines with consistent field names across services, specifically so the correlation ID (and other common fields) could be queried directly rather than requiring text-pattern matching.
3. Rolled it out incrementally, starting with the two or three services most frequently involved in cross-service investigations, rather than attempting a big-bang change across the whole fleet at once, to prove the value and work out format conventions before wider adoption.

Measurable impact: tracked informally at first (asking engineers directly whether recent investigations felt faster) and then more concretely by comparing the TIME-TO-DIAGNOSIS on a sample of incidents before and after rollout for the services that had adopted it; investigations involving those services dropped from a typical 20-30 minutes of manual log correlation to a single structured query pulling the full cross-service trail for a given correlation ID in under a minute, a roughly 20x reduction specifically in the log-correlation phase of debugging (not the whole investigation, which still requires understanding and fixing the actual bug, but the mechanical, previously-wasted part of it).

Trade-offs and pitfalls

The change required upfront work from every team that adopted it (updating their logging calls, agreeing on field-name conventions) and some initial resistance from engineers comfortable with the existing free-text format; the case for adoption was made concrete and low-risk by piloting on a small number of services first and demonstrating the measured time savings, rather than mandating it broadly before it had proven value anywhere.

Direct answer

A concrete example: an ML-serving system's overall error rate looked normal, but prediction ENTROPY (a measure of how confident/certain the model's outputs were) started drifting upward over several hours, which was the actual leading indicator of an underlying feature-pipeline problem that hadn't yet produced enough outright errors to trip the standard error-rate alert.

Structured elaboration

The specific signals that mattered:

1. Prediction entropy/confidence distribution, not just correctness: a model producing predictions with steadily INCREASING entropy (more uncertain, less confident outputs) even while its measurable error rate stays within normal bounds is often an early signal that something upstream has changed, since a model facing genuinely out-of-distribution input tends to become less confident even on examples it happens to still get "right" by chance.
2. Feature-distribution drift for specific input features, checked once entropy flagged something worth investigating: comparing the live feature distributions against a recent healthy baseline for each feature the model depends on, looking for which specific feature(s) had shifted.
3. Latency percentiles, checked as a corroborating signal: a feature-computation pipeline that's degrading (not failing outright, but taking a different, slower code path, perhaps due to a cache miss or fallback logic) can show up as a latency shift correlated with the SAME time window as the entropy drift, adding independent evidence toward a shared root cause.

How these signals guided the next debugging step: the entropy drift alone wasn't specific enough to point at a cause, but it was specific enough to justify pulling feature-distribution comparisons (rather than starting from scratch with no direction); the feature-distribution check then localized the drift to ONE specific input feature whose distribution had shifted meaningfully from its historical baseline, at which point the investigation moved from "something is off" to "this specific feature's upstream computation needs checking," a much narrower and more tractable next step. Tracing that one feature's computation back further revealed a caching layer serving increasingly-stale values for it (a cache-invalidation bug introduced by an unrelated recent change), which explained both the entropy drift (the model was seeing subtly wrong values for that one feature, making it less confident) and the latency shift (the specific fallback path triggered when the cache was in this degraded state was slower than the normal path).

Worked example, made concrete: the drifting feature was a "days since last activity" recency feature, and the staleness bug meant it was increasingly under-reporting recency (making users look less recently active than they actually were) as the cache aged without proper invalidation; this shifted the model's input distribution gradually, subtly enough that no single prediction looked obviously wrong, but consistently enough to show up clearly in the aggregate entropy trend well before it would have shown up as a measurable accuracy or error-rate change.

Trade-offs and pitfalls

The value of watching a signal like prediction entropy, beyond the obvious error-rate and latency metrics, is specifically that it can surface a genuine underlying problem BEFORE it manifests as an outright, easily-alerted-on failure; the trade-off is that entropy drift alone is a fuzzy, non-specific signal (it says "something changed" without saying what), so it functions best as a trigger for a targeted follow-up investigation (feature-distribution comparison) rather than as a metric precise enough to alert directly on without a human interpreting it in context.

Direct answer

The most instructive debugging stories for ML systems usually involve a bug that LOOKS like a model-quality problem but is actually a data or infrastructure problem, since that's the exact class of confusion that's both common in ML systems specifically and genuinely hard to untangle without disciplined hypothesis testing, distinguishing this kind of story from a generic "I found a bug" narrative.

Structured elaboration

A strong answer to this question establishes: (1) Context: what the system does and what the symptom actually was, stated precisely (a specific metric degraded by a specific amount, not "the model got worse"). (2) The initial, reasonable-but-wrong hypothesis, and why it seemed plausible at the time; a genuinely challenging bug usually starts with the investigator (and often the whole team) believing the wrong thing for a while, and being honest about that is part of what makes the story credible and instructive. (3) The specific hypotheses tested, each with the tool or experiment used to test it and what it ruled in or out. (4) Why it was difficult: usually because the bug's symptom was consistent with MULTIPLE plausible causes, or because the tooling to distinguish them didn't exist yet and had to be built as part of the investigation. (5) How you persisted: what kept the investigation moving when early hypotheses failed, rather than giving up or shipping a workaround for the symptom without understanding the cause. (6) The outcome and organizational learning: not just the specific fix, but what changed about how the team works as a result (new tooling, a new validation step, a documented gotcha).

Worked example

A recommendation model's offline evaluation metrics looked fine after a retraining pipeline change, but the online A/B test showed a real, consistent quality regression. Initial hypothesis (reasonable, and initially believed): the new training data had some subtle quality issue not caught by the existing checks. Testing this: ran the new training data through every existing data-quality check, all passed, and manually spot-checked a sample, finding nothing obviously wrong, which didn't disprove the hypothesis but didn't confirm it either, so the investigation continued rather than declaring the data innocent prematurely. Second hypothesis, motivated by the offline/online metric mismatch specifically (since offline eval looked fine, the issue plausibly lived somewhere between "the model as trained" and "the model as actually served," not in the model's learned behavior itself): compared the SERVED model's raw outputs, for a fixed set of test inputs, against what the offline evaluation pipeline computed for those same inputs. This is what was genuinely difficult: no existing tooling directly compared serving-path outputs against offline-eval-path outputs for identical inputs, so building that comparison harness WAS the investigation's real work, not a side task. That comparison revealed a feature-computation discrepancy: a feature was computed slightly differently in the serving path than in the offline training/eval path (a subtle training-serving skew, one feature's normalization used a stale statistic in production that the offline pipeline had recomputed fresh), which the offline evaluation had no way to catch since it never exercised the actual serving code path at all. What kept the investigation moving through two "clean" checks that didn't pan out: treating each clean result as genuinely informative (narrowing WHERE the bug could be) rather than as a dead end, and being willing to build new tooling (the serving-vs-offline comparison harness) rather than assuming existing tools were sufficient to find every possible bug.

Organizational learning: the comparison harness built during this investigation became a permanent pre-deployment check, run automatically before any future training-pipeline or feature-computation change ships, specifically catching training-serving skew before it reaches an A/B test rather than after.

Trade-offs and pitfalls

The temptation under pressure to ship a fix for the SYMPTOM (retrain with different data, tweak the model) without understanding the actual mechanism would have left the underlying training-serving skew risk in place for the next feature change; investing in building the comparison tooling took longer than a quick symptomatic fix would have, but produced both the correct root cause AND a reusable safeguard against the whole bug class recurring.

Direct answer

Investigating a doubled LLM-serving cost after a new feature means instrumenting to see WHERE the extra compute/IO is going per request (since "cost doubled" is an aggregate that hides which specific path is responsible), reproducing the cost spike in a controlled way to confirm the new feature is genuinely the cause and not a coincidental traffic change, and then choosing an optimization proportional to the actual bottleneck rather than applying every available technique at once.

Structured elaboration

1. Instrumentation to collect a per-request compute/IO breakdown. Add timing and token-count instrumentation around each stage of request handling: input tokenization/length, model forward-pass time, output token count/length, and any retrieval or pre/post-processing steps the new feature added. Cost for most LLM-serving setups scales with total tokens processed (input plus output) and/or compute time, so knowing exactly how the NEW feature changed those numbers per request is the direct link between "a feature shipped" and "cost doubled."
2. Tests to reproduce the cost spike deliberately, rather than waiting to observe it passively in aggregate billing: replay a fixed, representative sample of requests through both the pre-feature and post-feature code paths (if the old path can still be exercised) and compare token counts and compute time directly, isolating the feature's actual marginal cost from any concurrent, unrelated traffic changes.
3. Optimization options to evaluate, matched to what the instrumentation reveals:
   • If the feature substantially increased INPUT length (e.g., it added retrieved context or expanded the prompt): consider whether all of that context is actually necessary, and whether a smaller, more targeted retrieval or a summarization step could reduce token count without harming output quality.
   • If it increased OUTPUT length or generation calls: consider caching for repeated/similar requests (if the feature's outputs are often identical or near-identical for common inputs), batching concurrent requests to improve hardware utilization even if per-request cost doesn't change, or a smaller/distilled model for the specific sub-task the new feature performs, if it doesn't need the full model's capability.
   • If it added extra model calls (e.g., a multi-step chain where the old path was single-step): consider quantization or a smaller model specifically for the new step, or restructuring to avoid a redundant call if one exists.
4. Rollout strategy for whichever optimization is chosen: validate the optimization against the SAME representative sample used to measure the original cost increase, confirming both the cost reduction and no meaningful quality regression, before a full rollout; a staged rollout (a small percentage of traffic first) lets you confirm the cost and quality effects hold at real production scale before committing fully.
5. KPIs to monitor post-optimization: cost per inference (the direct target metric), latency (since some optimizations, specifically quantization (reducing the numeric precision of the model's weights, e.g. from 32-bit to 8-bit numbers, to shrink compute and memory cost) or distillation (training a smaller, faster model to mimic a larger one's outputs), can trade a small amount of quality or slightly different latency characteristics for cost, and need to be watched to confirm they don't regress user experience), and accuracy/quality (via whatever the feature's existing quality metric is, or a small human-evaluation sample if none exists, specifically to catch a cost optimization that saved money by quietly degrading output quality).

Worked example

Per-request instrumentation shows the new feature roughly TRIPLED average input token count by prepending a large block of retrieved context to every prompt, while output length stayed roughly the same; the input-token increase alone accounts for almost the entire cost increase, since this deployment's cost model is dominated by total tokens processed. Investigating the retrieval step shows it was returning a fixed, generously-sized context window regardless of the actual query's relevance needs. The chosen optimization: a relevance-filtering step that trims retrieved context to only the passages actually likely to matter for the specific query (measured via a cheap similarity score before ever reaching the expensive model call), reducing average input tokens by roughly 60% while a staged rollout confirms output quality (measured against the feature's existing evaluation set) stays within an acceptable margin of the unfiltered baseline.

Trade-offs and pitfalls

Reaching for a broad optimization technique (distillation, quantization) without first confirming via instrumentation WHERE the cost actually went risks investing significant engineering effort in a technique that addresses the wrong bottleneck (e.g., optimizing generation speed when the real cost driver was input token count); the instrumentation step is what makes the chosen optimization's return on effort predictable rather than a guess.