Debugging and Systematic Troubleshooting Questions

Direct answer

Roll back when the signals show the blast radius is growing or user-facing impact is severe and the fix isn't yet proven; keep debugging in place when the signals are stable or narrowing and you have a low-risk path to more evidence. The rubric is about trend and confidence, not just the current error rate: a flat 2% error rate you understand is safer to sit with briefly than a 0.5% rate that's climbing and whose cause is still unknown.

Structured elaboration

Concrete signals that push toward rollback:

• Error rate or latency is still climbing, not plateaued, meaning the blast radius is actively growing while you investigate.
• The failure touches a critical path (payments, auth, data writes) where continued exposure risks data integrity or revenue, not just degraded UX.
• You cannot yet state a specific hypothesis for the cause. If you don't know what's wrong, you can't bound how bad it might get, and every minute of uncertainty is a minute of unbounded risk.
• The most recent deploy is a plausible cause and rolling it back is cheap (a well-tested, low-risk rollback path exists). When rollback is cheap and the cost of being wrong is high, the asymmetry favors rolling back even before the cause is confirmed.

Signals that support continuing to debug in place:

• The metric is flat or already recovering on its own (e.g., a transient dependency blip that's clearing), meaning the risk is bounded and shrinking.
• You have a specific, testable hypothesis and the evidence to confirm or reject it is minutes away, not hours.
• The failure is isolated to a non-critical path or a small, known user segment, so continued exposure has a low, well-understood ceiling.
• Rolling back has its own real cost (loses a needed migration, reintroduces a different known bug, or the rollback path itself is unproven), making rollback riskier than the current state.

Two more factors that belong in the rubric alongside the raw signals: your technical confidence in a proposed quick patch, and your monitoring capability while you wait. A patch you are highly confident in, that you can deploy and verify within minutes, tips the balance toward fixing forward even at a moderate error rate; a patch you're guessing at does not, no matter how appealing "just ship the fix" feels under pressure. Separately, if your monitoring can only tell you the aggregate error rate but not WHO is affected or WHY, you are debugging half-blind, and that itself is a reason to prefer the safer, well-understood state (rollback) over continuing to poke at a system you can't fully observe.

Worked example

A rubric applied in production: after a deploy, error rate on the checkout endpoint rises from 0.1% to 1.5% and holds flat for four minutes while you check logs. The rate is not climbing further, the failures are concentrated in one specific edge case (carts with a discount code applied), and a log line points directly at a null-handling bug in the new discount logic. Decision: continue debugging, because the blast radius is bounded, isolated, and you have a specific hypothesis you can confirm in the next two minutes. Contrast: if that same 1.5% had climbed to 6% over those four minutes with no clear pattern in the failing requests, the correct call flips to immediate rollback, because the trend is the dominant signal, not the absolute number.

Trade-offs and pitfalls

The rubric fails when applied to a single snapshot instead of a trend: a rate that looks acceptable in isolation can be five minutes from becoming a major incident, or a rate that looks alarming can already be resolving itself. The discipline is to always ask "is this getting better, worse, or staying the same" before deciding, and to treat "I don't have a hypothesis yet" as itself a strong vote toward rollback, since it means you cannot bound the risk of waiting.

Direct answer

A 12% conversion drop for one cohort right after a UI redesign has at least four structurally different candidate causes: broken or changed event tracking (the drop is a measurement artifact, not real), an A/B bucketing error (the "treatment" cohort isn't actually who you think it is), a genuine feature shift interacting badly with the model (the redesign changed some input the recommendation model depends on), or a real behavior change (the redesign genuinely made the experience worse for this cohort). Each has a distinct, checkable signature, so the investigation should check them roughly in that order, cheapest and most likely to invalidate the whole finding first.

Structured elaboration

1. Check event tracking and instrumentation first, since a redesign is one of the most common triggers for a broken/changed event: confirm the conversion event itself still fires correctly and consistently after the redesign (a renamed button, a changed DOM structure, or a client-side tracking snippet tied to an element that no longer exists can silently stop counting conversions that are actually still happening). This is checked first because if tracking is broken, nothing else in the investigation matters until it's fixed.
2. Check A/B bucketing correctness. Confirm the affected cohort is genuinely who the experiment intends (no bucketing skew introduced by the redesign rollout mechanism itself, and no overlap/contamination between the redesigned and non-redesigned cohorts); a bucketing bug can produce a completely spurious "drop" that has nothing to do with the redesign's actual effect on user behavior.
3. Check for feature shifts affecting the recommendation model specifically. If the redesign changed what's visible, how items are laid out, or what implicit signals the model relies on (position, visibility, prior click patterns it was trained on), the model's recommendations for this cohort could have genuinely gotten worse as a side effect of the redesign changing its INPUT distribution, even though the model itself wasn't touched.
4. Check for genuine user-behavior changes, if the above are all ruled out: session recordings or funnel analysis for the affected cohort specifically, looking for where in the flow users are dropping off differently post-redesign, to distinguish "the whole flow converts worse" from "one specific new UI element is confusing or blocking users at one specific step."

Prioritized mitigation steps, matched to what's found:

• If tracking is broken: fix the instrumentation first, and DON'T conclude anything about the redesign's real impact until you have trustworthy data again, even if that means waiting for the fix to bake before re-evaluating.
• If bucketing is broken: fix the assignment logic and, depending on severity, consider whether the experiment needs to restart cleanly rather than trying to salvage contaminated data.
• If it's a feature-shift issue: this may warrant a quick model rollback or retrain specifically for the affected input distribution, buying time while a more durable fix (retraining incorporating the new UI's actual behavior patterns) is developed.
• If it's a genuine UX regression: targeted UX changes to the specific problematic element, informed by exactly where the funnel analysis showed users dropping off, rather than a broad redesign-reversion that would also undo any genuine improvements the redesign made elsewhere.

Worked example

Checking event tracking first shows conversions ARE still firing correctly, ruling out step 1. Checking bucketing shows the affected cohort is correctly and cleanly assigned, ruling out step 2. Checking recommendation-model behavior for this cohort shows its output distribution shifted meaningfully post-redesign, and tracing why shows the redesign changed the DEFAULT sort order of recommended items (a layout decision made without realizing the model's confidence-based ranking assumed a specific baseline position-bias pattern it was trained against (position bias: users tend to click higher-ranked items more often regardless of true relevance, so a model trained on one ranking order can silently misread its own confidence once that order changes)), confirming a genuine feature-shift/model-interaction issue, not a data or measurement problem, and not simply "the redesign is worse" in a vague UX sense.

Validating the root cause before committing to a fix

Tracing a plausible mechanism (the sort-order change) is not the same as having proven it's the actual cause, so before committing to reverting or retraining around the new sort order, validate it causally rather than trusting the correlation: run a small, controlled test that restores the OLD default sort order for a held-out slice of the affected cohort only (effectively a mini A/B test against the redesign's own default), and confirm conversion for that specific slice recovers toward its pre-redesign baseline. If it does, the feature-shift/model-interaction diagnosis is confirmed causally, and the full fix can be rolled out with confidence; if that held-out slice does NOT recover, the traced mechanism was only a coincidence or a partial explanation, and the investigation needs to continue rather than shipping a fix for the wrong cause. The same before-committing discipline applies to the other candidate causes: for a suspected bucketing bug, correct the bucketing for a small slice and confirm conversion normalizes before rolling the fix out broadly; for a suspected tracking bug, manually verify counts for a sample window before trusting the corrected data going forward.

Trade-offs and pitfalls

The trap in a cross-functional metric-regression investigation like this is picking the FIRST plausible-sounding explanation (often "the redesign made the UX worse," since that's the most intuitive story given the timing) without ruling out the cheaper-to-check, more mundane causes (broken tracking, bucketing errors) first; those mundane causes are also, empirically, quite common specifically around UI redesigns, and skipping them risks spending significant design and engineering effort responding to a metric drop that was never real in the first place.

Direct answer

Nightly latency spikes at constant request volume mean the cause is TIME-based, not LOAD-based, which immediately rules out anything that scales with traffic and points toward scheduled work (background jobs, cache expiry/warming cycles, scheduled maintenance) or periodic garbage-collection/compaction cycles that happen to cluster around the same time each night, independent of how many requests are actually arriving.

Structured elaboration

1. Confirm the pattern is genuinely time-correlated, not coincidentally volume-correlated. Check request-volume metrics for the exact spike windows across several nights; if volume is truly flat while latency spikes, that rules out ordinary capacity-driven causes and strongly implicates something scheduled or periodic. Alongside the metrics, pull application-level logs for the exact spike windows specifically (not just the surrounding hours), looking for warning-level entries, retry counts, or error codes clustered in that window that a metrics dashboard alone would not surface.
2. Check for scheduled background jobs running on the same host/cluster around the spike time: a cache-warming job, a log-rotation or compaction task, a batch analytics job sharing infrastructure, a scheduled backup; cross-reference the exact spike timestamps against the cron/scheduler's OWN execution logs (start time, end time, and any error or retry entries for each scheduled run), not just a summary job-history view, since a summary view can round timestamps in a way that obscures a precise correlation.
3. Check caching policies specifically for a periodic expiry pattern. If a cache (in-process or shared) has a TTL (time-to-live: how long a cached entry is kept before it expires) that causes many entries to expire around the same time each night (a fixed-time cache-refresh schedule, or a TTL set relative to a fixed daily reset rather than per-entry insertion time), the resulting simultaneous cache-miss storm can produce exactly this nightly-latency-spike-at-flat-volume signature.
4. Check for batch windows in any request-batching logic; if requests are batched with a time-based flush (rather than purely size-based), and something periodic changes the batch-fill rate at that hour (fewer concurrent requests overnight meaning batches take longer to fill and therefore wait longer before flushing, even at flat OVERALL system load), that's a plausible, easy-to-overlook mechanism, especially ironic since it's a case where LOWER traffic at night could paradoxically increase per-request latency via the batching wait.
5. Check garbage-collection patterns for a scheduled or load-triggered full GC that happens to land at a consistent time nightly (some GC strategies trigger based on heap-growth patterns that, combined with a consistent daily traffic/memory-allocation shape, can cluster around the same hour even without being explicitly scheduled); the runtime's own GC logs (pause duration and frequency per collection cycle, not just an aggregate GC-time metric) show directly whether pause TIMING specifically clusters at the same nightly window, rather than requiring that to be inferred indirectly from latency alone.
6. Check for multi-tenant interference if the infrastructure is shared: another tenant's own scheduled nightly job competing for the same underlying resources (CPU, disk IO, network) can produce a latency spike for YOUR service with zero change to your own traffic or code, visible only by checking host-level (not just your-service-level) resource metrics during the spike window.
7. Check for scheduled maintenance: automated OS patching, container/host restarts, or infrastructure-level maintenance windows that a platform team runs on a schedule independent of any application team's knowledge.

Worked example

Cross-referencing spike timestamps against the platform's job scheduler shows a company-wide log-aggregation/rotation job running nightly on shared infrastructure, and host-level (not service-level) CPU and disk-IO metrics show a clear spike during exactly the same window, confirming multi-tenant interference from that job rather than anything in the service's own code or configuration. Immediate mitigation: request the shared job be rescheduled to a lower-traffic window for this service, or move this service to isolated infrastructure if the interference is severe enough to warrant it; a code-level fix wouldn't have helped at all here, since nothing about the service's own logic was the actual cause.

Trade-offs and pitfalls

The temptation to profile the SERVICE's own code first (since that's what's directly controllable) can waste significant time when the actual cause is external, shared-infrastructure interference invisible to service-level metrics alone; checking host-level, not just service-level, resource metrics during the spike window is often the fastest way to distinguish "something in my own code" from "something external happening to my host," and should be checked early given the flat-volume/time-correlated signature already points away from the service's own request-handling logic.

Direct answer

Investigating a doubled LLM-serving cost after a new feature means instrumenting to see WHERE the extra compute/IO is going per request (since "cost doubled" is an aggregate that hides which specific path is responsible), reproducing the cost spike in a controlled way to confirm the new feature is genuinely the cause and not a coincidental traffic change, and then choosing an optimization proportional to the actual bottleneck rather than applying every available technique at once.

Structured elaboration

1. Instrumentation to collect a per-request compute/IO breakdown. Add timing and token-count instrumentation around each stage of request handling: input tokenization/length, model forward-pass time, output token count/length, and any retrieval or pre/post-processing steps the new feature added. Cost for most LLM-serving setups scales with total tokens processed (input plus output) and/or compute time, so knowing exactly how the NEW feature changed those numbers per request is the direct link between "a feature shipped" and "cost doubled."
2. Tests to reproduce the cost spike deliberately, rather than waiting to observe it passively in aggregate billing: replay a fixed, representative sample of requests through both the pre-feature and post-feature code paths (if the old path can still be exercised) and compare token counts and compute time directly, isolating the feature's actual marginal cost from any concurrent, unrelated traffic changes.
3. Optimization options to evaluate, matched to what the instrumentation reveals:
   • If the feature substantially increased INPUT length (e.g., it added retrieved context or expanded the prompt): consider whether all of that context is actually necessary, and whether a smaller, more targeted retrieval or a summarization step could reduce token count without harming output quality.
   • If it increased OUTPUT length or generation calls: consider caching for repeated/similar requests (if the feature's outputs are often identical or near-identical for common inputs), batching concurrent requests to improve hardware utilization even if per-request cost doesn't change, or a smaller/distilled model for the specific sub-task the new feature performs, if it doesn't need the full model's capability.
   • If it added extra model calls (e.g., a multi-step chain where the old path was single-step): consider quantization or a smaller model specifically for the new step, or restructuring to avoid a redundant call if one exists.
4. Rollout strategy for whichever optimization is chosen: validate the optimization against the SAME representative sample used to measure the original cost increase, confirming both the cost reduction and no meaningful quality regression, before a full rollout; a staged rollout (a small percentage of traffic first) lets you confirm the cost and quality effects hold at real production scale before committing fully.
5. KPIs to monitor post-optimization: cost per inference (the direct target metric), latency (since some optimizations, specifically quantization (reducing the numeric precision of the model's weights, e.g. from 32-bit to 8-bit numbers, to shrink compute and memory cost) or distillation (training a smaller, faster model to mimic a larger one's outputs), can trade a small amount of quality or slightly different latency characteristics for cost, and need to be watched to confirm they don't regress user experience), and accuracy/quality (via whatever the feature's existing quality metric is, or a small human-evaluation sample if none exists, specifically to catch a cost optimization that saved money by quietly degrading output quality).

Worked example

Per-request instrumentation shows the new feature roughly TRIPLED average input token count by prepending a large block of retrieved context to every prompt, while output length stayed roughly the same; the input-token increase alone accounts for almost the entire cost increase, since this deployment's cost model is dominated by total tokens processed. Investigating the retrieval step shows it was returning a fixed, generously-sized context window regardless of the actual query's relevance needs. The chosen optimization: a relevance-filtering step that trims retrieved context to only the passages actually likely to matter for the specific query (measured via a cheap similarity score before ever reaching the expensive model call), reducing average input tokens by roughly 60% while a staged rollout confirms output quality (measured against the feature's existing evaluation set) stays within an acceptable margin of the unfiltered baseline.

Trade-offs and pitfalls

Reaching for a broad optimization technique (distillation, quantization) without first confirming via instrumentation WHERE the cost actually went risks investing significant engineering effort in a technique that addresses the wrong bottleneck (e.g., optimizing generation speed when the real cost driver was input token count); the instrumentation step is what makes the chosen optimization's return on effort predictable rather than a guess.

Direct answer

Prioritize concurrent bugs and incidents using an explicit rubric weighing severity, user impact, reproducibility, rollback cost, and business KPIs together, not any single factor alone, since a high-severity-sounding bug with low actual user impact and an expensive rollback can rank BELOW a "smaller" bug that's actively costing revenue and has a trivial fix available.

Structured elaboration

The rubric, and how each factor is weighed:

• Severity: how bad is the failure mode itself (data loss/corruption ranks above a cosmetic issue, a security-relevant bug ranks above a performance blip), independent of how many users are currently affected.
• User impact: how many users/requests are affected right now, and is that number growing, stable, or shrinking; a severe bug affecting a handful of users may rank below a moderate bug affecting a large fraction of traffic.
• Reproducibility: a reliably reproducible bug can be diagnosed and fixed faster (lower time-to-resolution for the same engineering effort) than an intermittent one, which affects how quickly EACH candidate bug can actually be resolved if picked next, not just how bad it is.
• Rollback cost: if a bug traces to a specific recent change, how cheap and safe is reverting that change right now; a bug with a trivial, low-risk rollback available should often be handled immediately regardless of its rank on other factors, since the fix is nearly free.
• Business KPIs: which specific business metric is being affected (revenue, a contractual SLA, a compliance requirement) and how directly; a bug affecting a metric with hard, immediate business consequences (a broken payment flow) generally outranks one affecting a softer, longer-horizon metric even at similar technical severity.

Applying the rubric during a busy incident window: first, quickly triage EVERY open issue against the rubric (a few minutes per issue, not a deep investigation), to get a relative ranking rather than working issues in the order they arrived; second, look specifically for any issue with BOTH meaningful impact AND a cheap available mitigation (like a trivial rollback), since these should jump the queue regardless of their raw severity ranking, because the cost of addressing them is so low relative to the benefit; third, re-triage periodically as the window continues, since impact and severity can both change (a bug's blast radius growing, or a rollback becoming available partway through investigation of one issue), rather than treating the initial ranking as fixed for the whole incident window.

Worked example

Three concurrent issues during a busy window: (A) a high-severity-sounding data-consistency bug affecting a small, specific edge case (low current user impact, no clear quick fix, moderate rollback risk since the change is tangled with other recent work); (B) a moderate-severity bug causing a checkout-flow error for roughly 5% of transactions (clear, growing user impact, directly hitting a revenue KPI, and traced quickly to a specific recent config change with a trivial, low-risk rollback available); (C) a low-severity cosmetic UI bug reported by a few users (minimal impact, no urgency). Applying the rubric: (B) is prioritized FIRST despite being technically "less severe" than (A) in the abstract, specifically because it combines real, growing, revenue-affecting impact with an almost-free rollback fix, making it both the highest-leverage and fastest issue to resolve. (A) is prioritized second, staffed for a proper investigation given its rollback isn't cheap and its actual mechanism needs to be understood before a safe fix can be applied. (C) is deprioritized entirely for the duration of the busy window, revisited once the window calms down.

Trade-offs and pitfalls

A common mistake is prioritizing purely by SEVERITY LABEL (treating "critical" as automatically first regardless of current impact or fix cost), which can leave a rapidly-growing, revenue-affecting, cheaply-fixable issue waiting behind a technically-severe-but-currently-narrow, expensive-to-fix one; the rubric's explicit multi-factor weighing exists specifically to avoid that trap, and re-triaging periodically (rather than committing to the initial ranking for the whole window) accounts for the fact that these factors genuinely change as an incident window progresses.