Monitoring, Logging, and Observability Questions

Use metrics to localize which dimension (service, endpoint, region, deployment) the 5xxs are actually concentrated in before touching logging at all, then use exemplars to jump straight from that metric spike to a handful of real trace IDs, and only then turn up structured logging narrowly, scoped to the localized service and time window rather than everywhere. The fact that logs show almost nothing is itself a clue worth taking seriously: it often means the 5xx isn't being generated inside application code at all (a gateway/load-balancer timeout, a connection reset, a resource limit hit before the request ever reached a log statement), which is exactly the case where blanket log-level increases wouldn't have helped anyway.

Framework

1. Localize with metrics first. Break the 5xx rate down by every dimension you have: sum(rate(5xx[5m])) by (service, region, deployment_version). This is cheap (metrics are already low-cardinality and aggregated) and usually narrows a fleet-wide-looking problem down to one service or one deployment within minutes.

2. Attach exemplars to the metric, not just the count. An exemplar is a sampled trace ID attached to a specific bucket of a histogram or counter, so a spike in the 5xx counter or the p99 latency bucket comes with a direct link to one or more real traces that landed in that bucket. This turns "something is wrong in service X" into "here is an actual example request that failed," without a log search.

3. Follow the exemplar trace to see where the request actually died. If the trace shows the request never reaching application code (e.g. the last span is at a load balancer or a client-side connection attempt with no corresponding server span), that itself is the finding: the error is being generated upstream of the app (timeout, connection refusal, TLS/handshake failure), which explains why app-level logs show nothing, because the app never got the request or never got to finish handling it.

4. Only now, scope up logging narrowly. If the trace does show the request reaching the app and failing there, raise log verbosity (or sampling rate) for that specific service and route, for a bounded time window, rather than turning up verbosity fleet-wide, which mostly adds noise and ingestion cost without adding signal for a problem you've already localized.

Worked example

Say a 10-minute window shows 600,000 total requests split evenly across three backend services (A, B, C at 200,000 each), and 480 total 5xxs over that window, an aggregate rate of:

That 0.08% aggregate is easy to dismiss as noise. Breaking it down by service:

Service C accounts for 83.3% of all 5xxs, and its own rate is 10x the other two services' baseline rate. That's the localization: the aggregate number was hiding a strong, concentrated signal in one service, exactly what the by-dimension breakdown surfaces and a single fleet-wide number can't.

From here, an exemplar attached to C's 5xx counter points at real failing traces from C specifically, and following one of those traces (rather than searching C's logs blind) shows whether the failure originates in C's own code or in whatever C is calling.

Trade-offs and pitfalls

• Turning up DEBUG logging everywhere, which the scenario explicitly warns against, is expensive twice over: it floods the log pipeline with noise that makes the real signal harder to find, and under genuine resource pressure (which could be part of what's causing the 5xxs), the extra logging load can itself make things worse.
• Exemplars are only as useful as the histogram bucket boundaries they're attached to: a coarsely-bucketed latency histogram might not have a bucket edge anywhere near the actual latency of the failing requests, so the exemplar trace may not represent the specific failure mode you're chasing. Bucket boundaries matter and are worth reviewing before an incident, not during one.
• A gateway or load balancer that returns its own 5xx (on an upstream timeout, say) without the request ever reaching application code is a classic blind spot: application logs, application traces, and application metrics all look clean, because from the app's point of view, nothing happened. Catching this requires looking at the load balancer's own metrics/logs, which is exactly why "logs show hardly any error messages" is a clue pointing outside the app, not a dead end.
• High-cardinality tags (a specific customer ID, a specific request ID) should never be added directly to metric labels to make localization "more precise," since that turns a cheap aggregated query into an expensive, potentially cluster-destabilizing one; that level of detail belongs in the sampled traces and logs, not in the metric's label set.

Direct answer

I'd start with metrics, because they answer the cheapest, most important question first: is this an isolated blip or a real, ongoing problem, and is it affecting one endpoint or many? Metrics take seconds to check and tell you where to point traces and logs next, whereas jumping straight into logs on a single failed request risks spending several minutes reading detail for something that turns out to be a one-off retry that already resolved itself.

Prioritized first steps

1. Metrics first: check the error rate and latency (p95/p99) for the timing-out endpoint over the last 15 to 30 minutes. This shows whether it's a sustained pattern worth investigating now, or an isolated spike worth a note but not an emergency, and whether it's scoped to one endpoint or spread across the service.
2. Traces next, if the metric confirms a real pattern: pull a trace for a recent timed-out request to see which span in the call chain is actually slow or hanging. This shows where the time is going: our own code, a specific downstream dependency, or a shared resource like a connection pool.
3. Logs last, once I know where to look: with the specific span and a request ID in hand, pull the logs for that exact request to get the precise error, a timeout waiting on what exactly, instead of searching broadly through every log line mentioning "timeout."

Decision flow

flowchart TD
    A[Request times out] --> B[Check metrics first]
    B --> C{Isolated to one endpoint or widespread?}
    C -->|One endpoint| D[Pull traces for that endpoint]
    C -->|Widespread| E[Check shared dependency: DB, network, downstream API]
    D --> F[Find the slow span]
    E --> F
    F --> G[Pull logs for that span's request ID]
    G --> H[Read the error detail to confirm root cause]

If forced to pick only one signal

Under real time pressure, say a spike in slow logins and only one tool available before the next status update is due, I'd still pick metrics, not traces or logs, because metrics answer the triage question fastest: is this getting worse, better, or staying flat, and how many users are affected right now. That's what's needed to decide how urgently to escalate, even before the root cause is known. Traces or logs alone can explain one request beautifully while leaving me blind to whether the problem is spreading, which is the wrong trade-off in the first few minutes of an incident.

Trade-offs and pitfalls

• Starting in logs is tempting because it feels like "real investigation," but without a metric to scope the problem first, it's easy to spend the first several minutes reading detail about a request that isn't representative of what's actually happening.
• Traces are only useful if sampling actually captured the failing request; on a low-sample-rate service, the trace for the specific timeout you care about may simply not exist, in which case logs (which usually aren't sampled the same way) become the fallback.
• This ordering (metrics, then traces, then logs) is a good default, not a law: if metrics are known to be unreliable for this specific service, a common gap right after a migration, before dashboards catch up, it's reasonable to start with whichever signal is actually trustworthy.

Direct answer

Unstructured logging is free-form text intended for a human to read line by line; structured logging emits each entry as a consistent, machine-parsable record (typically JSON) with named fields, so a log-query tool can filter, aggregate, and correlate across millions of lines the way you'd query a database. Log levels (DEBUG, INFO, WARN, ERROR) are an orthogonal concept from structure: they control which entries you even keep or surface, so during an incident an on-call engineer isn't wading through routine noise to find the handful of lines that actually explain what broke.

Structured versus unstructured

When to log at each level, and why it matters during an incident

Worked example: a structured log line for a failed request

{
  "timestamp": "2026-07-17T15:04:05.123Z",
  "level": "ERROR",
  "service": "payments-api",
  "request_id": "req-8f21",
  "trace_id": "trace-a93c",
  "status_code": 502,
  "duration_ms": 247,
  "error_type": "UpstreamTimeout",
  "message": "upstream charge provider timed out"
}

During an incident, an on-call engineer can now do something like find all ERROR entries with error_type: UpstreamTimeout in the last 15 minutes, grouped by service, instead of searching for the word "timeout" across every service's free-text logs and hoping the wording matches. The shared trace_id also lets them jump directly from this log line to the distributed trace for the same request.

Trade-offs and pitfalls

• Logging everything at INFO "just in case" defeats the purpose of levels: if INFO volume is as high as DEBUG would be, the on-call engineer is back to searching through noise. Levels only help if they're used with discipline.
• Structured logging without a shared, enforced schema across services becomes almost as unqueryable as unstructured text, just in JSON clothing; a status_code field that's a string in one service and an integer in another breaks cross-service queries.
• Sensitive fields (user identifiers, tokens, payment details) need to be redacted or hashed at the point of logging, not cleaned up later; once something sensitive is in a log aggregator, deleting it retroactively is unreliable.
• DEBUG-level logging left on in production is a common, avoidable cost problem: log ingestion is usually billed by volume, and DEBUG noise can dominate that cost without adding proportional value.

A trace is the record of one end-to-end request as it moves through however many services it touches, identified by a single trace ID shared across all of them. A span is one timed unit of work inside that trace, typically "one service doing its part," with a start time, an end time, and a link to its parent span, so spans form a tree that shows both the order and the nesting of work. Context propagation is how that trace ID (and the current span ID, so the next service knows whose child to be) gets carried from one service to the next, almost always as HTTP headers on the outbound call.

Framework

Trace and span, concretely. If a request hits Service A, which calls Service B, which calls Service C, the whole thing is one trace, and each service's handling of its part is one span: span A (root, no parent), span B (child of A), span C (child of B). Each span records its own start/end time, so summing them up (with overlaps) reconstructs where the total request time actually went, rather than just knowing the total was slow.

How propagation actually works. The de facto standard is the W3C Trace Context header, traceparent, with the shape version-traceid-parentspanid-flags. Service A creates the trace ID and its own span ID, and sends both downstream in the header on its call to B. Service B reads that header, creates its own span as a child of A's span ID (same trace ID, new span ID), and forwards its own span ID downstream to C. Every hop repeats this: read the incoming header, create a child span, propagate your own span ID onward.

sequenceDiagram
    participant Client
    participant A as Service A
    participant B as Service B
    participant C as Service C
    Client->>A: POST /checkout (no traceparent)
    A->>A: create trace T, span S_A (root)
    A->>B: call, header traceparent: T-S_A
    B->>B: create span S_B (parent S_A)
    B->>C: call, header traceparent: T-S_B
    C->>C: create span S_C (parent S_B)
    C-->>B: response
    B-->>A: response
    A-->>Client: response

Tying it to logs and metrics. The same trace ID and span ID get attached to structured log lines emitted while handling each span, so a slow span found in the tracing UI can be clicked through to the exact log lines from that piece of the request, instead of having to guess a time window and grep for it.

Worked example

Using the flow above: the request enters as trace T with root span S_A. Service A's outbound call to B carries header traceparent: 00-T-S_A-01. Service B creates span S_B as a child of S_A, and its own outbound call to C carries traceparent: 00-T-S_B-01. If C's span turns out to take 900ms out of a 950ms total request, the trace view shows that immediately as "C is where nearly all the time went," which is the entire point: without the shared trace ID and the parent/child span links, you'd have three separate services' worth of logs and no structural way to connect "A was slow" to "actually, it was waiting on C."

Trade-offs and pitfalls

• The most common way this breaks in practice is a hop that doesn't propagate the header, most often at an async boundary (a message queue, a background job): the trace just stops there, and what should have been one connected trace becomes two disconnected fragments with no link between them.
• Sampling decisions have to travel with the trace, not be made independently at each service: if A decides to sample this trace in but C independently decides to sample it out, you get a trace with a missing leaf and no way to tell whether that's a real gap or a lost span.
• It's easy to conflate spans with logs early on: a span is specifically a timed unit of work with a parent/child relationship, not just "a log line with a timestamp." That structure (the tree, the durations) is what makes tracing useful for finding where time went, which plain timestamped logs can't reconstruct on their own.

Direct answer: Flapping means the underlying issue is real and ongoing, it just isn't crossing the alert condition cleanly. Fix it with hysteresis (a different threshold to clear an alert than to raise it) and grouping (collapse the repeated fire/clear events into one ongoing incident), not by silencing the alert, silencing would hide the exact persistent problem the question is asking you not to hide.

Structured elaboration

Applying these without hiding the real problem

• Hysteresis and debounce reduce the number of individual fire/clear notifications, but the underlying incident record should stay open and visible the entire time the condition keeps recurring, not just during the moments it's actively firing.
• Grouping should merge notifications, not merge away the evidence: the incident should retain a count of how many times it flapped and the full timestamp history, so on-call can see "this has fired 14 times in the last hour" rather than a single anonymous alert with no memory of the pattern.
• Escalate severity based on persistence, not suppress it: a condition that's been flapping for an hour is a stronger signal that something is wrong, not a weaker one, and the escalation should reflect that (raise severity or broaden the audience) rather than quietly muting it because "it keeps clearing on its own."

Worked example

A disk-usage metric oscillates between 78% and 82% because of periodic log rotation, with a single static threshold at 80%. With no hysteresis, this fires and clears repeatedly as the metric crosses 80% each cycle. Applying hysteresis with a fire threshold of 85% and a clear threshold of 75%:

• The metric's actual range (78% to 82%) never touches either the fire line (85%) or, once fired, the clear line (75%) inconsistently, it stays comfortably inside the dead zone between the two thresholds and doesn't flap.
• If disk usage genuinely climbs past 85% (a real leak, not the periodic rotation pattern), it fires once and stays fired until it drops all the way back under 75%, giving one clean incident instead of a stream of fire/clear notifications for the same underlying trend.

This is a case where the "fix" isn't detecting the flapping better, it's recognizing that a single threshold was never the right model for a metric with a normal 78 to 82% oscillation range, and setting the dead zone wide enough to cover that normal range is what actually solves it.

Trade-offs & pitfalls

• Every one of these techniques trades detection speed for noise reduction. A condition that genuinely needs an immediate page (imminent data loss, a security event) should not be debounced or given wide hysteresis, the noise-reduction techniques above are for conditions where a few minutes of delay is an acceptable cost for not paging someone six times an hour.
• Grouping by a fingerprint that's too broad merges genuinely unrelated issues into one incident (masking that there are actually two separate problems); too narrow, and it fails to group the flaps it was meant to collapse. The fingerprint needs to key on the actual root-cause dimension (host, service, error signature), not just the alert name.
• A hysteresis dead zone set too wide delays real detection meaningfully; too narrow, and it doesn't solve the flapping at all. The right width comes from looking at the metric's actual normal oscillation range (as in the worked example), not from a generic default.