Monitoring, Logging, and Observability Questions

Direct answer

Observability rests on three complementary signal types: metrics, logs, and traces. Metrics tell you something is wrong and roughly how bad; logs tell you what specifically happened in a given event; traces tell you where in a multi-service request the time or failure occurred. None of the three alone gives a complete picture: a strong incident response usually starts with one pillar to detect and scope the problem, then pivots to another to find root cause.

The three pillars

Instrumentation example, one flow

For a checkout endpoint, an on-call engineer might instrument it like this: a checkout_requests_total counter metric with labels {status, payment_provider}, plus a checkout_latency_seconds histogram with the same labels for percentiles; a structured log line at the point of failure with fields {request_id, trace_id, error_type, payment_provider}; and a trace with spans named checkout.validate, checkout.charge, checkout.persist, each carrying the same trace_id that appears in the log line. The shared trace_id and request_id are what let you jump from "the metric moved" to "here is the specific failing request" to "here is the exact log line explaining why."

Trade-offs and pitfalls

• Treating one pillar as sufficient is the most common mistake: teams that only have logs end up searching blind during an incident because they have no aggregated signal telling them where to look first; teams that only have metrics can detect a problem but can't explain it.
• High-cardinality labels (like an unbounded user_id on a metric) turn cheap metrics into an expensive, slow-to-query mess; that data belongs in logs or traces instead.
• Trace sampling is a real trade-off: full sampling captures every rare failure but is expensive at scale; low sampling rates are cheap but can miss the exact failing request you need. Tail-based sampling (keep traces for slow or error requests) is a common middle ground.
• Retention windows differ by pillar in practice (metrics are cheap to keep for months, verbose logs and full traces are usually much more expensive to retain), which shapes how far back a postmortem can actually look.

Direct answer

I use the postmortem's core questions to pick the telemetry, not the other way around: what was the blast radius, when did the symptom actually start versus when did we detect it, and when did we actually recover versus when we thought we had. Each of those maps to a specific kind of evidence, and when that evidence isn't there, I say so explicitly in the writeup and treat the missing instrumentation itself as a finding, not a footnote.

Structured elaboration

Mapping questions to telemetry

When the data isn't there

I triangulate from whatever exists rather than guessing at a number: pager events, audit logs, deploy markers, ticket timelines, even a Slack thread's timestamps can bound a window even without direct telemetry. I write the uncertainty into the postmortem explicitly (for example, "resolution was between 14:40 and 14:55 based on the last error log and the first clean synthetic check") instead of picking a single number that implies more precision than the evidence supports. If a gap kept us from answering a core question well, that becomes a remediation item, adding the missing metric or log field, with the same priority as any other action item, because a postmortem that can't reconstruct its own timeline next time is itself a reliability gap.

Worked example

Here's how the timestamps-to-metrics translation actually works, with a fully specified example timeline: the first anomalous metric (elevated error rate) appears at 14:02:00, the alert fires at 14:11:00, and the error rate returns to baseline at 14:47:00.

The 9-minute detection gap is exactly what would prompt a "why didn't this page sooner" line in the postmortem, and it's only computable because the first-anomalous-metric timestamp existed at all; if the metric that would have shown 14:02:00 wasn't being collected, the writeup could only report the 36-minute TTR from the alert and would need to flag "true onset time unknown" rather than silently starting the clock at the alert.

Trade-offs and pitfalls

The main pitfall is anchoring the whole timeline on whatever telemetry happens to be convenient (usually the alert-fired timestamp) rather than the true onset, which systematically understates detection gaps and hides exactly the problem a postmortem is supposed to surface. A related trap is averaging across metrics with different time resolutions, for example computing TTD by comparing a metric sampled every 5 minutes against a log sampled every second, without accounting for the coarser metric's own uncertainty band. Being explicit about that uncertainty is better than a false-precision number that looks rigorous but isn't.

Direct answer

Unstructured logging is free-form text intended for a human to read line by line; structured logging emits each entry as a consistent, machine-parsable record (typically JSON) with named fields, so a log-query tool can filter, aggregate, and correlate across millions of lines the way you'd query a database. Log levels (DEBUG, INFO, WARN, ERROR) are an orthogonal concept from structure: they control which entries you even keep or surface, so during an incident an on-call engineer isn't wading through routine noise to find the handful of lines that actually explain what broke.

Structured versus unstructured

When to log at each level, and why it matters during an incident

Worked example: a structured log line for a failed request

{
  "timestamp": "2026-07-17T15:04:05.123Z",
  "level": "ERROR",
  "service": "payments-api",
  "request_id": "req-8f21",
  "trace_id": "trace-a93c",
  "status_code": 502,
  "duration_ms": 247,
  "error_type": "UpstreamTimeout",
  "message": "upstream charge provider timed out"
}

During an incident, an on-call engineer can now do something like find all ERROR entries with error_type: UpstreamTimeout in the last 15 minutes, grouped by service, instead of searching for the word "timeout" across every service's free-text logs and hoping the wording matches. The shared trace_id also lets them jump directly from this log line to the distributed trace for the same request.

Trade-offs and pitfalls

• Logging everything at INFO "just in case" defeats the purpose of levels: if INFO volume is as high as DEBUG would be, the on-call engineer is back to searching through noise. Levels only help if they're used with discipline.
• Structured logging without a shared, enforced schema across services becomes almost as unqueryable as unstructured text, just in JSON clothing; a status_code field that's a string in one service and an integer in another breaks cross-service queries.
• Sensitive fields (user identifiers, tokens, payment details) need to be redacted or hashed at the point of logging, not cleaned up later; once something sensitive is in a log aggregator, deleting it retroactively is unreliable.
• DEBUG-level logging left on in production is a common, avoidable cost problem: log ingestion is usually billed by volume, and DEBUG noise can dominate that cost without adding proportional value.