Monitoring, Logging, and Observability Questions

Log only an explicit allowlist of fields (never a raw dump of a request or user object), transform anything that identifies a person before it's serialized (hash it with a per-environment salt if you need to correlate events without storing the raw value, or drop it entirely if you don't), and enforce that transform in a shared logging library so no individual call site has to remember. "Verify it in practice" means treating this like any other production guarantee: automated tests on the redaction function itself, plus a periodic scan of what actually got shipped, not a one-time code review.

Framework

Field classification. Split fields into three buckets before deciding what goes in a log line:

• Safe to log directly: service, env, timestamp, request_id/trace_id, route, status, latency_ms, error_code. None of these identify a person.
• Needs a transform before logging: user/customer identifiers. Hash with a per-environment salt (so the same hash can't be correlated across environments, and isn't trivially reversible by brute force against a small ID space) if downstream tooling needs to group events by user without knowing who the user is; otherwise, omit entirely.
• Never logged: raw email, password, SSN, full payment details, or a raw request/response body that might contain any of the above nested inside it.

Where enforcement lives. Put a redact() pass in the shared logging library, applied to every structured payload right before serialization, walking nested objects and arrays recursively rather than only checking top-level keys (a sensitive field nested inside a metadata object is just as much a leak as one at the top level). Default to an allowlist (log nothing outside a defined set of fields) rather than a denylist (log everything except a list of known-bad keys), because a denylist fails open: a brand-new field added next sprint gets logged raw by default until someone remembers to add it to the denylist. An allowlist fails closed: a new field simply doesn't appear in logs until someone deliberately adds it, which is the safer default direction to fail in.

Defense in depth at the shipping layer. Even with library-level enforcement, add a second, independent scan in the log-shipping pipeline that pattern-matches for known PII shapes (email format, SSN format, credit-card-shaped digit sequences) and redacts or quarantines anything it finds, treating this as a backstop for the rare bug or bypass, not the primary control.

Verifying no leakage in practice. Two separate checks, testing two separate things:

• Unit tests on the redaction function itself: known sensitive keys and values get replaced, nested objects and arrays are covered, and a few common secret-shaped strings (a JWT-looking token, a sk_live_-style API key pattern) never survive redaction, run in CI against every change to the logging library.
• A periodic (not one-time) scan of a sample of actually-shipped logs, running the same pattern-matchers as the shipping-layer backstop, that fails a build or pages someone if it finds a hit. This is what actually catches the gap between "the library was supposed to redact this" and "it did," since library bugs and new fields that slipped past the allowlist review both show up here first.

Worked example

Take an unredacted log candidate an application might otherwise emit:

{"user_id": "u_48213", "email": "j.smith@example.com", "action": "checkout", "amount_cents": 4999}

email is dropped entirely (never-log bucket). user_id is correlatable but not directly identifying, so it's hashed with a per-environment salt rather than logged raw. Using salt prod-2026-salt and computing SHA-256("prod-2026-salt|u_48213"):

giving a final logged line of:

{"user_id_hash": "086f4b2b73ba655c20b555ad4be7c5cb7e25516786d0ed29d7c51498544a830b", "action": "checkout", "amount_cents": 4999}

This line still supports "how many checkout events did this same user have this week" (group by user_id_hash) without anyone reading the logs being able to read off the raw user_id or the email address at all. The hash is deterministic (same salt, same input, always the same output), which is what makes the correlation use case still work.

Trade-offs and pitfalls

• Hashing alone doesn't protect a small or guessable ID space: if user_id values are short sequential integers, an attacker with the salt (or without it, if it's not actually secret) can just hash every candidate ID and match against the logged hash, a straightforward brute-force/rainbow-table-style attack. A salt raises the bar but doesn't fix a fundamentally small ID space; for genuinely sensitive correlation needs, a keyed HMAC (a hash computed together with a secret key, so unlike a plain salted hash it can't be recomputed by anyone without that key) or a separate tokenization service is the stronger design.
• An allowlist has an ongoing maintenance cost: every new field a team wants to log requires a deliberate addition to the allowlist, which can feel like friction. That friction is the point (it forces a conscious decision about whether a new field is safe), but it does mean the review process for adding fields has to be fast enough that people don't route around it by nesting new data inside an already-allowed catch-all field, which would defeat the whole design.
• The shipping-layer pattern-match backstop can be evaded by anything that doesn't match the expected shape (unusual formatting, base64-encoded PII, a non-US ID format), so it should never be treated as the primary control, only as a net that catches obvious mistakes the library-level redaction missed.
• Redacting too aggressively has its own cost: dropping a field that turns out to be genuinely needed for debugging (and wasn't actually PII) makes incident response harder. Field classification should be reviewed with both privacy and debuggability in mind, not privacy alone.

Direct answer

I use the postmortem's core questions to pick the telemetry, not the other way around: what was the blast radius, when did the symptom actually start versus when did we detect it, and when did we actually recover versus when we thought we had. Each of those maps to a specific kind of evidence, and when that evidence isn't there, I say so explicitly in the writeup and treat the missing instrumentation itself as a finding, not a footnote.

Structured elaboration

Mapping questions to telemetry

When the data isn't there

I triangulate from whatever exists rather than guessing at a number: pager events, audit logs, deploy markers, ticket timelines, even a Slack thread's timestamps can bound a window even without direct telemetry. I write the uncertainty into the postmortem explicitly (for example, "resolution was between 14:40 and 14:55 based on the last error log and the first clean synthetic check") instead of picking a single number that implies more precision than the evidence supports. If a gap kept us from answering a core question well, that becomes a remediation item, adding the missing metric or log field, with the same priority as any other action item, because a postmortem that can't reconstruct its own timeline next time is itself a reliability gap.

Worked example

Here's how the timestamps-to-metrics translation actually works, with a fully specified example timeline: the first anomalous metric (elevated error rate) appears at 14:02:00, the alert fires at 14:11:00, and the error rate returns to baseline at 14:47:00.

The 9-minute detection gap is exactly what would prompt a "why didn't this page sooner" line in the postmortem, and it's only computable because the first-anomalous-metric timestamp existed at all; if the metric that would have shown 14:02:00 wasn't being collected, the writeup could only report the 36-minute TTR from the alert and would need to flag "true onset time unknown" rather than silently starting the clock at the alert.

Trade-offs and pitfalls

The main pitfall is anchoring the whole timeline on whatever telemetry happens to be convenient (usually the alert-fired timestamp) rather than the true onset, which systematically understates detection gaps and hides exactly the problem a postmortem is supposed to surface. A related trap is averaging across metrics with different time resolutions, for example computing TTD by comparing a metric sampled every 5 minutes against a log sampled every second, without accounting for the coarser metric's own uncertainty band. Being explicit about that uncertainty is better than a false-precision number that looks rigorous but isn't.

Direct answer

Unstructured logging is free-form text intended for a human to read line by line; structured logging emits each entry as a consistent, machine-parsable record (typically JSON) with named fields, so a log-query tool can filter, aggregate, and correlate across millions of lines the way you'd query a database. Log levels (DEBUG, INFO, WARN, ERROR) are an orthogonal concept from structure: they control which entries you even keep or surface, so during an incident an on-call engineer isn't wading through routine noise to find the handful of lines that actually explain what broke.

Structured versus unstructured

When to log at each level, and why it matters during an incident

Worked example: a structured log line for a failed request

{
  "timestamp": "2026-07-17T15:04:05.123Z",
  "level": "ERROR",
  "service": "payments-api",
  "request_id": "req-8f21",
  "trace_id": "trace-a93c",
  "status_code": 502,
  "duration_ms": 247,
  "error_type": "UpstreamTimeout",
  "message": "upstream charge provider timed out"
}

During an incident, an on-call engineer can now do something like find all ERROR entries with error_type: UpstreamTimeout in the last 15 minutes, grouped by service, instead of searching for the word "timeout" across every service's free-text logs and hoping the wording matches. The shared trace_id also lets them jump directly from this log line to the distributed trace for the same request.

Trade-offs and pitfalls

• Logging everything at INFO "just in case" defeats the purpose of levels: if INFO volume is as high as DEBUG would be, the on-call engineer is back to searching through noise. Levels only help if they're used with discipline.
• Structured logging without a shared, enforced schema across services becomes almost as unqueryable as unstructured text, just in JSON clothing; a status_code field that's a string in one service and an integer in another breaks cross-service queries.
• Sensitive fields (user identifiers, tokens, payment details) need to be redacted or hashed at the point of logging, not cleaned up later; once something sensitive is in a log aggregator, deleting it retroactively is unreliable.
• DEBUG-level logging left on in production is a common, avoidable cost problem: log ingestion is usually billed by volume, and DEBUG noise can dominate that cost without adding proportional value.

Direct answer

Observability rests on three complementary signal types: metrics, logs, and traces. Metrics tell you something is wrong and roughly how bad; logs tell you what specifically happened in a given event; traces tell you where in a multi-service request the time or failure occurred. None of the three alone gives a complete picture: a strong incident response usually starts with one pillar to detect and scope the problem, then pivots to another to find root cause.

The three pillars

Instrumentation example, one flow

For a checkout endpoint, an on-call engineer might instrument it like this: a checkout_requests_total counter metric with labels {status, payment_provider}, plus a checkout_latency_seconds histogram with the same labels for percentiles; a structured log line at the point of failure with fields {request_id, trace_id, error_type, payment_provider}; and a trace with spans named checkout.validate, checkout.charge, checkout.persist, each carrying the same trace_id that appears in the log line. The shared trace_id and request_id are what let you jump from "the metric moved" to "here is the specific failing request" to "here is the exact log line explaining why."

Trade-offs and pitfalls

• Treating one pillar as sufficient is the most common mistake: teams that only have logs end up searching blind during an incident because they have no aggregated signal telling them where to look first; teams that only have metrics can detect a problem but can't explain it.
• High-cardinality labels (like an unbounded user_id on a metric) turn cheap metrics into an expensive, slow-to-query mess; that data belongs in logs or traces instead.
• Trace sampling is a real trade-off: full sampling captures every rare failure but is expensive at scale; low sampling rates are cheap but can miss the exact failing request you need. Tail-based sampling (keep traces for slow or error requests) is a common middle ground.
• Retention windows differ by pillar in practice (metrics are cheap to keep for months, verbose logs and full traces are usually much more expensive to retain), which shapes how far back a postmortem can actually look.