Monitoring, Logging, and Observability Questions
Understanding running systems through their signals. Covers metrics, logs, and traces, instrumentation, dashboards, alerting design, and log analysis and correlation for debugging production. Emphasizes designing observability so problems are detectable and diagnosable before users are affected.
What's the difference between structured and unstructured logging? Also, walk through when you'd log at DEBUG versus INFO versus WARN versus ERROR, and how that choice affects an on-call engineer during an incident.
Sample Answer
Direct answer
Unstructured logging is free-form text intended for a human to read line by line; structured logging emits each entry as a consistent, machine-parsable record (typically JSON) with named fields, so a log-query tool can filter, aggregate, and correlate across millions of lines the way you'd query a database. Log levels (DEBUG, INFO, WARN, ERROR) are an orthogonal concept from structure: they control which entries you even keep or surface, so during an incident an on-call engineer isn't wading through routine noise to find the handful of lines that actually explain what broke.
Structured versus unstructured
| Unstructured | Structured | |
|---|---|---|
| Format | Free text, e.g. 2026-07-17 ERROR payment failed for user 123: timeout | Consistent fields, e.g. {"level":"ERROR","service":"payments","user_id":"123","error_type":"timeout"} |
| Querying | Regex/grep, brittle if the message wording ever changes | Filter and aggregate directly on fields (status_code >= 500), stable across wording changes |
| Correlation | Hard to reliably join with traces or other services | A shared trace_id/request_id field lets you pivot straight from a metric spike to the exact request's log lines |
| Best for | Quick local debugging, human-only reading | Production systems at any real scale, automated alerting on log content |
When to log at each level, and why it matters during an incident
| Level | Use for | Effect on an on-call engineer during an incident |
|---|---|---|
| DEBUG | Fine-grained internal state, useful only when actively investigating | Should normally be off in production (or sampled), since it's high-volume noise; if it's flooding the log stream, it drowns out the ERROR line the engineer actually needs |
| INFO | Normal, expected events: a request completed, a job started | Confirms the system is doing what it should; useful for confirming a fix worked, not for finding the problem itself |
| WARN | Something unexpected happened but the system recovered or degraded gracefully (a retry succeeded, a fallback kicked in) | Early signal: a spike in WARN volume right before an incident often shows the system trying to compensate before it actually failed |
| ERROR | An operation failed and did not recover on its own | This is what the on-call engineer searches for first; ERROR entries should carry enough context (request_id, error_type, relevant IDs) to explain what failed without needing to reproduce it |
Worked example: a structured log line for a failed request
{
"timestamp": "2026-07-17T15:04:05.123Z",
"level": "ERROR",
"service": "payments-api",
"request_id": "req-8f21",
"trace_id": "trace-a93c",
"status_code": 502,
"duration_ms": 247,
"error_type": "UpstreamTimeout",
"message": "upstream charge provider timed out"
}
During an incident, an on-call engineer can now do something like find all ERROR entries with error_type: UpstreamTimeout in the last 15 minutes, grouped by service, instead of searching for the word "timeout" across every service's free-text logs and hoping the wording matches. The shared trace_id also lets them jump directly from this log line to the distributed trace for the same request.
Trade-offs and pitfalls
- Logging everything at INFO "just in case" defeats the purpose of levels: if INFO volume is as high as DEBUG would be, the on-call engineer is back to searching through noise. Levels only help if they're used with discipline.
- Structured logging without a shared, enforced schema across services becomes almost as unqueryable as unstructured text, just in JSON clothing; a
status_codefield that's a string in one service and an integer in another breaks cross-service queries. - Sensitive fields (user identifiers, tokens, payment details) need to be redacted or hashed at the point of logging, not cleaned up later; once something sensitive is in a log aggregator, deleting it retroactively is unreliable.
- DEBUG-level logging left on in production is a common, avoidable cost problem: log ingestion is usually billed by volume, and DEBUG noise can dominate that cost without adding proportional value.
What is a runbook, and what does a good one actually need to contain to be useful when someone's paged at 3am? Sketch what you'd want in one for a failed database migration.
Sample Answer
Direct answer
A runbook is a step-by-step operational document for handling a specific, known failure mode: what to check first, what commands to run, when to roll back versus push forward, and who to call if it gets worse. A good one is written so that someone half-awake at 3am who has never touched this exact system before can follow it without reconstructing context from scratch. The test of a good runbook is whether a different engineer than its author can execute it correctly under pressure.
What a runbook needs to contain
- Scope and severity: what specific failure this covers, and the severity/priority it corresponds to.
- Preconditions: what access, credentials, or tools you need before starting.
- Immediate triage steps: the first few things to check, in order, to confirm the diagnosis.
- Remediation steps: copy-pasteable commands with the expected output at each step, not prose descriptions of what to do.
- Verification: how to confirm the fix actually worked, not just that the command ran.
- Rollback path: a safe way back if remediation makes things worse, including its own preconditions (e.g. "requires a backup from the last 24 hours").
- Escalation: who to page next and when, by name/role/contact, not just "escalate if needed."
- Post-incident: where to file the incident ticket, and a note to update the runbook itself if a step was wrong or missing.
Worked example: failed database migration runbook
- Scope: prod schema migration failed mid-deploy. Severity: P1 if writes are blocked, P2 if only the migration job failed cleanly.
- Triage (first 5 minutes): tail the migration tool's log for the exact error; check
SELECT count(1) FROM pg_stat_activity WHERE state <> 'idle';to see if the migration left long-running locks; check the app's error dashboard for whether requests are actually failing yet. - Remediation, case A (migration failed cleanly, nothing partially applied): re-run the migration tool in dry-run mode first, then apply.
- Remediation, case B (partially applied, schema now inconsistent): put the app in read-only/maintenance mode to stop new writes, then decide between manually completing the migration or rolling back.
- Rollback (case B, if completing isn't safe): confirm the most recent backup timestamp, pause replication, restore with
pg_restore --clean --no-owner <backup>, then run a smoke test against a few critical read/write paths before removing maintenance mode. - Verification: re-run the app's smoke tests, spot-check row counts on affected tables against the pre-migration baseline.
- Escalation: if triage doesn't identify the cause within 10 minutes, or rollback is being considered, page the on-call DBA by name/rotation, not just "the DBA team."
- Post-incident: file the incident ticket with the timeline and root cause, and if any step above was missing or wrong, fix the runbook in the same pass as the incident writeup, not "later."
Trade-offs and pitfalls
- A runbook that's too generic ("check the logs, investigate, fix it") isn't actually a runbook, it's a checklist item pretending to be one; specificity is what makes it useful at 3am when judgment is impaired by fatigue.
- Runbooks rot: a step that references a tool or dashboard that got replaced months ago is worse than no runbook, because it wastes time and erodes trust in the whole document. Tie runbook review to any change in the system it covers, not a fixed calendar cadence alone.
- Over-indexing on "never improvise" can be as dangerous as no runbook at all: a good runbook documents when to deviate (e.g. "if replication lag exceeds a set threshold, stop and escalate instead of continuing") rather than pretending every failure mode was anticipated.
- Untested runbooks are a liability. The rollback path above should actually be exercised in a drill, not just written down, since commands like
pg_restore --cleanbehave differently depending on schema ownership and extensions that may not match what was true when the runbook was written.
What are the three pillars of observability? For each one, explain what kind of question it's best at answering, one blind spot it has on its own, and a concrete example of a production issue it would help you catch.
Sample Answer
Direct answer
Observability rests on three complementary signal types: metrics, logs, and traces. Metrics tell you something is wrong and roughly how bad; logs tell you what specifically happened in a given event; traces tell you where in a multi-service request the time or failure occurred. None of the three alone gives a complete picture: a strong incident response usually starts with one pillar to detect and scope the problem, then pivots to another to find root cause.
The three pillars
| Pillar | Best at answering | Blind spot alone | Production issue it would catch |
|---|---|---|---|
| Metrics | "Is something wrong right now, and how widespread?" (aggregated time series: rates, latencies, saturation) | No per-request context, can't tell you which specific request or user was affected | A slow memory leak: heap usage climbing steadily over days trips a capacity alert before an out-of-memory crash |
| Logs | "What exactly happened for this one request or event?" (discrete, timestamped records) | Expensive to query in aggregate at scale; no built-in sense of "normal," so you need to already suspect something to search for it | A payment failing with a specific exception, e.g. a null card-token field surfaced in the stack trace, that a dashboard would only show as "errors up" |
| Traces | "Where in the call chain did the time or failure happen?" (request-scoped, spans across services) | Usually sampled, so rare failures can be missed entirely; requires instrumentation and consistent context propagation to be useful | A checkout endpoint's p99 latency doubles; a trace shows 900ms of the 1000ms total sitting in a single downstream inventory-service span, isolating exactly which hop got slow |
Instrumentation example, one flow
For a checkout endpoint, an on-call engineer might instrument it like this: a checkout_requests_total counter metric with labels {status, payment_provider}, plus a checkout_latency_seconds histogram with the same labels for percentiles; a structured log line at the point of failure with fields {request_id, trace_id, error_type, payment_provider}; and a trace with spans named checkout.validate, checkout.charge, checkout.persist, each carrying the same trace_id that appears in the log line. The shared trace_id and request_id are what let you jump from "the metric moved" to "here is the specific failing request" to "here is the exact log line explaining why."
Trade-offs and pitfalls
- Treating one pillar as sufficient is the most common mistake: teams that only have logs end up searching blind during an incident because they have no aggregated signal telling them where to look first; teams that only have metrics can detect a problem but can't explain it.
- High-cardinality labels (like an unbounded user_id on a metric) turn cheap metrics into an expensive, slow-to-query mess; that data belongs in logs or traces instead.
- Trace sampling is a real trade-off: full sampling captures every rare failure but is expensive at scale; low sampling rates are cheap but can miss the exact failing request you need. Tail-based sampling (keep traces for slow or error requests) is a common middle ground.
- Retention windows differ by pillar in practice (metrics are cheap to keep for months, verbose logs and full traces are usually much more expensive to retain), which shapes how far back a postmortem can actually look.
That is every published Monitoring, Logging, and Observability question for Network Engineer so far. Browse the other topics in this category, or practice this one interactively.