Monitoring, Logging, and Observability Questions

Direct answer

Observability is the ability to ask new questions about a running system's internal state from its external outputs, metrics, logs, and traces, without shipping new code or adding new instrumentation for every question you didn't anticipate in advance. It's a broader goal than monitoring: monitoring is built around known failure modes you already predicted and set up dashboards and alerts for; observability is what lets you investigate the failure modes nobody predicted.

How metrics, logs, and traces each contribute

• Metrics: aggregated numeric signals over time (request rate, error rate, latency). Best for answering "is something wrong, and roughly how bad," cheaply and at a glance, which is why they're the backbone of dashboards and alerting.
• Logs: discrete, timestamped records of specific events. Best for answering "what exactly happened," with the detail (stack traces, payloads, IDs) that a metric can't carry.
• Traces: the path and timing of one request across every service it touched. Best for answering "where in this specific request did the time or failure occur," especially in a distributed system where the slow part could be any of several services.

When I'd reach for one over the others

I'd reach for metrics first, almost always, because they're the cheapest to check and tell me whether I even have a real problem worth digging into. From there, if the problem spans multiple services, I'd reach for traces to localize which hop is responsible; if I need to understand the specific reason a request failed, not just where it slowed down, I'd reach for logs, ideally pivoting there using the same trace or request ID that traces and metrics already pointed me to.

Worked example: an intermittent error spike

Metrics show an elevated error rate and p99 latency for service A over the last 20 minutes. Pulling a trace for a recent failing request shows most of the time sitting in a call to service B, with retries visible as repeated spans. Pulling the logs for that same request (using the shared trace ID) shows the actual failure: a JSON parsing error caused by a malformed field in service B's response, which is what's triggering the retries and driving up latency. The fix is a quick input-validation patch at the edge to reject the malformed field before it reaches service B, plus a cap on retries so a bad request doesn't compound into a latency spike.

Trade-offs and pitfalls

• Observability isn't a tool you buy, it's a property of the system, shaped by how well it's instrumented. Buying a vendor platform without consistent, correlated instrumentation (shared IDs across metrics, logs, and traces) gives you three separate data sources, not observability.
• Monitoring only covers what you thought to watch for in advance; a system can pass every dashboard and alert check while still failing in a way nobody predicted, which is exactly the gap observability is meant to close, by letting you investigate the unknown-unknowns after the fact instead of only the known-knowns.
• Over-instrumenting everything at maximum detail (full trace sampling, verbose logs everywhere) is expensive and can itself become a performance and cost problem; the goal is enough signal to investigate confidently, not maximum data volume.

Direct answer

Observability rests on three complementary signal types: metrics, logs, and traces. Metrics tell you something is wrong and roughly how bad; logs tell you what specifically happened in a given event; traces tell you where in a multi-service request the time or failure occurred. None of the three alone gives a complete picture: a strong incident response usually starts with one pillar to detect and scope the problem, then pivots to another to find root cause.

The three pillars

Instrumentation example, one flow

For a checkout endpoint, an on-call engineer might instrument it like this: a checkout_requests_total counter metric with labels {status, payment_provider}, plus a checkout_latency_seconds histogram with the same labels for percentiles; a structured log line at the point of failure with fields {request_id, trace_id, error_type, payment_provider}; and a trace with spans named checkout.validate, checkout.charge, checkout.persist, each carrying the same trace_id that appears in the log line. The shared trace_id and request_id are what let you jump from "the metric moved" to "here is the specific failing request" to "here is the exact log line explaining why."

Trade-offs and pitfalls

• Treating one pillar as sufficient is the most common mistake: teams that only have logs end up searching blind during an incident because they have no aggregated signal telling them where to look first; teams that only have metrics can detect a problem but can't explain it.
• High-cardinality labels (like an unbounded user_id on a metric) turn cheap metrics into an expensive, slow-to-query mess; that data belongs in logs or traces instead.
• Trace sampling is a real trade-off: full sampling captures every rare failure but is expensive at scale; low sampling rates are cheap but can miss the exact failing request you need. Tail-based sampling (keep traces for slow or error requests) is a common middle ground.
• Retention windows differ by pillar in practice (metrics are cheap to keep for months, verbose logs and full traces are usually much more expensive to retain), which shapes how far back a postmortem can actually look.

Direct answer

Alert fatigue is what happens when on-call engineers get so many low-value, noisy, or duplicate pages that they start treating all alerts as probably-not-real, including the ones that matter. It's a trust problem as much as a technical one: once someone has been paged repeatedly in a night for something that turned out to be nothing, the next page, which might be the real incident, gets a slower, more skeptical response.

How I'd prevent it on a team I'm leading

• Deduplication and grouping: alerts that share a root cause (same service, same error type) should collapse into a single incident with a count, not fire a separate page per occurrence. This is usually a config change in the alerting tool (fingerprinting by service and error signature) rather than a code change.
• Severity tuning tied to required response time: not every alert deserves a page. A three-tier split (page now, notify during business hours, dashboard-only) forces every new alert to justify why it needs to interrupt someone's sleep.
• Actionable-by-default policy: no new paging alert ships without a linked runbook and a clear "what to check first." An alert with no next step is a dashboard panel that accidentally has a pager attached.
• Automated remediation for known, safe, repeatable fixes: if the same alert reliably resolves by restarting a stuck worker or clearing a queue, and that action is safe and idempotent, automate it and only page if the automated fix fails.
• A regular noise review: periodically look at which alerts fired most often and whether they led to real action; alerts that never lead to action get tuned or removed, not left running indefinitely out of habit.

Worked example

Suppose a team's on-call rotation is getting paged for "queue depth over 100" on a background job processor, firing several times a week, always self-resolving within a few minutes without anyone doing anything. Applying the framework above: first, check whether these spikes line up with a predictable traffic pattern (a nightly batch job, say) and if so, either raise the threshold above that expected peak or add a time-of-day exception. Second, if the queue really can back up unpredictably but always self-resolves within a known window without intervention, the alert should require a longer sustain window (e.g. "queue depth over 100 for 15 minutes") so it only fires when it isn't going to resolve on its own. Third, if manual intervention when it does page is always the same action (scale up worker count), that's a strong automated-remediation candidate: auto-scale on the same threshold, and only page if depth is still elevated after the auto-scale has had time to take effect.

Trade-offs and pitfalls

• Automated remediation without an audit trail or human confirmation for higher-severity cases can turn a noisy-alert problem into a silent-failure problem: the system "fixes" itself repeatedly while masking a root cause that's getting worse.
• Tuning thresholds purely to reduce page volume, without checking against real past incidents, risks quietly increasing false negatives; the goal is signal-to-noise, not just fewer pages.
• Alert fatigue prevention is not a one-time project. It needs an ongoing review cadence, because new alerts get added faster than old noisy ones get cleaned up if nobody owns the process.

Direct answer

Unstructured logging is free-form text intended for a human to read line by line; structured logging emits each entry as a consistent, machine-parsable record (typically JSON) with named fields, so a log-query tool can filter, aggregate, and correlate across millions of lines the way you'd query a database. Log levels (DEBUG, INFO, WARN, ERROR) are an orthogonal concept from structure: they control which entries you even keep or surface, so during an incident an on-call engineer isn't wading through routine noise to find the handful of lines that actually explain what broke.

Structured versus unstructured

When to log at each level, and why it matters during an incident

Worked example: a structured log line for a failed request

{
  "timestamp": "2026-07-17T15:04:05.123Z",
  "level": "ERROR",
  "service": "payments-api",
  "request_id": "req-8f21",
  "trace_id": "trace-a93c",
  "status_code": 502,
  "duration_ms": 247,
  "error_type": "UpstreamTimeout",
  "message": "upstream charge provider timed out"
}

During an incident, an on-call engineer can now do something like find all ERROR entries with error_type: UpstreamTimeout in the last 15 minutes, grouped by service, instead of searching for the word "timeout" across every service's free-text logs and hoping the wording matches. The shared trace_id also lets them jump directly from this log line to the distributed trace for the same request.

Trade-offs and pitfalls

• Logging everything at INFO "just in case" defeats the purpose of levels: if INFO volume is as high as DEBUG would be, the on-call engineer is back to searching through noise. Levels only help if they're used with discipline.
• Structured logging without a shared, enforced schema across services becomes almost as unqueryable as unstructured text, just in JSON clothing; a status_code field that's a string in one service and an integer in another breaks cross-service queries.
• Sensitive fields (user identifiers, tokens, payment details) need to be redacted or hashed at the point of logging, not cleaned up later; once something sensitive is in a log aggregator, deleting it retroactively is unreliable.
• DEBUG-level logging left on in production is a common, avoidable cost problem: log ingestion is usually billed by volume, and DEBUG noise can dominate that cost without adding proportional value.

Direct answer

An SLI is the measured signal (e.g. the percentage of requests that were fast and successful), an SLO is the internal target for that signal over a time window (e.g. 99.9% of requests succeed under 300ms over a rolling 30 days), and an SLA is the external, often contractual promise built on top of an SLO, usually with margin, and real consequences (service credits, penalties) if it's breached. In short: the SLI measures, the SLO targets, the SLA promises with a business wrapper attached.

Defining each one concretely, for an e-commerce checkout service

Explained for a non-technical stakeholder: the SLO is the bar the engineering team holds itself to internally so problems get caught and fixed before they become customer-visible; the SLA is the, usually more forgiving, bar the business is willing to be held to externally, on paper, with money attached if it's missed. The gap between the two is deliberate headroom, not sloppiness.

Worked example: computing the error budget

The error budget is the amount of allowed failure baked into the SLO. It's what lets a team ship changes at all instead of freezing forever.

For the 99.9% SLO above:

Over the 30-day measurement window (30 days equals 2,592,000 seconds):

So this service is allowed about 43 minutes of budget-consuming failure (however that failure is defined: downtime, over-300ms responses, and so on) across a 30-day window before the SLO itself is breached. If checkout handles, say, 1,000,000 requests over that same window, the equivalent request-based budget is:

Both framings, time-based and request-based, describe the same budget; which one is more useful depends on whether the SLI is availability-style (was the service up) or ratio-style (what fraction of requests succeeded).

Trade-offs and pitfalls

• Confusing SLO and SLA in conversation causes real problems: teams sometimes design their alerting and release-gating around the SLA (the looser, contractual number) instead of the SLO (the tighter, internal number), which means by the time anyone notices, the team is already close to breaching the external promise with no margin left to react.
• An SLO with no error-budget policy attached is just a number on a dashboard; the value comes from what happens when the budget is nearly exhausted (freeze risky launches, redirect engineering time to reliability work), not from the target itself.
• Picking an SLI that doesn't reflect real user experience (e.g. "server process is running" instead of "requests succeed within an acceptable time") gives a green dashboard while users are still unhappy. The SLI has to be as close to what the user actually experiences as the team can measure.