Monitoring, Logging, and Observability Questions

Direct answer

Observability is the ability to ask new questions about a running system's internal state from its external outputs, metrics, logs, and traces, without shipping new code or adding new instrumentation for every question you didn't anticipate in advance. It's a broader goal than monitoring: monitoring is built around known failure modes you already predicted and set up dashboards and alerts for; observability is what lets you investigate the failure modes nobody predicted.

How metrics, logs, and traces each contribute

• Metrics: aggregated numeric signals over time (request rate, error rate, latency). Best for answering "is something wrong, and roughly how bad," cheaply and at a glance, which is why they're the backbone of dashboards and alerting.
• Logs: discrete, timestamped records of specific events. Best for answering "what exactly happened," with the detail (stack traces, payloads, IDs) that a metric can't carry.
• Traces: the path and timing of one request across every service it touched. Best for answering "where in this specific request did the time or failure occur," especially in a distributed system where the slow part could be any of several services.

When I'd reach for one over the others

I'd reach for metrics first, almost always, because they're the cheapest to check and tell me whether I even have a real problem worth digging into. From there, if the problem spans multiple services, I'd reach for traces to localize which hop is responsible; if I need to understand the specific reason a request failed, not just where it slowed down, I'd reach for logs, ideally pivoting there using the same trace or request ID that traces and metrics already pointed me to.

Worked example: an intermittent error spike

Metrics show an elevated error rate and p99 latency for service A over the last 20 minutes. Pulling a trace for a recent failing request shows most of the time sitting in a call to service B, with retries visible as repeated spans. Pulling the logs for that same request (using the shared trace ID) shows the actual failure: a JSON parsing error caused by a malformed field in service B's response, which is what's triggering the retries and driving up latency. The fix is a quick input-validation patch at the edge to reject the malformed field before it reaches service B, plus a cap on retries so a bad request doesn't compound into a latency spike.

Trade-offs and pitfalls

• Observability isn't a tool you buy, it's a property of the system, shaped by how well it's instrumented. Buying a vendor platform without consistent, correlated instrumentation (shared IDs across metrics, logs, and traces) gives you three separate data sources, not observability.
• Monitoring only covers what you thought to watch for in advance; a system can pass every dashboard and alert check while still failing in a way nobody predicted, which is exactly the gap observability is meant to close, by letting you investigate the unknown-unknowns after the fact instead of only the known-knowns.
• Over-instrumenting everything at maximum detail (full trace sampling, verbose logs everywhere) is expensive and can itself become a performance and cost problem; the goal is enough signal to investigate confidently, not maximum data volume.

Direct answer

A counter only goes up (or resets to zero on a process restart) and is for counting events, like total requests or errors. A gauge holds a point-in-time value that can go up or down, like current queue depth. A histogram and a summary both capture a distribution of observed values, like request latency, so you can compute percentiles, but they differ in where that computation happens: a histogram lets the backend compute percentiles at query time from raw bucket counts, while a summary computes them client-side and ships the already-calculated quantile.

The four types side by side

Aggregation semantics that matter for dashboards versus alerts

• For dashboards: histograms let one query produce fleet-wide p50/p95/p99 by summing buckets across every instance, which is what you want for an aggregate latency panel.
• For alerts: counters (via rate()) are what you alert on for error-rate thresholds, gauges are what you alert on for instantaneous saturation thresholds like queue depth above N, and histogram-derived percentiles are what you alert on for latency SLOs.
• Summaries are the odd one out for fleet-wide alerting, because averaging five instances' p99s is not the fleet's real p99. A single instance handling an unlucky slice of traffic gets diluted by the others and hides inside the average.

Worked example

For a fleet of $n$ instances each exposing a histogram with identical bucket boundaries, the fleet-wide count in bucket $le$ is additive:

and the fleet-wide quantile is computed by interpolating within the merged buckets $B_{le}$, not by averaging each instance's own quantile. This is exactly why histograms (raw counts, additive) are the right choice for fleet-wide latency, and why summaries (already-computed quantiles, not additive) are not: summation is associative, a pre-computed quantile is not.

Trade-offs and pitfalls

• Using a gauge for something that's really cumulative (like a running error count tracked as a gauge that resets on deploy) loses the ability to compute an accurate rate across restarts. Use a counter and let rate() handle resets.
• Choosing a summary because it's simpler and skipping the bucket-tuning work of a histogram is a common shortcut that quietly breaks fleet-wide percentile dashboards later, once the service scales past one instance.
• Histogram accuracy is bounded by bucket granularity: more buckets means better percentile accuracy but higher cardinality and storage cost per series.

Direct answer

Alert fatigue is what happens when on-call engineers get so many low-value, noisy, or duplicate pages that they start treating all alerts as probably-not-real, including the ones that matter. It's a trust problem as much as a technical one: once someone has been paged repeatedly in a night for something that turned out to be nothing, the next page, which might be the real incident, gets a slower, more skeptical response.

How I'd prevent it on a team I'm leading

• Deduplication and grouping: alerts that share a root cause (same service, same error type) should collapse into a single incident with a count, not fire a separate page per occurrence. This is usually a config change in the alerting tool (fingerprinting by service and error signature) rather than a code change.
• Severity tuning tied to required response time: not every alert deserves a page. A three-tier split (page now, notify during business hours, dashboard-only) forces every new alert to justify why it needs to interrupt someone's sleep.
• Actionable-by-default policy: no new paging alert ships without a linked runbook and a clear "what to check first." An alert with no next step is a dashboard panel that accidentally has a pager attached.
• Automated remediation for known, safe, repeatable fixes: if the same alert reliably resolves by restarting a stuck worker or clearing a queue, and that action is safe and idempotent, automate it and only page if the automated fix fails.
• A regular noise review: periodically look at which alerts fired most often and whether they led to real action; alerts that never lead to action get tuned or removed, not left running indefinitely out of habit.

Worked example

Suppose a team's on-call rotation is getting paged for "queue depth over 100" on a background job processor, firing several times a week, always self-resolving within a few minutes without anyone doing anything. Applying the framework above: first, check whether these spikes line up with a predictable traffic pattern (a nightly batch job, say) and if so, either raise the threshold above that expected peak or add a time-of-day exception. Second, if the queue really can back up unpredictably but always self-resolves within a known window without intervention, the alert should require a longer sustain window (e.g. "queue depth over 100 for 15 minutes") so it only fires when it isn't going to resolve on its own. Third, if manual intervention when it does page is always the same action (scale up worker count), that's a strong automated-remediation candidate: auto-scale on the same threshold, and only page if depth is still elevated after the auto-scale has had time to take effect.

Trade-offs and pitfalls

• Automated remediation without an audit trail or human confirmation for higher-severity cases can turn a noisy-alert problem into a silent-failure problem: the system "fixes" itself repeatedly while masking a root cause that's getting worse.
• Tuning thresholds purely to reduce page volume, without checking against real past incidents, risks quietly increasing false negatives; the goal is signal-to-noise, not just fewer pages.
• Alert fatigue prevention is not a one-time project. It needs an ongoing review cadence, because new alerts get added faster than old noisy ones get cleaned up if nobody owns the process.

Direct answer

Unstructured logging is free-form text intended for a human to read line by line; structured logging emits each entry as a consistent, machine-parsable record (typically JSON) with named fields, so a log-query tool can filter, aggregate, and correlate across millions of lines the way you'd query a database. Log levels (DEBUG, INFO, WARN, ERROR) are an orthogonal concept from structure: they control which entries you even keep or surface, so during an incident an on-call engineer isn't wading through routine noise to find the handful of lines that actually explain what broke.

Structured versus unstructured

When to log at each level, and why it matters during an incident

Worked example: a structured log line for a failed request

{
  "timestamp": "2026-07-17T15:04:05.123Z",
  "level": "ERROR",
  "service": "payments-api",
  "request_id": "req-8f21",
  "trace_id": "trace-a93c",
  "status_code": 502,
  "duration_ms": 247,
  "error_type": "UpstreamTimeout",
  "message": "upstream charge provider timed out"
}

During an incident, an on-call engineer can now do something like find all ERROR entries with error_type: UpstreamTimeout in the last 15 minutes, grouped by service, instead of searching for the word "timeout" across every service's free-text logs and hoping the wording matches. The shared trace_id also lets them jump directly from this log line to the distributed trace for the same request.

Trade-offs and pitfalls

• Logging everything at INFO "just in case" defeats the purpose of levels: if INFO volume is as high as DEBUG would be, the on-call engineer is back to searching through noise. Levels only help if they're used with discipline.
• Structured logging without a shared, enforced schema across services becomes almost as unqueryable as unstructured text, just in JSON clothing; a status_code field that's a string in one service and an integer in another breaks cross-service queries.
• Sensitive fields (user identifiers, tokens, payment details) need to be redacted or hashed at the point of logging, not cleaned up later; once something sensitive is in a log aggregator, deleting it retroactively is unreliable.
• DEBUG-level logging left on in production is a common, avoidable cost problem: log ingestion is usually billed by volume, and DEBUG noise can dominate that cost without adding proportional value.

Direct answer

An SLI is the measured signal (e.g. the percentage of requests that were fast and successful), an SLO is the internal target for that signal over a time window (e.g. 99.9% of requests succeed under 300ms over a rolling 30 days), and an SLA is the external, often contractual promise built on top of an SLO, usually with margin, and real consequences (service credits, penalties) if it's breached. In short: the SLI measures, the SLO targets, the SLA promises with a business wrapper attached.

Defining each one concretely, for an e-commerce checkout service

Explained for a non-technical stakeholder: the SLO is the bar the engineering team holds itself to internally so problems get caught and fixed before they become customer-visible; the SLA is the, usually more forgiving, bar the business is willing to be held to externally, on paper, with money attached if it's missed. The gap between the two is deliberate headroom, not sloppiness.

Worked example: computing the error budget

The error budget is the amount of allowed failure baked into the SLO. It's what lets a team ship changes at all instead of freezing forever.

For the 99.9% SLO above:

Over the 30-day measurement window (30 days equals 2,592,000 seconds):

So this service is allowed about 43 minutes of budget-consuming failure (however that failure is defined: downtime, over-300ms responses, and so on) across a 30-day window before the SLO itself is breached. If checkout handles, say, 1,000,000 requests over that same window, the equivalent request-based budget is:

Both framings, time-based and request-based, describe the same budget; which one is more useful depends on whether the SLI is availability-style (was the service up) or ratio-style (what fraction of requests succeeded).

Trade-offs and pitfalls

• Confusing SLO and SLA in conversation causes real problems: teams sometimes design their alerting and release-gating around the SLA (the looser, contractual number) instead of the SLO (the tighter, internal number), which means by the time anyone notices, the team is already close to breaching the external promise with no margin left to react.
• An SLO with no error-budget policy attached is just a number on a dashboard; the value comes from what happens when the budget is nearly exhausted (freeze risky launches, redirect engineering time to reliability work), not from the target itself.
• Picking an SLI that doesn't reflect real user experience (e.g. "server process is running" instead of "requests succeed within an acceptable time") gives a green dashboard while users are still unhappy. The SLI has to be as close to what the user actually experiences as the team can measure.