Direct answer
Observability rests on three complementary signal types: metrics, logs, and traces. Metrics tell you something is wrong and roughly how bad; logs tell you what specifically happened in a given event; traces tell you where in a multi-service request the time or failure occurred. None of the three alone gives a complete picture: a strong incident response usually starts with one pillar to detect and scope the problem, then pivots to another to find root cause.
The three pillars
| Pillar | Best at answering | Blind spot alone | Production issue it would catch |
|---|
| Metrics | "Is something wrong right now, and how widespread?" (aggregated time series: rates, latencies, saturation) | No per-request context, can't tell you which specific request or user was affected | A slow memory leak: heap usage climbing steadily over days trips a capacity alert before an out-of-memory crash |
| Logs | "What exactly happened for this one request or event?" (discrete, timestamped records) | Expensive to query in aggregate at scale; no built-in sense of "normal," so you need to already suspect something to search for it | A payment failing with a specific exception, e.g. a null card-token field surfaced in the stack trace, that a dashboard would only show as "errors up" |
| Traces | "Where in the call chain did the time or failure happen?" (request-scoped, spans across services) | Usually sampled, so rare failures can be missed entirely; requires instrumentation and consistent context propagation to be useful | A checkout endpoint's p99 latency doubles; a trace shows 900ms of the 1000ms total sitting in a single downstream inventory-service span, isolating exactly which hop got slow |
Instrumentation example, one flow
For a checkout endpoint, an on-call engineer might instrument it like this: a checkout_requests_total counter metric with labels {status, payment_provider}, plus a checkout_latency_seconds histogram with the same labels for percentiles; a structured log line at the point of failure with fields {request_id, trace_id, error_type, payment_provider}; and a trace with spans named checkout.validate, checkout.charge, checkout.persist, each carrying the same trace_id that appears in the log line. The shared trace_id and request_id are what let you jump from "the metric moved" to "here is the specific failing request" to "here is the exact log line explaining why."
Trade-offs and pitfalls
- Treating one pillar as sufficient is the most common mistake: teams that only have logs end up searching blind during an incident because they have no aggregated signal telling them where to look first; teams that only have metrics can detect a problem but can't explain it.
- High-cardinality labels (like an unbounded user_id on a metric) turn cheap metrics into an expensive, slow-to-query mess; that data belongs in logs or traces instead.
- Trace sampling is a real trade-off: full sampling captures every rare failure but is expensive at scale; low sampling rates are cheap but can miss the exact failing request you need. Tail-based sampling (keep traces for slow or error requests) is a common middle ground.
- Retention windows differ by pillar in practice (metrics are cheap to keep for months, verbose logs and full traces are usually much more expensive to retain), which shapes how far back a postmortem can actually look.
