Monitoring, Logging, and Observability Questions

Direct answer

A dashboard's real job is to answer a specific question fast during a specific workflow, most often "is something wrong, and where," not to display every metric you happen to be collecting. A dashboard that looks impressive but doesn't map to a decision an on-call engineer needs to make in the first minute of an incident is decoration, not tooling.

What separates the two kinds

Genuinely useful:

• Answers a specific triage question at a glance: is it up, is it degraded, where's the blast radius.
• Uses the right visual for the data: percentiles rather than averages for latency, a rate rather than a raw ever-growing counter.
• Leads to action: every panel maps to either a decision or the next diagnostic step, ideally with a link to a runbook or a deeper drill-down.

Looks impressive, isn't useful:

• Dozens of panels covering every metric the team happens to instrument, on the theory that more data means more insight.
• Vanity metrics: an uptime percentage displayed to several decimal places with no context for what "down" actually meant.
• Raw averages that hide tail latency, or so many stacked graphs the eye can't find the anomaly.

How to avoid building the second kind

• Design backward from the workflow: sit with an on-call engineer and ask what question they're actually trying to answer at 3am, then build the smallest dashboard that answers it.
• Cap panel count per dashboard. A useful rule of thumb: if it doesn't fit on one screen without scrolling, it's two dashboards, not one.
• Review usage. If a panel hasn't been looked at during an actual incident in months, it's a candidate for removal, not automatic proof it's useless, but a prompt to ask why.

Worked example

For an on-call engineer covering a checkout service, a genuinely useful dashboard is around five panels: error rate with an SLO line, p50/p95/p99 latency, request rate, dependency health (payment processor, database), and a deploy and incident annotation timeline. Each one answers "is it broken, how broken, and did something just change," which are exactly the three questions someone opens the dashboard to answer during an incident. A 40-panel version of the same dashboard usually isn't more informative, it's the same five questions buried under panels nobody consults under time pressure.

Trade-offs and pitfalls

• Over-indexing on "comprehensive" trades signal for noise. Comprehensiveness belongs in a drill-down view, not the front page.
• Building for an audience that isn't the primary user, like a leadership-style summary dashboard, dilutes the on-call dashboard's focus. Keep them as separate dashboards with separate owners.
• Common wrong turn: adding a panel "just in case" during a specific investigation and never removing it afterward. That's how dashboards accumulate to 40 panels one incident at a time.

Direct answer

A counter only goes up (or resets to zero on a process restart) and is for counting events, like total requests or errors. A gauge holds a point-in-time value that can go up or down, like current queue depth. A histogram and a summary both capture a distribution of observed values, like request latency, so you can compute percentiles, but they differ in where that computation happens: a histogram lets the backend compute percentiles at query time from raw bucket counts, while a summary computes them client-side and ships the already-calculated quantile.

The four types side by side

Aggregation semantics that matter for dashboards versus alerts

• For dashboards: histograms let one query produce fleet-wide p50/p95/p99 by summing buckets across every instance, which is what you want for an aggregate latency panel.
• For alerts: counters (via rate()) are what you alert on for error-rate thresholds, gauges are what you alert on for instantaneous saturation thresholds like queue depth above N, and histogram-derived percentiles are what you alert on for latency SLOs.
• Summaries are the odd one out for fleet-wide alerting, because averaging five instances' p99s is not the fleet's real p99. A single instance handling an unlucky slice of traffic gets diluted by the others and hides inside the average.

Worked example

For a fleet of $n$ instances each exposing a histogram with identical bucket boundaries, the fleet-wide count in bucket $le$ is additive:

and the fleet-wide quantile is computed by interpolating within the merged buckets $B_{le}$, not by averaging each instance's own quantile. This is exactly why histograms (raw counts, additive) are the right choice for fleet-wide latency, and why summaries (already-computed quantiles, not additive) are not: summation is associative, a pre-computed quantile is not.

Trade-offs and pitfalls

• Using a gauge for something that's really cumulative (like a running error count tracked as a gauge that resets on deploy) loses the ability to compute an accurate rate across restarts. Use a counter and let rate() handle resets.
• Choosing a summary because it's simpler and skipping the bucket-tuning work of a histogram is a common shortcut that quietly breaks fleet-wide percentile dashboards later, once the service scales past one instance.
• Histogram accuracy is bounded by bucket granularity: more buckets means better percentile accuracy but higher cardinality and storage cost per series.

Direct answer: Stop comparing the metric to a single number and start comparing it to what's expected for that specific time and segment: build a baseline per relevant dimension (region, tenant, hour-of-day, day-of-week), alert on deviation from that baseline rather than an absolute value, and give yourself an explicit, auditable way to suppress alerts during known, planned deviations instead of quietly widening the threshold until the alert stops meaning anything.

Structured elaboration

Diagnosing which noise source you actually have

• Seasonality noise: the metric is fine, it just has a predictable daily or weekly shape (nightly batch jobs, weekday-vs-weekend traffic) that a flat threshold can't distinguish from a real problem.
• Segment-baseline noise: different regions or tenants have genuinely different normal traffic levels, so a single global threshold is simultaneously too loose for a high-traffic segment and too tight for a low-traffic one.
• Planned-change noise: a maintenance window, migration, or intentional traffic shift temporarily makes the metric look abnormal for a known, bounded reason.

Fixing seasonality and segment noise

• Compute a rolling baseline per segment: same hour-of-day, same day-of-week, scoped to the specific region or tenant, not a single fleet-wide number.
• Alert on relative deviation from that baseline (for example, "more than 30% above the same-hour-last-week baseline for this tenant") rather than an absolute value, so a low-traffic tenant and a high-traffic tenant are each compared to their own normal, not to each other's.
• Require the deviation to persist across more than one evaluation window before paging, a single noisy data point shouldn't trigger, a sustained deviation should.

Fixing planned-change noise: explicit suppression, not silent threshold-widening

• A known nightly batch job that causes an expected spike should be handled by teaching the baseline about it (the baseline for that hour already expects the spike), not by loosening the alert threshold globally, which would also hide a real regression at that same hour.
• A planned maintenance or migration window is different: it's a one-time, bounded deviation. Use a temporary, explicitly time-boxed suppression window (start time, end time, scope, and a named owner and reason), not a permanent config change. Communicate the suppression window to on-call before it starts, and have it auto-expire and reactivate the normal alert rule at the end time rather than relying on someone remembering to turn it back on.
• Validate any new threshold or baseline against historical data (a week or two of past traffic for that segment) before relying on it in production, so the first real test of the new rule isn't the next live incident.

Worked example: switching from a global absolute count to a per-tenant relative rate

The original rule: alert if errors exceed 50 in a 5-minute window, fleet-wide.

Large tenant A, normal weekday evening peak: 80,000 requests in 5 minutes, 80 errors, an error rate of:

This is tenant A's completely normal baseline error rate, but the absolute count (80) exceeds the global threshold (50), so it pages every weekday evening for no real reason, this is the seasonality/segment noise the question describes.

Small tenant B, normal 5-minute window: 5,000 requests, 5 errors, also a 0.1% baseline error rate. During a real regression, tenant B's error rate rises to 0.6% (30 errors out of 5,000 requests), a genuine 6x jump. Under the old absolute-count rule, 30 errors never crosses the 50-error threshold, so this real regression never pages at all.

Redesigned rule: alert if a tenant's error rate exceeds 0.5% (5x its own same-hour-last-week baseline of roughly 0.1%), sustained across two consecutive 5-minute windows.

• Tenant A's evening peak stays at 0.1%, well under the 0.5% relative threshold, so it correctly stops firing.
• Tenant B's regression at 0.6% crosses the 0.5% relative threshold and correctly fires, catching a real problem the old absolute rule missed entirely.

The same redesign both eliminates a nightly false positive on the large tenant and catches a false negative on the small tenant, because the underlying comparison changed from "count against a fleet-wide number" to "rate against this segment's own normal," which is the actual fix, not a looser or tighter version of the same rule.

Trade-offs & pitfalls

• The most common failure mode here is quietly widening a global threshold until the noise stops, which also silently raises the bar for detecting a real regression at the exact times (batch windows, planned migrations) when something is most likely to actually go wrong. Segment-and-time-aware baselines fix the noise without paying that cost.
• A suppression window with no auto-expiry is a liability: it gets forgotten, and the service runs unmonitored for that window indefinitely until someone notices during an actual incident. Auto-expiry and an explicit owner/reason on every suppression window are what make this safe to use routinely rather than a one-off hack.
• Per-segment baselines multiply the number of things that can silently drift wrong (a stale baseline for a shrinking tenant, for instance), so they need the same periodic review a static threshold does, this isn't a set-and-forget improvement over the static case, it's a more accurate model that still needs maintenance.
• Rolling changes out to alert rules and notification routing during a migration window specifically also needs a rollback plan: if the migration itself causes an unexpected real incident, the suppression window shouldn't also suppress the alert that would have caught it, scope suppression as narrowly as possible (the specific expected symptom, not the whole service) so genuine problems during the window can still page.

Direct answer

Isolate at every layer a noisy tenant could actually hurt: rate-limit ingest per tenant at the edge so one tenant can't consume shared ingest capacity, give each tenant its own queue or partition so a backlog for one doesn't block others, and cap query concurrency and cost per tenant so one expensive search can't starve the shared search cluster. None of that requires fully dedicated infrastructure per tenant, which doesn't scale economically to thousands of tenants; it requires enforced quotas at each shared layer plus graceful degradation instead of hard rejection when a tenant goes over.

Structured elaboration

Isolation by layer

• Ingest / edge. Authenticate and rate-limit per tenant (a token bucket per API key) before anything reaches the shared pipeline, so a runaway producer is throttled at the door rather than after it's already loaded the system.
• Queueing. Route each tenant to its own logical queue or partition (a per-tenant Kafka partition, or a dedicated topic for the largest tenants); a backlog in one tenant's queue doesn't block consumers processing everyone else's.
• Compute / indexing. Resource quotas (CPU/memory limits per tenant's processing pool) prevent one tenant's parsing or indexing load from starving another's, with the largest tenants optionally getting dedicated node pools while the long tail shares a pool under quota.
• Storage / query. Per-tenant query concurrency limits and timeouts stop one expensive search from monopolizing the shared search cluster; tiered storage (hot for recent data, cold/object-store for older) keeps per-tenant cost proportional to what they actually query, not just what they ingest.

graph TD
  T1[Tenant A agent] --> G[Edge gateway rate limit]
  T2[Tenant B agent] --> G
  G --> Q1[Tenant A queue]
  G --> Q2[Tenant B queue]
  Q1 --> S1[Shared index cluster]
  Q2 --> S1
  S1 --> C1[Cold object storage]

What to actually build

A token-bucket rate limiter per tenant at the gateway is the first and cheapest control, since it stops the problem before it costs anything downstream. Behind that, per-tenant quotas (max ingest rate, max retention, max query concurrency) enforced with soft warnings before hard caps give tenants (and their account owners) visibility before they get throttled, rather than a silent, confusing drop. For the handful of genuinely large tenants, dedicating infrastructure (their own queue partitions, their own index shards) is worth the operational cost; for the long tail of small tenants, shared infrastructure with enforced quotas is the only economical option.

Worked example

Make the token-bucket rate limit concrete. Configure a bucket that refills at 1,000 events/sec sustained, with a burst capacity of 5,000 tokens, and a tenant that suddenly starts sending 3,000 events/sec:

The burst capacity absorbs the spike for 2.5 seconds before the bucket empties; after that, the tenant is throttled down to the sustained 1,000 events/sec refill rate, and every event above that is either queued (if there's per-tenant queue depth for it) or dropped with a counted, visible rejection, rather than being allowed to flow through and degrade shared ingest for everyone else. That 2.5-second number is a direct consequence of the bucket size and rates chosen; a larger burst capacity buys the tenant more slack before throttling kicks in, at the cost of a larger worst-case spike the shared system has to absorb.

Trade-offs and pitfalls

Fully dedicated infrastructure per tenant gives the strongest isolation but doesn't scale economically past a small number of the largest tenants, so a hybrid model (dedicated for the top N, shared-with-quotas for everyone else) is usually the right shape rather than an all-or-nothing choice. Hard rejection at the quota boundary is simple to implement but a poor experience for a legitimately bursty tenant having a real (not noisy) traffic spike, so prefer graceful degradation, like automatically raising the sampling rate down for the tenant temporarily, over a flat "reject everything past the limit" cutoff. Finally, quotas that are set once and never revisited become either too tight (throttling legitimate growth) or too loose (no longer actually protecting anyone) as tenants' real usage patterns change, so pair the quota system with periodic review and tenant-visible usage dashboards rather than treating the initial numbers as permanent.

Direct answer

I'd start from user experience outward: request success rate, latency (p95/p99), and traffic or throughput are the three signals that most directly reflect whether real users are having a good or bad time right now, so those are what I'd page on if I could only pick a handful. Everything else (queue depth, resource usage, thread-pool saturation, dependency health) is valuable for diagnosis and capacity planning, but it's typically a leading indicator or a root-cause detail rather than something a first responder needs to be paged on directly.

What I'd monitor, and how I'd prioritize

This is the familiar "four golden signals" framing (latency, traffic, errors, saturation), narrowed here to the three that most directly track user-visible harm, with saturation kept as a fast diagnostic step rather than a primary page.

Worked example: deciding what pages versus what doesn't, for a checkout service

If I could only page on a handful of signals for this service, I'd page on error rate crossing a sustained threshold (checkout failing for real users), p95/p99 latency crossing a threshold (checkout technically succeeding but painfully slow), and a sudden traffic drop (which often means something upstream, like a CDN or DNS issue, broke before requests even reach us, and a pure error-rate alert would miss it because there's no request to error on).

I'd deliberately not page on CPU or memory alone: high CPU that isn't yet causing elevated latency or errors is useful to know about, and worth watching as a leading indicator for a proactive look, but paging on it directly tends to produce alerts that fire before there's any actual user impact, which is exactly the kind of noisy, not-yet-actionable signal that trains people to ignore pages. The right response to rising CPU with no user-facing symptom yet is usually "look at this during business hours and consider scaling," not "wake someone up."

Trade-offs and pitfalls

• Paging on too many signals defeats the purpose of prioritizing at all; if everything can page, the team is back to full alert fatigue with extra steps. The "handful" constraint in the question is doing real work: it forces a genuine choice about what's essential.
• Traffic-drop alerts need a sensible baseline that accounts for real traffic patterns (day of week, time of day, known low-traffic periods); a naive fixed threshold will false-positive constantly on legitimate quiet periods.
• Saturation metrics are tempting to page on because they feel proactive, but a resource that's near its limit and staying stable isn't yet a user-facing problem. The discipline is to treat saturation as a diagnostic and capacity-planning signal, escalating to a page only once it actually starts producing errors or latency.