Monitoring, Logging, and Observability Questions

Direct answer

An average blends every request into one number, so a small slice of very slow requests can be swamped by a much larger slice of fast ones and barely move the mean. Percentiles like p95 (the value at or below which 95% of requests fell) or p99 directly answer "how bad is the worst-affected fraction of my users," which is usually the question that matters for user experience, because a business rarely cares that the typical request was fast if a meaningful slice of users had a broken experience.

Why this matters for user-facing latency

• p50 (median): roughly what a typical user experiences.
• p95 / p99: the experience of the slowest 5% or 1% of requests. For a service handling meaningful traffic, that slice is still a lot of real, unhappy users.
• An average can look healthy even when a meaningful slice of users are having a bad time, because a badly skewed distribution (mostly fast, with a tail of very slow) pulls the mean only a little while leaving the tail severe.

Worked example

Consider 100 requests to an endpoint in a one-minute window: 90 of them complete in 100ms (normal path), and 10 of them complete in 2000ms (a cache-miss path hitting a cold backend).

Average:

If the team's informal latency expectation is "under 300ms," a 290ms average looks completely fine on a dashboard.

Now the percentiles, using the nearest-rank method (rank equals the ceiling of the percentile times the sample count):

Sorting the 100 requests ascending, positions 1 through 90 are the 100ms requests and positions 91 through 100 are the 2000ms requests. Position 95 falls in the 91-100 range, and so does position 99:

So the average says "290ms, basically fine," while p95 says "1 in 20 requests take 2 seconds," a large gap from the typical request. That 10% slow slice, invisible in the average, is exactly the group worth investigating (a cache eviction policy, an unhealthy backend replica, or a cold-start path).

Trade-offs and pitfalls

• Percentiles computed per-host and then averaged across hosts are mathematically wrong; you can't average percentiles the way you can average counts. Percentiles need to be computed over the merged distribution, which is why streaming approximation structures like t-digest or HDR histograms exist, so services can merge summaries across hosts without keeping every raw sample.
• p99.9 and beyond gets noisy on lower-traffic endpoints: with only a few hundred requests in a window, the 99.9th percentile might be a single outlier request, not a stable signal. Match the percentile you alert on to the traffic volume you actually have.
• Percentiles alone don't tell you why the tail is slow, they tell you it exists; a metric spike still needs a trace or log dive to explain the cause, the way this example's cache-miss path was inferred but not directly observed.

Direct answer

For a growing company with a small operations team, default to a hosted platform unless one team already has strong operational muscle for a specific piece of the stack (most often metrics, via Prometheus). The variable that actually decides this is engineer-hours available for care and feeding, not sticker price: a self-hosted stack usually undercuts hosted pricing on paper, but that gap closes or reverses once you count the SRE time spent on cluster capacity, upgrades, and retention tuning.

Decision framework

When to tip toward each:

• Hosted: team is small, time-to-value matters more than unit cost, nobody owns the observability stack as their primary job, or you need APM/anomaly detection you don't want to build yourself.
• Self-hosted: you already run Kubernetes at scale with a platform team that can treat observability as just another workload, data residency or compliance forces on-prem storage, or ingestion volume is high enough that hosted per-GB pricing becomes the single largest line item in the infra budget.
• Hybrid: a common middle path is hosted APM and log search (where turnkey correlation and UI matter most) paired with a self-hosted Prometheus for metrics you already understand and want fast, cheap, high-resolution queries on. This works because metrics are the cheapest and most mechanical piece to self-host, while tracing and log search UX is where vendors differentiate most. Re-evaluate the split as the team and ingestion volume grow.

Worked example

The crossover point between the two options can be derived, not guessed, once you write cost as a function of ingestion volume.

Let $G$ be GB/day ingested. Model hosted cost as a per-GB ingestion rate, and self-hosted cost as a fixed infra floor plus a much cheaper per-GB storage rate plus ongoing engineer-hour labor:

Using illustrative, explicitly-assumed rates (not any specific vendor's current published price, since list prices change): hosted ingestion rate $r_{hosted} = \$0.10$/GB, self-hosted object-storage rate $r_{storage} = \$0.02$/GB, fixed self-hosted compute floor $C_{infra} = \$300$/month, and $H = 4$ hours/month of engineer time at a loaded rate $R_{labor} = \$150$/hour:

Under these assumptions, below roughly 375 GB/day of ingestion, hosted comes out cheaper on pure dollar terms even before counting the small-team operational risk. Above it, self-hosting's lower per-GB rate starts to outweigh the fixed compute and labor floor. The point isn't the exact number, it's that a candidate should reason in terms of where the cost curves cross, not assert a universal winner.

Trade-offs and pitfalls

• Sunk-cost fallacy: don't keep self-hosting because it was already built, once engineer time becomes scarce elsewhere the calculus can flip.
• Hidden costs of hosted: egress and API costs, per-seat pricing for dashboard users, and price increases once you're locked in and migration feels expensive.
• Hidden costs of self-hosted: the observability stack becomes a second production system that itself needs monitoring, on-call, and capacity planning. If it goes down during an incident, you're debugging blind.
• Migration cost is asymmetric: moving off a hosted platform later means rebuilding dashboards and alerts in a new query language; moving off self-hosted OSS is comparatively easier since OpenTelemetry-based data is portable to almost any backend.
• Common wrong turn: picking self-hosted purely because a spreadsheet says it's cheaper while ignoring engineer-hours, then rediscovering the true cost six months later at the first major version upgrade.

Split the problem into a short, fully-indexed hot tier for day-to-day operational search, and a much cheaper cold tier of compressed columnar files on object storage for the long tail, with a lightweight catalog (not a full-text index) that maps time ranges and a few coarse identifiers to the specific files a query needs. Forensic point-lookups ("find everything about this one request from eight months ago") and batch analytics ("scan a year of logs for a pattern") are different access patterns and should be served differently: the catalog gets you to the right files for a point-lookup, while a batch engine scanning the columnar files directly serves analytics, and neither needs the cold tier to be a fully-indexed search cluster.

Framework

Tiering. Hot tier (days, full search index) for active operations. Cold tier (the long retention window) as compressed, partitioned columnar files (Parquet/ORC), partitioned by date and service so both access patterns can prune to the relevant subset without scanning everything.

The catalog, not full-text indexing, is what makes cold-tier forensics tractable. Index a small, deliberately chosen set of identifiers, most usefully trace_id or request_id, to (file, partition) pointers, so a forensic point-lookup for one specific request goes: catalog lookup for the ID, straight to the handful of files that contain it, rather than a full scan of a year of data. Sizing this catalog correctly matters a lot, and naive "index every line" quickly stops paying for itself, shown below.

Batch analytics uses the same columnar files directly, via a scan engine (Spark/Trino/Athena-style) with partition pruning and predicate pushdown, since analytics workloads (aggregate over a time range, find a pattern across many requests) don't need row-level lookup, they need efficient columnar scanning, which Parquet/ORC already provide without any extra indexing.

Lifecycle automation, not manual cleanup, moves data through the tiers and eventually deletes it per retention policy, with an explicit legal-hold flag that can override the TTL for specific data under investigation or compliance hold.

Worked example

Assume 2 TB/day of raw logs, and (stated as a planning assumption to validate against real data, not a measured fact) roughly 8x compression converting to a columnar, dictionary-encoded, zstd-compressed format:

Over a 1-year retention window, compressed footprint versus keeping raw for comparison:

which checks out against the assumed 8x ratio, as it must (the two numbers are the same assumption expressed two ways).

Catalog sizing is where the real design decision lives. At an average log line size of 500 bytes, 2 TB/day is:

A naive "index every line" catalog, at a compact 40 bytes per index entry (an ID plus a file/partition pointer):

That's 160 GB/day just for the index, against 250 GB/day for the compressed log data itself, the same order of magnitude as the data it's supposed to be a lightweight pointer into. Indexing at the individual-line grain has quietly stopped being cheap.

The fix is to index at trace_id grain instead of line grain, since many log lines share one trace_id. Assuming an average of 20 lines per trace:

An 8 GB/day catalog is a 20x reduction from the naive per-line version, and a small fraction (about 3%) of the 250 GB/day of compressed log data it points into, which is the shape a catalog should actually have: cheap relative to the data, not comparable to it. This is the concrete reasoning that separates "we added an index" from "we added an index that's actually worth what it costs."

Trade-offs and pitfalls

• Indexing at too fine a grain (as the naive per-line calculation shows) can erase most of the cost savings tiering was supposed to deliver; indexing at too coarse a grain (say, only by day and service, nothing more specific) makes forensic point-lookups require scanning entire daily partitions instead of jumping straight to the relevant files. The trace_id-level index above is a deliberate middle ground, chosen because it matches how forensic investigations actually query (by request, not by arbitrary line), not because finer or coarser is wrong in the abstract.
• The 8x compression figure is a planning assumption and needs validating against a real pilot on your actual log shape: logs with a lot of free-text/stack-trace content compress differently than uniform structured fields, and the catalog-sizing math above scales directly off whatever the real compression ratio turns out to be.
• Legal holds and lifecycle automation can conflict directly: an automated TTL-based deletion job and a compliance requirement to preserve specific records indefinitely need an explicit override mechanism (a hold flag checked before any delete), not an assumption that "we'll remember to exclude that data" during a routine cleanup job.
• Batch analytics over the cold tier is inherently higher-latency than a hot-tier query; that has to be communicated as an explicit trade-off to anyone used to interactive search, since "why is this query taking minutes instead of being instant like the hot dashboard" is a predictable point of friction if it isn't set as an expectation up front.

Direct answer: Stop comparing the metric to a single number and start comparing it to what's expected for that specific time and segment: build a baseline per relevant dimension (region, tenant, hour-of-day, day-of-week), alert on deviation from that baseline rather than an absolute value, and give yourself an explicit, auditable way to suppress alerts during known, planned deviations instead of quietly widening the threshold until the alert stops meaning anything.

Structured elaboration

Diagnosing which noise source you actually have

• Seasonality noise: the metric is fine, it just has a predictable daily or weekly shape (nightly batch jobs, weekday-vs-weekend traffic) that a flat threshold can't distinguish from a real problem.
• Segment-baseline noise: different regions or tenants have genuinely different normal traffic levels, so a single global threshold is simultaneously too loose for a high-traffic segment and too tight for a low-traffic one.
• Planned-change noise: a maintenance window, migration, or intentional traffic shift temporarily makes the metric look abnormal for a known, bounded reason.

Fixing seasonality and segment noise

• Compute a rolling baseline per segment: same hour-of-day, same day-of-week, scoped to the specific region or tenant, not a single fleet-wide number.
• Alert on relative deviation from that baseline (for example, "more than 30% above the same-hour-last-week baseline for this tenant") rather than an absolute value, so a low-traffic tenant and a high-traffic tenant are each compared to their own normal, not to each other's.
• Require the deviation to persist across more than one evaluation window before paging, a single noisy data point shouldn't trigger, a sustained deviation should.

Fixing planned-change noise: explicit suppression, not silent threshold-widening

• A known nightly batch job that causes an expected spike should be handled by teaching the baseline about it (the baseline for that hour already expects the spike), not by loosening the alert threshold globally, which would also hide a real regression at that same hour.
• A planned maintenance or migration window is different: it's a one-time, bounded deviation. Use a temporary, explicitly time-boxed suppression window (start time, end time, scope, and a named owner and reason), not a permanent config change. Communicate the suppression window to on-call before it starts, and have it auto-expire and reactivate the normal alert rule at the end time rather than relying on someone remembering to turn it back on.
• Validate any new threshold or baseline against historical data (a week or two of past traffic for that segment) before relying on it in production, so the first real test of the new rule isn't the next live incident.

Worked example: switching from a global absolute count to a per-tenant relative rate

The original rule: alert if errors exceed 50 in a 5-minute window, fleet-wide.

Large tenant A, normal weekday evening peak: 80,000 requests in 5 minutes, 80 errors, an error rate of:

This is tenant A's completely normal baseline error rate, but the absolute count (80) exceeds the global threshold (50), so it pages every weekday evening for no real reason, this is the seasonality/segment noise the question describes.

Small tenant B, normal 5-minute window: 5,000 requests, 5 errors, also a 0.1% baseline error rate. During a real regression, tenant B's error rate rises to 0.6% (30 errors out of 5,000 requests), a genuine 6x jump. Under the old absolute-count rule, 30 errors never crosses the 50-error threshold, so this real regression never pages at all.

Redesigned rule: alert if a tenant's error rate exceeds 0.5% (5x its own same-hour-last-week baseline of roughly 0.1%), sustained across two consecutive 5-minute windows.

• Tenant A's evening peak stays at 0.1%, well under the 0.5% relative threshold, so it correctly stops firing.
• Tenant B's regression at 0.6% crosses the 0.5% relative threshold and correctly fires, catching a real problem the old absolute rule missed entirely.

The same redesign both eliminates a nightly false positive on the large tenant and catches a false negative on the small tenant, because the underlying comparison changed from "count against a fleet-wide number" to "rate against this segment's own normal," which is the actual fix, not a looser or tighter version of the same rule.

Trade-offs & pitfalls

• The most common failure mode here is quietly widening a global threshold until the noise stops, which also silently raises the bar for detecting a real regression at the exact times (batch windows, planned migrations) when something is most likely to actually go wrong. Segment-and-time-aware baselines fix the noise without paying that cost.
• A suppression window with no auto-expiry is a liability: it gets forgotten, and the service runs unmonitored for that window indefinitely until someone notices during an actual incident. Auto-expiry and an explicit owner/reason on every suppression window are what make this safe to use routinely rather than a one-off hack.
• Per-segment baselines multiply the number of things that can silently drift wrong (a stale baseline for a shrinking tenant, for instance), so they need the same periodic review a static threshold does, this isn't a set-and-forget improvement over the static case, it's a more accurate model that still needs maintenance.
• Rolling changes out to alert rules and notification routing during a migration window specifically also needs a rollback plan: if the migration itself causes an unexpected real incident, the suppression window shouldn't also suppress the alert that would have caught it, scope suppression as narrowly as possible (the specific expected symptom, not the whole service) so genuine problems during the window can still page.

Direct answer

Split this into two tracks. In the next few days, go after the two named drivers directly with reversible, low-risk levers: tag hygiene to cap cardinality (the number of distinct label/tag value combinations a metric or log field produces, since every unique combination becomes its own stored time series or index entry), and adaptive sampling that keeps close to full fidelity for errors and anomalies while cutting the sampling rate hard for routine traffic. Over the following weeks to months, replace flat sampling with a real tail-based pipeline and separate high-cardinality debug context from the aggregate metrics store, so cost scales with genuinely interesting traffic instead of total traffic.

Structured elaboration

Diagnose the drivers first

High-cardinality tags (raw user IDs, session IDs, request IDs used as labels) multiply the number of unique time series or indexed log fields, and cost in most telemetry backends scales with that cardinality, not just with event volume. Full trace sampling multiplies volume directly: capturing 100% of traces costs roughly proportional to however many times more spans that is than whatever sampling rate you had before.

Next few days: quick, reversible levers

• Tag hygiene / cardinality caps. Identify the highest-cardinality fields and stop using raw free-form values as indexed labels; hash or drop them, or move them into unindexed log body content instead.
• Adaptive sampling as a stopgap. Keep close to 100% sampling for traces tied to errors or already-firing alerts, and drop the sampling rate hard for routine, healthy traffic. This is the single biggest lever because it directly reverses the "full trace sampling" driver while explicitly protecting the traces you'd need during a P0.
• Shorten retention on the highest-cardinality dimensions rather than cutting them entirely, so recent incidents keep full fidelity and only old, rarely-queried data ages out sooner.

Bigger architectural changes (weeks to months)

• True tail-based sampling pipeline: buffer full trace data briefly, decide what to keep after seeing whether the trace was anomalous (error, high latency, matches an active alert), not before.
• Separate the debug stream from the metrics store: high-cardinality request-level context lives in a short-retention, low-cost store; aggregated metrics and dashboards run on a high-throughput store that never carries per-request cardinality.
• Schema and cardinality linting in CI so a new high-cardinality label can't ship without review, which is what let costs triple in the first place.
• Multi-region Prometheus federation is a concrete example of the kind of durable architectural change worth pursuing: each region runs its own low-cardinality local Prometheus for fast local queries and alerting, and a global layer only pulls pre-aggregated, low-cardinality rollups for cross-region dashboards, instead of every region shipping raw high-cardinality series to one central store.

graph LR
  L1[Region A local Prometheus] --> G[Global federated query layer]
  L2[Region B local Prometheus] --> G
  L3[Region C local Prometheus] --> G
  G --> D[Cross region dashboards]

• Multi-tenant isolation is the complication that makes this harder at scale: without per-team or per-service cardinality quotas, one noisy team can blow through the whole organization's telemetry budget again even after you've fixed the current spike, so the architectural fix needs enforcement, not just cleanup.

What not to sacrifice

Whatever you cut, keep near-full fidelity specifically for traces linked to errors, active alerts, or recent deploys. That is what "investigate a major incident quickly" actually depends on; the routine, healthy-path traffic is where nearly all of the safe savings live.

Worked example

Here is a fully worked numeric illustration with pinned assumptions (not a real account's actual numbers, since none were given, but a concrete, checkable model of how the two named drivers combine and how the near-term fix reduces them). Model per-second ingest cost as sampling rate times request rate times average indexed bytes per span, holding request volume constant at 50,000 requests/sec:

$C_0$ is the baseline before the growth: 50% trace sampling, 1.0 KB of indexed bytes per span. $C_1$ is today: sampling went to 100% (a 2x factor) and indexed bytes per span grew to 1.5 KB because of added high-cardinality tags (a 1.5x factor), and 2 times 1.5 is exactly 3, reproducing the "tripled" in the question from the two named drivers.

Now apply the near-term fix: keep 100% sampling only for the roughly 2% of traffic that's already flagged as an error or anomaly, drop routine traffic sampling to 20%, and revert the indexed-bytes-per-span back to 1.0 KB via tag hygiene:

That's an 85.6% cut from today's cost, using only the two quick levers, and it lands below the original pre-growth baseline ($10.8$ vs $25$ MB/s) while still capturing every error and anomalous trace at full fidelity. The exact percentages depend on the assumed error-traffic fraction and sampling rates, but the structure of the calculation, and the fact that error/anomaly traffic is what stays at full fidelity, is the part that generalizes.

Trade-offs and pitfalls

Aggressive sampling on "routine" traffic means you lose fidelity on the silent tail: patterns that are unusual but don't yet trip an error or alert (a slow memory creep, a rare-but-legitimate code path) become harder to investigate retroactively, because they were never flagged as interesting at capture time. Cardinality caps can silently drop a dimension someone actually needed for a future investigation if they're applied without review, so pair caps with a lightweight approval process rather than a blanket ban. Retention cuts trade long-tail trend analysis and slow-burn postmortems for savings, which is usually the right trade for cost but should be a stated decision, not an accident.