Observability and Monitoring Architecture Questions

Direct answer

Don't compute percentiles from raw points at query time. Have every collector maintain a local mergeable sketch (t-digest or an HDR histogram) over each rollup window, ship the sketch instead of the raw values, and merge sketches hierarchically (minute into hour, hour into day) so a query over weeks or months merges a small number of pre-combined objects instead of scanning raw samples. The accuracy you give up is bounded and tunable: it comes from the sketch's compression parameter, not from randomly dropping data, so you can trade a known amount of tail-percentile error for a known reduction in merge work.

Structured elaboration

Sketch choice:

• t-digest: variable bucket width, denser (more accurate) near the tails where percentile queries usually care most, mergeable, size controlled by a compression parameter $\delta$ (roughly, number of centroids).
• HDR histogram: fixed relative-error buckets across a configured value range, deterministic worst-case error bound, very fast to merge (array addition), best when the metric's range is known and bounded (for example request latency in milliseconds).
• Use t-digest for open-ended or skewed distributions; use HDR when the value range is known and the deterministic bound matters more than tail density.

Hierarchical rollups: each collector emits a 1-minute sketch. A rollup job merges 60 one-minute sketches into an hourly sketch, and 24 hourly sketches into a daily sketch. Retention is tiered: keep 1-minute sketches for 7 days, hourly for 30 days, daily beyond that. This is what makes "merge partial sketches from many collectors" tractable at high fan-in: each collector contributes one small sketch per window rather than a stream of raw points, and the merge tree does the aggregation once instead of at every query.

Merging at query time: the query picks the coarsest rollup tier that covers the requested range, merges those objects (t-digest merge is just concatenating and re-clustering centroids; HDR merge is bucket-wise addition), and computes the percentile from the merged sketch.

Handling very high fan-in (many collectors, very high ingest volume): the same hierarchy scales horizontally: instead of one job merging all collectors' sketches directly, shard collectors into groups, merge within a group first, then merge group-level sketches at the next tier. This bounds the fan-in of any single merge step regardless of how many collectors exist upstream.

flowchart LR
    C1[Collector] -- 1-min sketch --> M1[Minute merge]
    C2[Collector] -- 1-min sketch --> M1
    M1 -- 60 merged --> H[Hourly sketch]
    H -- 24 merged --> D[Daily sketch]
    Q[Percentile query] -- picks coarsest tier covering range --> H
    Q --> D
    D -- retained 30d+ --> COLD[(Cold rollup store)]

Worked example

Merge count reduction. For a 30-day 99th-percentile query, merging at 1-hour granularity versus 1-minute granularity:

Querying against hourly rollups means merging 60x fewer objects than querying against minute rollups, which is the primary win of hierarchical sketching: it shrinks the object count the query has to combine, not just the byte count of any one object.

t-digest size at a given compression. A t-digest with compression $\delta$ holds roughly $\delta$ centroids, each storing a mean and a count (two 8-byte doubles = 16 bytes):

At $\delta = 100$: $\text{digestBytes} = 1{,}600$ bytes per digest.

Storage over a year, at scale. For $\text{services} = 5{,}000$ tracked series, keeping hourly digests for 365 days:

Compare to keeping raw 15s points for those same 5,000 series for a year, at 16 bytes/point:

Worth being honest about what this number says: at $\delta = 100$, sketch storage is only 2.4x smaller than keeping the equivalent raw points, because each digest is representing many raw points (5,760 raw points/day per series versus 24 digests/day per series, so each digest stands in for 240 points). The real win from sketching at this compression level is the 60x merge-count reduction above, not storage. If storage is the binding constraint, dropping $\delta$ to 50 halves digest size (800 bytes) and roughly doubles the storage reduction factor to about 4.8x, at the cost of coarser centroid resolution and larger tail error.

Trade-offs & pitfalls

Common wrong turns: merging sketches from different tiers as if they were interchangeable (a 1-minute sketch and an hourly sketch built with different compression settings can merge, but mixing merge granularities inconsistently across a query makes the resulting error bound harder to reason about, so keep the compression parameter fixed across a tier); treating a merged sketch's percentile as exact (it is an estimate with error concentrated where the sketch is coarsest, which for t-digest is the middle of the distribution, not the tails); and picking HDR for an unbounded-range metric (queue depth that can spike arbitrarily) where the fixed value range either clips outliers or forces an oversized histogram, when t-digest's variable bucketing is the better fit there.

Build access control as three layers that compose: identity and provisioning (who is this, what teams are they in), a policy engine that maps identity plus team to permitted actions on specific resources, and an immutable audit trail of every access decision. The 50-team scale is what makes a policy-engine approach (rather than hand-rolled per-resource checks) worth the setup cost.

Design

flowchart LR
  A[SSO / SAML IdP] --> B[SCIM Provisioning]
  B --> C[Team / Group Mapping]
  C --> D[Policy Engine: OPA]
  D --> E[Dashboard and Query ACL]
  D --> F[Platform Admin Scope]
  E --> G[Audit Log: append-only]
  F --> G

• Identity and provisioning: SSO via SAML/OIDC for authentication, SCIM (or LDAP sync) for provisioning, so team membership changes (new hire, team transfer, offboarding) propagate automatically instead of relying on manual grants.
• RBAC model: a small set of coarse roles (Viewer, Editor, TeamLead, PlatformAdmin) combined with resource-scoped permissions (dashboard:read, dashboard:write, query:run, query:save, data:ingest). Evaluate as (subject, action, resource, context) through a policy engine (e.g., OPA/Rego) rather than scattering if user.role == 'admin' checks through application code, since 50 teams' worth of dashboard-sharing and cross-team-visibility rules will otherwise become unmaintainable ad hoc logic.
• Namespace/tenant isolation: each team's dashboards, saved queries, and (if applicable) raw telemetry are tagged with a team/namespace ID; the policy engine denies cross-team access by default, and sharing is an explicit grant, not an implicit consequence of both teams querying the same underlying data.
• Audit trail: append-only log of every access decision (actor, action, resource, timestamp, allow/deny), shipped to a separate system (SIEM or dedicated log store) so a compromised or misconfigured application instance can't retroactively edit its own audit history.

What's different about the platform-operator admin role

PlatformAdmin needs privileges no team role should have (upgrading the platform, changing global retention policy, accessing any team's data for support purposes) but that scope is exactly what makes it the highest-risk role. Treat it differently from every other role: require a separate approval workflow for admin grants, use time-boxed or break-glass access (temporary elevation with mandatory justification and MFA, auto-expiring) rather than standing admin permissions, and audit admin actions with the same rigor as the audit trail above, ideally with additional real-time alerting since a compromised admin credential is a platform-wide risk, not a single-team risk.

SaaS vs. on-prem

A quick sizing check on audit volume

A frequent unstated assumption is that audit logging is "too expensive to keep forever." For 50 teams with roughly 30 engineers each performing about 200 dashboard/query actions/day at 300 bytes/audit record:

teams, engineers_per_team, actions_per_day, record_bytes = 50, 30, 200, 300
total_engineers = teams * engineers_per_team                      # 1,500
events_per_day = total_engineers * actions_per_day                # 300,000
bytes_per_day = events_per_day * record_bytes                     # 90 MB/day
total_bytes_1yr = bytes_per_day * 365                              # ~32.85 GB/year

At roughly 32.85 GB for a full year of audit history across the whole platform, audit-log retention is cheap relative to telemetry storage; there's rarely a cost reason to truncate it, which supports keeping compliance-grade audit history far longer than raw telemetry.

Trade-offs and pitfalls

• Embedding authorization checks directly in each service (instead of a shared policy engine) is the design that looks fine at 5 teams and becomes unmaintainable at 50, because every new sharing rule requires a code change in every service that touches dashboards or queries.
• Standing admin access (a permanent PlatformAdmin role assignment with no expiry) is the most common real-world gap between a documented RBAC design and actual practice; break-glass, time-boxed elevation closes that gap but requires operational discipline to actually use.
• On-prem deployments that assume the SaaS identity/audit architecture transfers unchanged will hit friction the first time a customer's SSO provider or compliance requirement doesn't match the platform operator's assumptions; the abstraction boundary (pluggable IdP, pluggable audit sink) needs to be designed in from the start, not retrofitted.
• Least-privilege between on-call engineers and management is a common refinement: an on-call engineer needs broad read access across teams during an incident but not write/admin access, while a manager might need visibility without either; collapsing both into a single "Editor" role loses that distinction.

Set the policy by working backward from what each query pattern actually needs, then verify the storage savings with the arithmetic rather than picking round numbers and hoping. A reasonable concrete policy: raw metrics at native resolution for 14 days, 5-minute rollups for 1 year, hourly rollups for years 2 through 5; full traces for 30 days, then 1% sampled for the following 11 months.

Rationale by window

flowchart LR
  A[Raw ingest: native res] -->|14 days| B[Raw tier: hot]
  B -->|downsample| C[5-min tier: 1 year]
  C -->|downsample| D[Hourly tier: years 2-5]
  D -->|expire| E[Deleted]
  F[Trace ingest] -->|30 days full| G[Full trace tier]
  G -->|sample 1%| H[Sampled trace tier: 11 months]

• 14 days raw: covers essentially all incident debugging, since almost every retro or root-cause investigation happens within two weeks of the event, and alerting needs full resolution on recent data to avoid missing short spikes.
• 1 year at 5-minute rollups: supports capacity planning and seasonal comparisons (week-over-week, month-over-month) without needing per-second precision; 5 minutes is short enough to still show diurnal patterns clearly.
• Years 2-5 at hourly rollups: supports long-term trend and year-over-year growth analysis; anything finer than hourly at this age is rarely queried and expensive to keep.
• 30 days full traces: matches the raw-metrics window for the same reason, full-fidelity root cause work happens fast, and traces are the most expensive telemetry type per unit.
• 1% sampled for 11 more months: preserves enough statistical signal for "did this class of error exist a few months ago" investigations without paying for full trace volume; always retain 100% of traces tied to errors or SLO breaches regardless of the sampling rate (a fixed-percentage sample can otherwise miss the rare traces investigators actually want).

Verifying the storage savings

For 1,000,000 active series, using the same 2-bytes/compressed-raw-sample and 8-bytes/downsampled-point (4 aggregates: min, max, sum, count, at roughly 2 bytes each) assumptions used elsewhere in TSDB capacity planning:

series = 1_000_000
compressed_bytes_per_raw_sample = 2
agg_bytes_per_downsampled_point = 8

def samples(days, interval_s):
    return series * (days * 86400 / interval_s)

raw_bytes = samples(14, 15) * compressed_bytes_per_raw_sample
ds_bytes = samples(365, 300) * agg_bytes_per_downsampled_point
hourly_bytes = samples(1460, 3600) * agg_bytes_per_downsampled_point
total_tiered_bytes = raw_bytes + ds_bytes + hourly_bytes

allraw_bytes = samples(14 + 365 + 1460, 15) * compressed_bytes_per_raw_sample

Result: raw tier = 161.28 GB, 5-min tier = 840.96 GB, hourly tier = 280.32 GB, total tiered storage over the full 5-year window ≈ 1.283 TB, versus an all-raw-forever equivalent of ≈ 21.19 TB for the same window, a 16.5x reduction. The 5-minute tier dominates total storage (840 GB of the 1.28 TB) precisely because it covers the most time (1 year) at the finest surviving resolution; that's useful to know when deciding whether to push the raw window shorter or the 5-minute window's resolution coarser if the budget gets tighter.

For traces, at 10,000 traces/sec with an assumed 4 KB compressed size per trace: full 30-day retention stores about 103.68 TB, while the following 11 months at 1% sampling adds roughly 11.40 TB, so the sampled tail costs about 11% as much as the initial 30-day full window despite covering over 10x the time span.

What you can and can't still answer after each window closes

Trade-offs and pitfalls

• Percentile aggregates (p95, p99) do not survive naive downsampling: averaging five 1-minute p99 values is not the same number as the true p99 across that 5-minute window. If percentile fidelity matters at the rollup tier, you need to store enough of a histogram or sketch (not just min/max/sum/count) to recompute percentiles, which raises the per-point byte cost above the 8-byte assumption used here.
• A fixed sampling percentage for traces (1% flat) will statistically under-represent rare-but-important request types unless it's stratified or combined with the always-keep-errors/SLO-breach rule; a pure random sample optimizes for "typical" traffic, which is exactly what you don't need for debugging.
• Retention policy that isn't enforced automatically (a manual cleanup job, or "we'll get to it") tends to silently become the accidental real retention policy; tie expiry to the storage engine's native TTL/compaction mechanism rather than a side script.
• Communicating the policy to teams matters as much as setting it: if engineers don't know that a 3-week-old spike is only visible at 5-minute resolution, they'll draw wrong conclusions from a smoothed-out graph without realizing detail was lost.

Direct Answer

Decouple the pipeline from the sink with a durable buffer so a slow or unavailable time-series database does not propagate latency back to producers, wrap writes to the sink in a circuit breaker so a struggling database is not also hammered by retries, and treat the pipeline's own saturation state (queue depth, drop rate, breaker state) as a metric other teams can see, not an internal detail that only shows up as "my dashboard is missing data" after the fact.

Structured Elaboration

Resilience pipeline

flowchart LR
  PROD["Producers"] --> RL["Rate Limiter"]
  RL --> BUF[("Durable Buffer")]
  BUF --> CB{"Circuit Breaker"}
  CB -->|"closed"| TSDB[("TSDB Writer")]
  CB -->|"open"| RETRY["Backoff + Retry"]
  RETRY -.-> CB
  BUF --> HEALTH["Queue-Depth / Drop-Rate Metrics"]
  HEALTH --> DASH["Status Dashboard"]

Buffering
A durable, partitioned queue (Kafka, or an equivalent persistent broker) sits between collection and the storage writer. Producers write to the queue and get an ack independent of whether the writer is keeping up, which is what actually decouples the two.

Rate-limiting
A token-bucket limiter in front of the writer caps how fast it attempts to push into storage, so a recovering database is not immediately re-flooded the moment it comes back up. The bucket's burst capacity should match what the buffer can absorb, not an arbitrary number.

Circuit breaker
Wrap the storage write path in a breaker: open after a failure-rate threshold over a rolling window (for example, 50% errors over the last 10 attempts), during which writes fail fast into the buffer instead of blocking on a slow database. Move to half-open after a backoff period to test recovery with a small amount of traffic before fully closing again.

Retry strategy
Exponential backoff with jitter for transient errors, with a hard cap so retries do not themselves become a load source:

giving delays of 200, 400, 800, 1600, 3200, 6400 ms for attempts 1 through 6, reaching the 30-second cap by around attempt 9.

Surfacing pipeline health
Expose queue depth, drop rate, breaker state, and write-success rate as first-class metrics with their own dashboard and alerting, plus a documented behavior contract (how long buffered data survives, what gets dropped first under sustained pressure) so dependent teams know what to expect during an incident instead of just seeing gaps.

Worked Example

Assume normal load into the time-series database is 50,000 points/sec, matched by normal write capacity, so no backlog accumulates in steady state. During an incident, traffic spikes 10x to 500,000 points/sec while write capacity simultaneously drops to 20% of normal (10,000 points/sec) because the same incident is degrading the database itself, a stated worst-case scenario for sizing purposes.

Backlog growth rate:

Sizing the buffer to survive a 5-minute (300 s) incident before either recovery or an operator decision:

At 150 bytes/point (consistent with the same encoded-point-size assumption used for ingest sizing elsewhere):

Provisioning roughly 22 GB of buffer capacity is what actually backs a "we survive a 5-minute downstream outage at 10x traffic" claim. Set a shedding high-watermark at 80% of that: $0.8 \times 22.05 \approx 17.6\ \text{GB}$, past which the pipeline starts dropping lowest-priority series (non-alerting, debug-tier metrics) to preserve budget for anything feeding an active SLO or alert.

Trade-offs and Pitfalls

Buffering trades data loss for latency and cost: a bigger buffer survives a longer outage without dropping anything, but costs more to provision and, if it fills anyway, the operator now has a large backlog to drain (see S3's drain-time math for why backlog drain time can badly exceed outage length) rather than a clean, immediate failure.

A circuit breaker that opens too aggressively (a low failure threshold, a short window) can trip on ordinary transient blips and start buffering unnecessarily, adding latency for no real benefit; one that opens too conservatively keeps hammering an already-struggling database and can make the underlying incident worse. Tune the threshold against the sink's actual recovery behavior, not a default value copied from an unrelated system.

Shedding low-priority series under pressure only works if "priority" was decided in advance, not improvised during the incident. If every team believes their metrics are the important ones, the shedding policy needs an actual, pre-agreed tier assignment, or it becomes a political argument during the worst possible moment to have one.

Direct Answer

Use tail-based sampling with a short per-trace buffering window, so the sampling decision can see whether an error or rare condition appeared anywhere in the trace before deciding to keep or drop it. Split the policy in two: always retain traces carrying an error or rare-severity flag (unconditionally, outside the budget), then probabilistically sample the remaining normal traffic at whatever rate fits what budget is left. The buffer window is the dial between decision correctness (catching a late-arriving error span) and cost (memory held per in-flight trace, plus decision latency).

Structured Elaboration

Sampling pipeline

flowchart LR
  SPANS["Incoming Spans"] --> BUF[("Per-Trace Buffer, N sec window")]
  BUF --> DEC{"Trace complete or window expired"}
  DEC -->|"error / rare signal"| KEEP["Always Retain"]
  DEC -->|"normal"| SAMP["Budget-Aware Sampler"]
  SAMP -->|"sampled in"| KEEP
  SAMP -->|"sampled out"| DROP["Discard"]
  KEEP --> STORE[("Trace Storage")]

Buffering approach
Spans for a given trace ID accumulate in a short-lived per-trace buffer as they arrive out of order from different services. The buffer closes and a decision is made either when the trace looks complete (a root span closes) or when a maximum window expires, whichever comes first, so a trace that never completes cleanly (a dropped span, a crashed service) does not hold the buffer open indefinitely.

Signals feeding the sampling decision

• Any span carrying an error status or exception.
• A latency signal: a span exceeding a defined threshold for its operation.
• A rare, high-severity condition flag emitted explicitly by the service (a specific error code, a business-critical operation type).
• For the remaining normal traffic, a running measure of budget consumed so far in the current period, feeding a dynamic sampling probability.

Decision and retention policy

1. If any signal above fires: retain unconditionally, outside the budget calculation.
2. Otherwise: sample at a probability computed to hit the remaining budget for the period (see worked example), using a consistent, deterministic hash of the trace ID so the decision is reproducible if re-evaluated.

Latency versus correctness trade-off
A longer buffer window catches more late-arriving error signals (higher correctness) but holds more traces in memory longer and delays the retain/drop decision (higher latency and cost). A shorter window decides faster and cheaper but risks finalizing a trace as "normal" just before its error span arrives from a slow downstream hop.

Worked Example

Assume a fleet-wide rate of 50,000 traces/sec, an average trace size of 5,000 bytes (spans, tags, and metadata combined, a stated design input), and a measured error rate of 0.5% of traces (a stated assumption feeding the calculation, not a claim about any specific system).

Total traces/day: $50{,}000 \times 86{,}400 = 4.32\times10^{9}$.

Error traces/day (always kept): $0.005 \times 4.32\times10^{9} = 2.16\times10^{7}$, costing $2.16\times10^{7} \times 5{,}000\ \text{B} \approx 108\ \text{GB/day}$.

Given a fixed storage budget of 500 GB/day, the remaining budget for non-error traces is $500 - 108 = 392\ \text{GB/day}$.

Non-error traces/day: $4.32\times10^{9} - 2.16\times10^{7} = 4{,}298{,}400{,}000$.

Required sampling rate on non-error traffic to spend exactly the remaining budget:

So the system retains 100% of error traces and roughly 1.82% of normal traces, for a blended retention rate of about $500/21{,}600 \approx 2.31\%$ of all traces by volume (total unsampled data would be $4.32\times10^9 \times 5{,}000\ \text{B} \approx 21{,}600\ \text{GB/day}$).

Buffer memory. Holding a 5-second decision window at the full incoming rate: $50{,}000 \times 5{,}000\ \text{B} \times 5\ \text{s} = 1.25\times10^{9}\ \text{B} \approx 1.25\ \text{GB}$ of in-flight trace buffer needed per collector-tier aggregate, a concrete, provisionable memory figure rather than an open-ended "keep enough buffer" statement.

Trade-offs and Pitfalls

Head-based sampling (deciding at the very first span, before the rest of the trace exists) is cheaper and adds no buffering latency, but structurally cannot guarantee error retention: the decision happens before the error span, if any, has even occurred. Tail-based sampling is the only way to honor "always keep errors," at the cost of the buffer described above.

A fixed buffer window creates a silent correctness gap for the tail of the trace-duration distribution: if the window is 5 seconds but a small fraction of traces (the slowest, often the most interesting) take 8+ seconds end to end, those traces get finalized and possibly dropped before their late error span arrives. Mitigate by extending the window specifically for traces that have already crossed a latency threshold mid-flight, rather than using one fixed window for everything.

Dynamic budget-based sampling can create a feedback loop under a real incident: an incident produces more errors, which consumes more of the always-keep budget, which is fine, but if the incident also produces more overall traffic, the non-error sampling rate has to drop to compensate, right when operators most want visibility into the surrounding normal traffic for comparison. Consider reserving a small, fixed floor sampling rate for normal traffic that the budget calculation cannot squeeze to zero.