Observability and Monitoring Architecture Questions

Direct answer

Split retention into hot (raw, fast storage, days), warm (compressed, still full resolution, weeks), cold (downsampled and heavily sampled, up to a year), and archive (compliance-only raw extracts, years, cheapest storage). Alerting stays correct because every alert either reads from the hot/warm tiers where it needs full resolution, or is redesigned to work on the statistical properties (percentiles, error counts) that downsampling preserves. Compliance stays correct because the raw bytes a regulator might ask for are pulled into a separate, signed archive path that bypasses the operational downsampling entirely.

Tiering policy

Downsampling rule for metrics: never discard raw points, only aggregate. Roll up 10s scrapes into 1-minute buckets storing count, sum, min, max, and pre-computed percentile buckets (histogram, not just an average), so alerting on p95/p99 still works post-downsample. Averaging away the histogram is the single most common mistake here: an average of averages silently breaks p99-based alerts.

Downsampling rule for logs: keep 100% of error/warn-level events forever within the cold window; sample info/debug logs at a low, deterministic rate (hash on trace ID) so a given request's logs are either fully kept or fully dropped, never split. That preserves the ability to reconstruct a whole request's log trail if it was selected.

Keeping alerting correct across tiers

• Alert rules that fire on absolute thresholds (error count, latency p99) are pinned to run against hot/warm data only, since that is where fidelity is guaranteed.
• Longer-window alerts (weekly seasonality, slow burn SLO budget) run against cold-tier rollups, but only because the rollup step preserved histograms rather than plain averages, so the alert's percentile math still holds.
• Backtest every downsampling change: replay the last 90 days of raw data through the proposed rollup and confirm the alert would have fired at the same times it fired against raw data, before shipping the rollup config.

Keeping compliance correct across tiers

• Compliance retention is a separate write path, not a side effect of the operational tiers. At ingest, a policy tags records matching compliance scope (PII, regulated data classes, specific tenants); those records get a signed, immutable copy written to the archive tier at full fidelity, independent of what the operational hot/warm/cold pipeline does to the rest of the stream.
• The operational tiers are free to downsample or expire on their own schedule because the compliance obligation is already satisfied by the separate archive copy.
• Archive retrieval is on-demand (async restore), so day-to-day cost stays low, but an auditor's request for raw data from 3 years ago is answerable.

Worked example

Assume a service ingests $G = 500$ GB/day of combined logs and metrics, split 30% metrics / 70% logs, and the storage budget is a hot-tier unit price of $X = \$23\text{/TB-month}$ (a concrete stand-in for the given $X). Illustrative relative unit prices for the other tiers, typical of SSD-backed vs. compressed-object vs. archival storage:

Tier volumes (hot: 7 days raw; warm: next 23 days at 3:1 compression; cold: next 335 days, metrics down 10:1 and logs sampled to 1%; archive: a 2%-of-ingest compliance subset, retained 7 years = 2555 days, compressed 4:1):

Monthly cost:

Compare that to the naive policy of keeping everything raw, at hot-tier price, for the full 7-year compliance window:

Tiering plus targeted downsampling cuts the steady-state storage bill by roughly 99.5% relative to keeping everything at hot-tier fidelity for the retention window, while still satisfying a 7-year compliance obligation and keeping alerting on full-fidelity data for the first month.

flowchart LR
  Ingest[Raw Ingest] --> Hot[Hot 0-7d full res]
  Hot --> Warm[Warm 8-30d compressed 3x]
  Warm --> Cold[Cold 31-365d downsampled 10x, logs 1%]
  Cold --> Archive[Archive 365d+ compliance subset, 4x compressed]
  Hot --> AlertEval[Alert Evaluation]
  Warm --> AlertEval
  Cold --> AlertEval
  Archive --> Restore[On-demand Restore]
  Restore --> AlertEval
  Ingest --> Vault[Signed Compliance Vault]
  Vault --> Archive

Trade-offs and pitfalls

• Downsampling metrics to plain averages instead of histograms is the classic mistake: it looks fine until a p99-based SLO alert goes silent because the percentile can no longer be reconstructed from the rollup.
• Deterministic (hash-based) log sampling beats random sampling because it keeps a request's full log trail together instead of showing half a trace.
• Treating compliance retention as "whatever the operational tiers happen to still have" instead of a separate write path is a common design flaw: an operational cost-cutting change (shortening the cold window) can silently create a compliance gap unless the archive copy is decoupled from it.
• Archive restore latency (minutes to hours, depending on backend) is a real trade-off against cost; if legal holds require faster turnaround, that changes the archive tier choice and its price multiplier.
• Budget monitoring has to be proactive: alert on forecasted spend crossing the budget before the bill arrives, not after, since tier transitions are usually async and lag actual ingest growth.

Build access control as three layers that compose: identity and provisioning (who is this, what teams are they in), a policy engine that maps identity plus team to permitted actions on specific resources, and an immutable audit trail of every access decision. The 50-team scale is what makes a policy-engine approach (rather than hand-rolled per-resource checks) worth the setup cost.

Design

flowchart LR
  A[SSO / SAML IdP] --> B[SCIM Provisioning]
  B --> C[Team / Group Mapping]
  C --> D[Policy Engine: OPA]
  D --> E[Dashboard and Query ACL]
  D --> F[Platform Admin Scope]
  E --> G[Audit Log: append-only]
  F --> G

• Identity and provisioning: SSO via SAML/OIDC for authentication, SCIM (or LDAP sync) for provisioning, so team membership changes (new hire, team transfer, offboarding) propagate automatically instead of relying on manual grants.
• RBAC model: a small set of coarse roles (Viewer, Editor, TeamLead, PlatformAdmin) combined with resource-scoped permissions (dashboard:read, dashboard:write, query:run, query:save, data:ingest). Evaluate as (subject, action, resource, context) through a policy engine (e.g., OPA/Rego) rather than scattering if user.role == 'admin' checks through application code, since 50 teams' worth of dashboard-sharing and cross-team-visibility rules will otherwise become unmaintainable ad hoc logic.
• Namespace/tenant isolation: each team's dashboards, saved queries, and (if applicable) raw telemetry are tagged with a team/namespace ID; the policy engine denies cross-team access by default, and sharing is an explicit grant, not an implicit consequence of both teams querying the same underlying data.
• Audit trail: append-only log of every access decision (actor, action, resource, timestamp, allow/deny), shipped to a separate system (SIEM or dedicated log store) so a compromised or misconfigured application instance can't retroactively edit its own audit history.

What's different about the platform-operator admin role

PlatformAdmin needs privileges no team role should have (upgrading the platform, changing global retention policy, accessing any team's data for support purposes) but that scope is exactly what makes it the highest-risk role. Treat it differently from every other role: require a separate approval workflow for admin grants, use time-boxed or break-glass access (temporary elevation with mandatory justification and MFA, auto-expiring) rather than standing admin permissions, and audit admin actions with the same rigor as the audit trail above, ideally with additional real-time alerting since a compromised admin credential is a platform-wide risk, not a single-team risk.

SaaS vs. on-prem

A quick sizing check on audit volume

A frequent unstated assumption is that audit logging is "too expensive to keep forever." For 50 teams with roughly 30 engineers each performing about 200 dashboard/query actions/day at 300 bytes/audit record:

teams, engineers_per_team, actions_per_day, record_bytes = 50, 30, 200, 300
total_engineers = teams * engineers_per_team                      # 1,500
events_per_day = total_engineers * actions_per_day                # 300,000
bytes_per_day = events_per_day * record_bytes                     # 90 MB/day
total_bytes_1yr = bytes_per_day * 365                              # ~32.85 GB/year

At roughly 32.85 GB for a full year of audit history across the whole platform, audit-log retention is cheap relative to telemetry storage; there's rarely a cost reason to truncate it, which supports keeping compliance-grade audit history far longer than raw telemetry.

Trade-offs and pitfalls

• Embedding authorization checks directly in each service (instead of a shared policy engine) is the design that looks fine at 5 teams and becomes unmaintainable at 50, because every new sharing rule requires a code change in every service that touches dashboards or queries.
• Standing admin access (a permanent PlatformAdmin role assignment with no expiry) is the most common real-world gap between a documented RBAC design and actual practice; break-glass, time-boxed elevation closes that gap but requires operational discipline to actually use.
• On-prem deployments that assume the SaaS identity/audit architecture transfers unchanged will hit friction the first time a customer's SSO provider or compliance requirement doesn't match the platform operator's assumptions; the abstraction boundary (pluggable IdP, pluggable audit sink) needs to be designed in from the start, not retrofitted.
• Least-privilege between on-call engineers and management is a common refinement: an on-call engineer needs broad read access across teams during an incident but not write/admin access, while a manager might need visibility without either; collapsing both into a single "Editor" role loses that distinction.

Direct Answer

Pick the weakest delivery guarantee that still lets you reconstruct correctness at the aggregation layer, rather than paying the latency and complexity cost of strong guarantees everywhere. For most telemetry pipelines that means at-least-once delivery plus idempotent, deduplicated writes: true end-to-end exactly-once delivery across independent systems does not really exist, what you can build is a system that behaves as if it were exactly-once because duplicate deliveries have no effect on the result.

Structured Elaboration

At-most-once
A point may be silently lost, never duplicated. Cheapest and lowest-latency, but every dropped point is gone. Acceptable only for high-volume, low-value telemetry where a small, unbiased loss rate does not meaningfully change an aggregate.

At-least-once
No steady-state data loss, but retries can produce duplicates. This is the practical default for most producers (send, wait for ack, retry on timeout, occasionally re-deliver something that actually did arrive). Correctness at the aggregation layer requires deduplication: an idempotent write keyed by a unique identifier, so a duplicate delivery does not change the result.

Exactly-once
The goal is a metric aggregation that reflects each event's effect precisely once. In practice this is built, not given by any single component: it requires end-to-end idempotency (deterministic dedup keys), transactional or checkpointed writes (a write-ahead log tying the aggregate update and the consumer offset commit together atomically), and durable state. The cost is coordination overhead and added write latency.

Eventual consistency
Aggregates may be temporarily stale or incomplete, converging to correct only after all in-flight and late-arriving events settle. Handled with event-time windowing, watermarks that define how late an event can arrive before it's excluded, and a correction mechanism (recompute or emit a delta) for events that show up after their window closed.

Making a weaker guarantee correct in practice

• Deduplication IDs: a producer-assigned, monotonically increasing sequence number per stream. The consumer tracks a high-water mark per stream and discards anything at or below it.
• Idempotent writes: aggregation writes as upserts keyed by (metric, labels, timestamp window), so replaying the same input overwrites rather than accumulates.
• Write-ahead logs: the consumer writes its aggregate update and its new checkpoint offset in a single atomic step, then acks the upstream offset. On crash and restart, it replays only the un-checkpointed range, which is safe because that replay is itself idempotent.

Checkpoint and dedup flow

flowchart LR
  EVT["Event with sequence id"] --> DEDUP{"seq > high-water mark?"}
  DEDUP -->|"yes, new"| APPLY["Apply to Aggregate"]
  DEDUP -->|"no, duplicate"| DISCARD["Discard"]
  APPLY --> CKPT[("Atomic: Aggregate + Checkpoint")]
  CKPT --> ACK["Ack Offset Upstream"]

Worked Example

Assume 3 producers each sending 100 events into a 1-minute rollup window, for a correct total of 300 events. One producer's network connection times out on 2 of its events, so it retries both, and (because at-least-once delivery does not deduplicate at the transport layer) both the original and the retried copy arrive.

Without deduplication, the consumer counts:

A small number for one window, but a systematic bias that compounds across every window and every retried event at scale, not a one-time rounding error.

With deduplication (each producer assigns a monotonically increasing sequence number per stream, and the consumer tracks a high-water mark per producer, discarding anything at or below the mark it has already seen):

The dedup mechanism restores the exact correct count using only transport-layer at-least-once delivery, no exactly-once broker feature required.

Trade-offs and Pitfalls

Sequence-number-based dedup (a high-water mark per stream) only works correctly for pure retries, redelivery of the same sequence number. It breaks if the transport can reorder events beyond a small window, since a legitimately new event with a lower sequence number than one already seen would be incorrectly discarded as a duplicate. If your transport allows significant reordering, dedup by explicit unique event ID with a time-bounded lookup window instead of a strict high-water mark.

Exactly-once semantics inside a single system (say, a Kafka Streams application with transactional writes) do not extend automatically to an external sink. The moment you write to an external time-series database or object store, you are back to needing idempotent writes at that boundary, transactional guarantees inside the stream processor do not make the external write exactly-once by themselves.

Choosing at-least-once plus dedup as the default is right for the vast majority of telemetry, but not for every case: a billing-critical count (metered usage that directly drives a customer invoice) may warrant paying the extra coordination cost for stronger guarantees specifically at that boundary, while everything else stays on the cheaper default.

Direct answer

Require every signal (metric, log line, trace span) to carry a small set of common identifiers, most importantly trace_id (or a request/correlation ID where full tracing isn't wired up yet), plus service_name and a timestamp. Build a lightweight mapping index, keyed by those identifiers, that points to where each related log or span physically lives (shard, offset, time range) rather than storing the events themselves twice. When an identifier is missing, fall back to narrowing by time window and service first, then use a scored, probabilistic match on top of that narrowed candidate set instead of searching the whole corpus.

Structured elaboration

Required identifiers per event:

• trace_id / span_id: the strong identifier, present when full distributed tracing is instrumented.
• service_name, host_or_pod_id: contextual keys usable even when tracing isn't present.
• event_timestamp: needed for the time-window fallback and for ordering.
• metric_dimensions / anomaly_id: on the metrics side, the anomaly detector emits an anomaly document carrying whatever identifiers were present on the underlying series.

Index structure, not a duplicate copy: the mapping store is a pointer index (topic/partition/offset for logs, trace/span ID for traces), not a second copy of log or trace bodies. Logs and traces stay in their existing systems (search index, trace store); the correlation index only tells the query which shard and offset to fetch.

Query flow:

1. Direct ID lookup, if the anomaly carries a trace_id: O(1) pointer lookup into the mapping store.
2. Indexed search, if not: query the log/trace index filtered by service_name and a time window around the anomaly.
3. Probabilistic fallback, if identifiers are sparse: score candidates from step 2 by temporal proximity, shared host, and payload fingerprint, and return ranked results with a visible confidence, not a silent best guess.

flowchart LR
    A[Metric anomaly] --> ID{trace_id present?}
    ID -- yes --> M[(Mapping index: id -> pointer)]
    M --> L1[Fetch log/trace by pointer]
    ID -- no --> TW[Filter: service + time window]
    TW --> SC[Score candidates: proximity, host, fingerprint]
    SC --> L2[Ranked candidates + confidence]

Maintaining the mapping: on ingest, if a strong identifier is present, upsert a bounded (capped) pointer list into the mapping store for that ID and set a TTL matching the signal's own retention. A background job prunes expired entries so the index doesn't grow unbounded past what the underlying logs and traces themselves retain.

Worked example

Index sizing. Assume the cluster produces $50{,}000$ events/sec across logs and trace spans combined:

Each mapping entry (topic, partition, offset, timestamp, service ID) is about 40 bytes:

At a 7-day retention window matching typical hot log retention:

This is the number that justifies capped pointer lists and TTL-based pruning in the design above: an unbounded, uncapped index at this event rate would grow past a terabyte inside a week even though it stores no event bodies, only pointers.

Fallback candidate-set size when the identifier is missing. With 200 services sharing the $4.32 \times 10^9$ daily events roughly evenly:

Narrowing to a $\pm 2.5$s window (5s total) around the anomaly timestamp before service-level scoring:

Filtering by service and a 5-second window before running any fuzzy scoring shrinks the search space from 21.6 million events/day for that service down to about 1,250 candidates, small enough for a scoring pass to run against on every anomaly without scanning the full day's log volume.

Trade-offs & pitfalls

Common wrong turns: treating the mapping index as a full copy of the underlying signals instead of a pointer index (this doubles storage for no correlation benefit and creates a second source of truth to keep consistent); running the fuzzy-scoring pass over the entire day's events instead of narrowing by service and time window first, which turns an O(1,250) scoring problem into an O(21,600,000) one for no accuracy gain; and silently returning the single top-scored fuzzy match without surfacing its confidence, which looks like a correlation but is actually a guess an on-call engineer has no way to sanity-check.

Direct answer

Don't compute percentiles from raw points at query time. Have every collector maintain a local mergeable sketch (t-digest or an HDR histogram) over each rollup window, ship the sketch instead of the raw values, and merge sketches hierarchically (minute into hour, hour into day) so a query over weeks or months merges a small number of pre-combined objects instead of scanning raw samples. The accuracy you give up is bounded and tunable: it comes from the sketch's compression parameter, not from randomly dropping data, so you can trade a known amount of tail-percentile error for a known reduction in merge work.

Structured elaboration

Sketch choice:

• t-digest: variable bucket width, denser (more accurate) near the tails where percentile queries usually care most, mergeable, size controlled by a compression parameter $\delta$ (roughly, number of centroids).
• HDR histogram: fixed relative-error buckets across a configured value range, deterministic worst-case error bound, very fast to merge (array addition), best when the metric's range is known and bounded (for example request latency in milliseconds).
• Use t-digest for open-ended or skewed distributions; use HDR when the value range is known and the deterministic bound matters more than tail density.

Hierarchical rollups: each collector emits a 1-minute sketch. A rollup job merges 60 one-minute sketches into an hourly sketch, and 24 hourly sketches into a daily sketch. Retention is tiered: keep 1-minute sketches for 7 days, hourly for 30 days, daily beyond that. This is what makes "merge partial sketches from many collectors" tractable at high fan-in: each collector contributes one small sketch per window rather than a stream of raw points, and the merge tree does the aggregation once instead of at every query.

Merging at query time: the query picks the coarsest rollup tier that covers the requested range, merges those objects (t-digest merge is just concatenating and re-clustering centroids; HDR merge is bucket-wise addition), and computes the percentile from the merged sketch.

Handling very high fan-in (many collectors, very high ingest volume): the same hierarchy scales horizontally: instead of one job merging all collectors' sketches directly, shard collectors into groups, merge within a group first, then merge group-level sketches at the next tier. This bounds the fan-in of any single merge step regardless of how many collectors exist upstream.

flowchart LR
    C1[Collector] -- 1-min sketch --> M1[Minute merge]
    C2[Collector] -- 1-min sketch --> M1
    M1 -- 60 merged --> H[Hourly sketch]
    H -- 24 merged --> D[Daily sketch]
    Q[Percentile query] -- picks coarsest tier covering range --> H
    Q --> D
    D -- retained 30d+ --> COLD[(Cold rollup store)]

Worked example

Merge count reduction. For a 30-day 99th-percentile query, merging at 1-hour granularity versus 1-minute granularity:

Querying against hourly rollups means merging 60x fewer objects than querying against minute rollups, which is the primary win of hierarchical sketching: it shrinks the object count the query has to combine, not just the byte count of any one object.

t-digest size at a given compression. A t-digest with compression $\delta$ holds roughly $\delta$ centroids, each storing a mean and a count (two 8-byte doubles = 16 bytes):

At $\delta = 100$: $\text{digestBytes} = 1{,}600$ bytes per digest.

Storage over a year, at scale. For $\text{services} = 5{,}000$ tracked series, keeping hourly digests for 365 days:

Compare to keeping raw 15s points for those same 5,000 series for a year, at 16 bytes/point:

Worth being honest about what this number says: at $\delta = 100$, sketch storage is only 2.4x smaller than keeping the equivalent raw points, because each digest is representing many raw points (5,760 raw points/day per series versus 24 digests/day per series, so each digest stands in for 240 points). The real win from sketching at this compression level is the 60x merge-count reduction above, not storage. If storage is the binding constraint, dropping $\delta$ to 50 halves digest size (800 bytes) and roughly doubles the storage reduction factor to about 4.8x, at the cost of coarser centroid resolution and larger tail error.

Trade-offs & pitfalls

Common wrong turns: merging sketches from different tiers as if they were interchangeable (a 1-minute sketch and an hourly sketch built with different compression settings can merge, but mixing merge granularities inconsistently across a query makes the resulting error bound harder to reason about, so keep the compression parameter fixed across a tier); treating a merged sketch's percentile as exact (it is an estimate with error concentrated where the sketch is coarsest, which for t-digest is the middle of the distribution, not the tails); and picking HDR for an unbounded-range metric (queue depth that can spike arbitrarily) where the fixed value range either clips outliers or forces an oversized histogram, when t-digest's variable bucketing is the better fit there.