Observability and Monitoring Architecture Questions

Set the policy by working backward from what each query pattern actually needs, then verify the storage savings with the arithmetic rather than picking round numbers and hoping. A reasonable concrete policy: raw metrics at native resolution for 14 days, 5-minute rollups for 1 year, hourly rollups for years 2 through 5; full traces for 30 days, then 1% sampled for the following 11 months.

Rationale by window

flowchart LR
  A[Raw ingest: native res] -->|14 days| B[Raw tier: hot]
  B -->|downsample| C[5-min tier: 1 year]
  C -->|downsample| D[Hourly tier: years 2-5]
  D -->|expire| E[Deleted]
  F[Trace ingest] -->|30 days full| G[Full trace tier]
  G -->|sample 1%| H[Sampled trace tier: 11 months]

• 14 days raw: covers essentially all incident debugging, since almost every retro or root-cause investigation happens within two weeks of the event, and alerting needs full resolution on recent data to avoid missing short spikes.
• 1 year at 5-minute rollups: supports capacity planning and seasonal comparisons (week-over-week, month-over-month) without needing per-second precision; 5 minutes is short enough to still show diurnal patterns clearly.
• Years 2-5 at hourly rollups: supports long-term trend and year-over-year growth analysis; anything finer than hourly at this age is rarely queried and expensive to keep.
• 30 days full traces: matches the raw-metrics window for the same reason, full-fidelity root cause work happens fast, and traces are the most expensive telemetry type per unit.
• 1% sampled for 11 more months: preserves enough statistical signal for "did this class of error exist a few months ago" investigations without paying for full trace volume; always retain 100% of traces tied to errors or SLO breaches regardless of the sampling rate (a fixed-percentage sample can otherwise miss the rare traces investigators actually want).

Verifying the storage savings

For 1,000,000 active series, using the same 2-bytes/compressed-raw-sample and 8-bytes/downsampled-point (4 aggregates: min, max, sum, count, at roughly 2 bytes each) assumptions used elsewhere in TSDB capacity planning:

series = 1_000_000
compressed_bytes_per_raw_sample = 2
agg_bytes_per_downsampled_point = 8

def samples(days, interval_s):
    return series * (days * 86400 / interval_s)

raw_bytes = samples(14, 15) * compressed_bytes_per_raw_sample
ds_bytes = samples(365, 300) * agg_bytes_per_downsampled_point
hourly_bytes = samples(1460, 3600) * agg_bytes_per_downsampled_point
total_tiered_bytes = raw_bytes + ds_bytes + hourly_bytes

allraw_bytes = samples(14 + 365 + 1460, 15) * compressed_bytes_per_raw_sample

Result: raw tier = 161.28 GB, 5-min tier = 840.96 GB, hourly tier = 280.32 GB, total tiered storage over the full 5-year window ≈ 1.283 TB, versus an all-raw-forever equivalent of ≈ 21.19 TB for the same window, a 16.5x reduction. The 5-minute tier dominates total storage (840 GB of the 1.28 TB) precisely because it covers the most time (1 year) at the finest surviving resolution; that's useful to know when deciding whether to push the raw window shorter or the 5-minute window's resolution coarser if the budget gets tighter.

For traces, at 10,000 traces/sec with an assumed 4 KB compressed size per trace: full 30-day retention stores about 103.68 TB, while the following 11 months at 1% sampling adds roughly 11.40 TB, so the sampled tail costs about 11% as much as the initial 30-day full window despite covering over 10x the time span.

What you can and can't still answer after each window closes

Trade-offs and pitfalls

• Percentile aggregates (p95, p99) do not survive naive downsampling: averaging five 1-minute p99 values is not the same number as the true p99 across that 5-minute window. If percentile fidelity matters at the rollup tier, you need to store enough of a histogram or sketch (not just min/max/sum/count) to recompute percentiles, which raises the per-point byte cost above the 8-byte assumption used here.
• A fixed sampling percentage for traces (1% flat) will statistically under-represent rare-but-important request types unless it's stratified or combined with the always-keep-errors/SLO-breach rule; a pure random sample optimizes for "typical" traffic, which is exactly what you don't need for debugging.
• Retention policy that isn't enforced automatically (a manual cleanup job, or "we'll get to it") tends to silently become the accidental real retention policy; tie expiry to the storage engine's native TTL/compaction mechanism rather than a side script.
• Communicating the policy to teams matters as much as setting it: if engineers don't know that a 3-week-old spike is only visible at 5-minute resolution, they'll draw wrong conclusions from a smoothed-out graph without realizing detail was lost.

Default to logical isolation (shared infrastructure with hard per-tenant quotas and QoS enforcement) for the bulk of tenants, and offer physical isolation (dedicated shards or node pools) as an explicit, metered upgrade path for tenants whose usage or SLA requirements outgrow what shared quotas can safely guarantee. The isolation model and the migration path are two sides of the same design.

Architecture

flowchart LR
  A[Tenant Requests] --> B[Ingress: Auth and RBAC]
  B --> C[Per-Tenant Shard / Queue]
  C --> D[Shared Ingestion Pool]
  C --> E[Dedicated Ingestion Pool]
  D --> F[Query Gateway: QoS Scheduler]
  E --> F
  F --> G[Shared Query Compute]
  F --> H[Dedicated Query Compute]
  I[Billing / Quota Manager] --> B
  I --> F

• Logical isolation: per-tenant partitions/queues on shared compute, enforced with token-bucket rate limits at ingress and query-time concurrency caps; cheapest, and sufficient for the majority of tenants whose usage is well within their quota most of the time.
• Physical isolation: dedicated shard, node pool, or account for a tenant; strongest guarantee, but the operational and cost overhead of running fully separate infrastructure per tenant doesn't scale to hundreds of tenants, so it has to be selective.
• Query-level QoS: priority classes (interactive dashboard queries vs. batch/backfill queries), per-tenant concurrency limits, and admission control that sheds low-priority load before it degrades everyone; this is what actually prevents a noisy tenant's expensive query from starving others on shared compute, since ingestion isolation alone doesn't protect the read path.
• Billing-aware quotas: map each tenant's plan tier to a concrete ingest-rate and query-concurrency quota; soft-limit warnings before hard throttling, and an explicit overdraft/pay-as-you-go path rather than a silent hard cutoff. Retention is part of the same per-tenant contract, not a platform-wide constant: a tenant's plan tier should set its own retention window (e.g., 7 days on a shared/basic tier vs. 90 days on a dedicated tier), enforced as tenant-scoped TTL policy in the storage layer so one tenant's longer retention SLA doesn't force everyone else to pay for the same window.

Sizing the admission-control headroom

The core quantitative question for logical isolation is: how much burst capacity can the shared pool actually absorb before a legitimate burst from one tenant risks starving others? Take a platform with total ingest capacity $C = 2{,}000{,}000$ samples/sec shared across $N = 500$ tenants, where baseline quotas are provisioned to consume a target fraction $u$ of total capacity (leaving headroom for bursts), and tenants are allowed to burst up to $m\times$ their baseline:

If a fraction $f$ of tenants burst simultaneously while the rest sit at baseline, total load must stay under capacity:

Substituting $\text{baseline}_{\text{tenant}} = uC/N$ and simplifying:

With $u = 0.6$ (provision baseline to consume 60% of capacity, leaving 40% headroom) and $m = 5$ (allow a 5x burst):

C, N, u, m = 2_000_000, 500, 0.6, 5
baseline = (u * C) / N          # 2,400 samples/sec/tenant
burst = m * baseline            # 12,000 samples/sec/tenant
f_max = (1/u - 1) / (m - 1)     # 0.1667
max_bursting = f_max * N        # 83.3 tenants

Result: baseline quota is 2,400 samples/sec/tenant, burst allowance is 12,000 samples/sec/tenant, and up to about 16.7% of tenants (roughly 83 of 500) can burst simultaneously at 5x without exceeding total capacity. Plugging $f_{max}$ back into the original inequality confirms it lands exactly at capacity (2,000,000 samples/sec), which is the check that the derivation is self-consistent. This is the number that should actually drive the admission controller's global burst budget, not a guess: if more than ~83 tenants try to burst at once, the controller has to start denying or queuing burst requests rather than granting them all.

Migrating a tenant from shared to dedicated

1. Trigger: sustained usage consistently near quota (not just occasional bursts), or an explicit SLA purchase requiring guaranteed isolation.
2. Provision dedicated shard/node pool ahead of cutover.
3. Dual-write or replicate the tenant's recent data into the new dedicated shard while it's still live on the shared pool.
4. Cut over routing at the control plane (ingress rules keyed on tenant ID) once the dedicated shard is caught up; this should be a routing change, not a data migration event, so it can be near-zero-downtime.
5. Decommission the tenant's shared-pool footprint after a verification window, and keep the cutover reversible in case the dedicated shard has an unexpected issue.

Trade-offs and pitfalls

• Sizing baseline quotas at $u$ close to 1.0 (using nearly all capacity for guaranteed baseline) leaves almost no burst headroom, which defeats the purpose of a shared pool; the $u$ vs. burst-headroom trade-off above should be an explicit, revisited decision, not a default.
• Query-level QoS is often skipped because ingestion isolation feels like "the isolation problem," but an expensive ad-hoc query from one tenant can degrade shared query compute even when every tenant's ingestion is perfectly isolated; both paths need protection independently.
• A migration path that isn't reversible (no fallback if the dedicated shard has a problem post-cutover) turns a capacity upgrade into a risk event; always keep the shared-pool footprint alive through a verification window.
• Billing-aware quotas without a clear soft-limit warning stage turn every quota breach into a support ticket; the graduated response (warn, throttle, then hard-limit) matters as much as the quota number itself.

Direct Answer

Pick the weakest delivery guarantee that still lets you reconstruct correctness at the aggregation layer, rather than paying the latency and complexity cost of strong guarantees everywhere. For most telemetry pipelines that means at-least-once delivery plus idempotent, deduplicated writes: true end-to-end exactly-once delivery across independent systems does not really exist, what you can build is a system that behaves as if it were exactly-once because duplicate deliveries have no effect on the result.

Structured Elaboration

At-most-once
A point may be silently lost, never duplicated. Cheapest and lowest-latency, but every dropped point is gone. Acceptable only for high-volume, low-value telemetry where a small, unbiased loss rate does not meaningfully change an aggregate.

At-least-once
No steady-state data loss, but retries can produce duplicates. This is the practical default for most producers (send, wait for ack, retry on timeout, occasionally re-deliver something that actually did arrive). Correctness at the aggregation layer requires deduplication: an idempotent write keyed by a unique identifier, so a duplicate delivery does not change the result.

Exactly-once
The goal is a metric aggregation that reflects each event's effect precisely once. In practice this is built, not given by any single component: it requires end-to-end idempotency (deterministic dedup keys), transactional or checkpointed writes (a write-ahead log tying the aggregate update and the consumer offset commit together atomically), and durable state. The cost is coordination overhead and added write latency.

Eventual consistency
Aggregates may be temporarily stale or incomplete, converging to correct only after all in-flight and late-arriving events settle. Handled with event-time windowing, watermarks that define how late an event can arrive before it's excluded, and a correction mechanism (recompute or emit a delta) for events that show up after their window closed.

Making a weaker guarantee correct in practice

• Deduplication IDs: a producer-assigned, monotonically increasing sequence number per stream. The consumer tracks a high-water mark per stream and discards anything at or below it.
• Idempotent writes: aggregation writes as upserts keyed by (metric, labels, timestamp window), so replaying the same input overwrites rather than accumulates.
• Write-ahead logs: the consumer writes its aggregate update and its new checkpoint offset in a single atomic step, then acks the upstream offset. On crash and restart, it replays only the un-checkpointed range, which is safe because that replay is itself idempotent.

Checkpoint and dedup flow

flowchart LR
  EVT["Event with sequence id"] --> DEDUP{"seq > high-water mark?"}
  DEDUP -->|"yes, new"| APPLY["Apply to Aggregate"]
  DEDUP -->|"no, duplicate"| DISCARD["Discard"]
  APPLY --> CKPT[("Atomic: Aggregate + Checkpoint")]
  CKPT --> ACK["Ack Offset Upstream"]

Worked Example

Assume 3 producers each sending 100 events into a 1-minute rollup window, for a correct total of 300 events. One producer's network connection times out on 2 of its events, so it retries both, and (because at-least-once delivery does not deduplicate at the transport layer) both the original and the retried copy arrive.

Without deduplication, the consumer counts:

A small number for one window, but a systematic bias that compounds across every window and every retried event at scale, not a one-time rounding error.

With deduplication (each producer assigns a monotonically increasing sequence number per stream, and the consumer tracks a high-water mark per producer, discarding anything at or below the mark it has already seen):

The dedup mechanism restores the exact correct count using only transport-layer at-least-once delivery, no exactly-once broker feature required.

Trade-offs and Pitfalls

Sequence-number-based dedup (a high-water mark per stream) only works correctly for pure retries, redelivery of the same sequence number. It breaks if the transport can reorder events beyond a small window, since a legitimately new event with a lower sequence number than one already seen would be incorrectly discarded as a duplicate. If your transport allows significant reordering, dedup by explicit unique event ID with a time-bounded lookup window instead of a strict high-water mark.

Exactly-once semantics inside a single system (say, a Kafka Streams application with transactional writes) do not extend automatically to an external sink. The moment you write to an external time-series database or object store, you are back to needing idempotent writes at that boundary, transactional guarantees inside the stream processor do not make the external write exactly-once by themselves.

Choosing at-least-once plus dedup as the default is right for the vast majority of telemetry, but not for every case: a billing-critical count (metered usage that directly drives a customer invoice) may warrant paying the extra coordination cost for stronger guarantees specifically at that boundary, while everything else stays on the cheaper default.

For high write throughput with long retention, pick a system built around a distributed, horizontally-scalable ingest path and object-storage-backed long-term data: VictoriaMetrics (cluster mode) or Cortex/Mimir. Plain single-node Prometheus and open-source InfluxDB do not scale writes horizontally, so neither can be "the" storage engine at this scale; they can still front the workload as a scrape/agent layer.

How to reason about the comparison

A time-series storage engine's fitness for a workload comes down to four structural questions, not brand names:

1. Can it accept writes from more than one process at once (horizontal ingest), or is there a single write owner?
2. Does long-range querying require scanning many raw blocks, or does it have a compaction/downsampling path that keeps range queries fast as retention grows?
3. Is tenant/query isolation built into the storage layer or bolted on?
4. What is the operational unit you have to run and keep healthy (one binary vs. a distributed cluster of ingesters, queriers, compactors, and an object store)?

Recommendation, with a sized example

Take a concrete version of "high write throughput, long retention": 5,000,000 active series scraped every 15 seconds, retained for 1 year.

Uncompressed, each sample is 16 bytes (8-byte timestamp + 8-byte float), so raw daily volume is $28.8\times10^{9} \times 16 = 460.8$ GB/day, which is why nobody stores time series uncompressed. Using an assumed post-compression average of 1.3 bytes/sample (delta-of-delta timestamps plus XOR-encoded values on reasonably well-behaved metrics, an assumption, not a measurement):

At 333k samples/sec sustained, a single Prometheus process is already past what one box comfortably ingests and indexes without heavy vertical scaling and shard-by-hand federation; you need a system that shards ingest natively. Between VictoriaMetrics cluster and Cortex/Mimir, the deciding factor is usually operational headcount: VictoriaMetrics ships fewer distinct components for the same job (no separate ring-coordination service, no separate compactor/store-gateway/ruler split) and is the better default when you don't need Cortex's native multi-tenant billing isolation. If the workload also has to serve genuinely separate customers with hard tenant isolation guarantees (not just label-based logical separation), Cortex or Mimir's first-class tenant model earns its extra operational surface.

Trade-offs and pitfalls

• The most common wrong turn is picking Thanos or Cortex purely because they're "the Prometheus-compatible answer" without checking whether the extra components (Store Gateway, Compactor, Ruler, an object store, and a coordination service) are actually staffed and monitored; an unmonitored Compactor silently failing means your long-range queries quietly get slower for months before anyone notices.
• InfluxDB OSS gets recommended reflexively by teams that already use InfluxQL/Flux tooling, but OSS genuinely does not horizontally scale writes; that only becomes true in Enterprise or Cloud, which changes the cost model entirely (licensing, not just infra).
• Compression ratio is workload-dependent, not a constant: dense, low-variance counters compress far better than noisy gauges, because standard time-series compression works by encoding the change from one value to the next, so a single "bytes/sample" number from a vendor benchmark rarely transfers directly to your cardinality mix.
• Multi-tenancy bolted on via label conventions (a tenant label plus query-time filtering) is not isolation: a noisy tenant's cardinality growth still degrades the shared ingest and index for everyone. Only Cortex/Mimir/VictoriaMetrics-cluster's structural tenant separation actually protects against that.

Time-series compression works because consecutive samples in a series are usually similar to each other: timestamps arrive at near-regular intervals, and values tend to change by small amounts (or not at all) between samples. Block-chunking, delta-of-delta timestamp encoding, XOR-based float compression, and label dictionary encoding each exploit a different piece of that similarity.

How each technique works

Block-chunking: samples are grouped into fixed time-window or fixed-count blocks (e.g., 2-hour windows), each with its own compressed byte stream and a small metadata header (start/end timestamp, min/max value). This lets a query engine skip whole blocks that fall outside a requested range without decompressing them, and it bounds how much has to be re-encoded when new data arrives (you only ever append to the current open block).

Delta-of-delta timestamp encoding: instead of storing each timestamp, store the delta from the previous timestamp, then the delta of that delta. For a series scraped at a fixed interval, the delta between consecutive timestamps is constant, so the delta-of-delta is exactly zero almost every time, which collapses to a single control bit per sample.

XOR-based float compression (Gorilla): XOR the current value's bit pattern against the previous value's bit pattern. If the value hasn't changed much, most of the leading bits (sign, exponent, high mantissa) and trailing bits (low mantissa) of the XOR result are zero, and only a short "meaningful" middle span needs to be stored explicitly.

Dictionary encoding for labels: label keys and values repeat across enormous numbers of series (env=prod appears on millions of series), so each distinct string is stored once in a dictionary and every series references it by a small integer ID instead of repeating the string.

Worked example: counter vs. sparse gauge

The following is executed Python (struct-level IEEE-754 bit manipulation, no external data) so the numbers are exactly reproducible:

import struct

def bits(f):
    return struct.unpack('>Q', struct.pack('>d', f))[0]

def leading_zeros(x, width=64):
    return width if x == 0 else width - x.bit_length()

def trailing_zeros(x, width=64):
    if x == 0: return width
    c = 0
    while (x & 1) == 0:
        x >>= 1; c += 1
    return c

def xor_meaningful_bits(prev, cur):
    x = bits(prev) ^ bits(cur)
    if x == 0: return 0, 64, 0
    lz, tz = leading_zeros(x), trailing_zeros(x)
    return lz, tz, 64 - lz - tz

# dense monotonic counter: http_requests_total, +1 per 15s scrape
print(xor_meaningful_bits(184320.0, 184321.0))
# sparse gauge, small fluctuation: temperature idling
print(xor_meaningful_bits(21.4, 21.9))
# sparse gauge, large jump: temperature spikes
print(xor_meaningful_bits(21.9, 87.2))

Output:

(28, 35, 1)   # counter +1 step: 1 meaningful bit
(16, 47, 1)   # gauge, small change: also 1 meaningful bit
(9, 0, 55)    # gauge, large jump: 55 meaningful bits (almost the full double)

And for delta-of-delta timestamps on a realistically jittered 15-second scrape, with the full input pinned so the result is exactly reproducible (10 explicit timestamps, which by the $D_i$ formula above yields 8 delta-of-delta values, since each output needs 3 consecutive timestamps):

timestamps = [
    1720000000, 1720000015, 1720000030, 1720000045, 1720000060,
    1720000075, 1720000089, 1720000105, 1720000120, 1720000134,
]
deltas = [timestamps[i] - timestamps[i - 1] for i in range(1, len(timestamps))]
delta_of_delta = [deltas[i] - deltas[i - 1] for i in range(1, len(deltas))]
print("deltas:", deltas)
print("delta-of-delta:", delta_of_delta)

Output:

deltas:          [15, 15, 15, 15, 15, 14, 16, 15, 14]
delta-of-delta:  [0, 0, 0, 0, -1, 2, -1, -1]

4/8 are exactly 0 (single control bit each): the first four scrape gaps land exactly on the 15-second interval, then jitter shows up as a 14s gap, a 16s gap, and two more gaps that miss the interval by one second, each producing a nonzero delta-of-delta.

Which techniques help most for which shape

The counter case shows why Gorilla-style encoding is the default for metrics generally: even though the value itself is always changing (climbing), the bit pattern delta between consecutive floats stays small as long as the relative step size is small, which is true for almost all real counters. The gauge case shows the failure mode: a big jump changes the exponent bits, which wipes out both the leading-zero and trailing-zero runs simultaneously, so the encoding degrades toward storing the value nearly raw.

Trade-offs and pitfalls

• A common wrong turn is assuming XOR compression is "for gauges" and delta-encoding is "for counters" as a rule; the real determinant is how much the bit pattern changes step to step, which correlates with relative magnitude change, not with the metric type label.
• Partial decompression for range queries requires the block-level index (min/max timestamp, min/max value) to be cheap to read without decoding the compressed body; if you skip that index to save space, every range query degrades to a full block scan.
• Block size is a real tuning knob, not a footnote: too small and per-block metadata overhead dominates; too large and a single noisy stretch (like the sparse-gauge jump) drags down the compression ratio for the whole block and increases decompression latency for small range queries.
• Dictionary encoding for labels needs a compaction/garbage-collection story; a dictionary that only grows (never reclaims IDs for labels that stop being used) becomes its own unbounded-cardinality problem over long retention.