Observability and Monitoring Architecture Questions

Direct Answer

Route queries by data age: hot is optimized for write and point-lookup latency at full resolution, warm keeps full resolution but trades write-path speed for better compression, and cold and archive downsample aggressively and lean on cheap object storage, accepting slower, block-oriented reads. The tiering only pays off if you also downsample, keeping full resolution for 7 years costs an order of magnitude more than the numbers below show.

Structured Elaboration

Rehydration is bounded by making every tier block-addressable by time range and label hash: a query for a specific series over a specific window only pulls the blocks that could contain it, using the manifest's Bloom filters (compact per-block structures that can say "this block definitely doesn't contain that series" without opening it) to skip the rest, rather than scanning a tier wholesale.

Tier flow

flowchart LR
  Q["Query Router"] --> HOT[("Hot: 1 day")]
  Q --> WARM[("Warm: 30 days")]
  Q --> COLD[("Cold: 1 year, downsampled")]
  Q --> ARC[("Archive: 7 years, downsampled")]
  HOT -->|"compaction"| WARM
  WARM -->|"downsample to 5min"| COLD
  COLD -->|"downsample to 1hr"| ARC
  ARC -.->|"on-demand rehydrate"| Q

Worked Example

Assume 1,000,000 active series, each scraped every 15 seconds (a stated design input). Raw points per series per day:

Total raw points/day across the fleet: $1{,}000{,}000 \times 5{,}760 = 5.76\times10^{9}$.

Hot (1 day). Assume time-series-optimized compression achieves roughly 2 bytes/point (a stated assumption for delta-of-delta timestamp and XOR value encoding, not a cited benchmark), plus 30% index overhead:

Warm (30 days). Larger compaction blocks improve the ratio slightly to 1.5 bytes/point, still at full resolution:

Cold (1 year). Downsample to 5-minute resolution, a 300s/15s = 20x reduction in point count, at 2 bytes/point:

Had the cold tier kept raw resolution instead, the year would cost $5.76\times10^{9} \times 365 \times 2\ \text{B} \approx 4{,}204.8\ \text{GB}$, about 20x more, which is the whole reason the downsample step exists.

Archive (years 2 to 7, 6 years). Downsample further to 1-hour resolution (24 samples/day), at 1.5 bytes/point since bulk cold storage compresses better:

Total 7-year footprint (before replication): $15.0 + 259.2 + 210.2 + 78.8 \approx 563.2\ \text{GB}$. Kept at raw resolution for all 7 years instead, the footprint would be roughly $4{,}204.8 \times 7 \approx 29{,}434\ \text{GB} \approx 29.4\ \text{TB}$, about 52x larger. That gap is the entire economic argument for tiering with downsampling rather than tiering on storage class alone.

Trade-offs and Pitfalls

Downsampling is lossy by design: once a cold-tier block collapses 20 raw points into one 5-minute aggregate, you cannot recover the original spikes inside that window. Decide per metric type whether that is acceptable (aggregate SLI counters, usually fine) or not (a metric feeding an anomaly-detection model that needs the raw shape, usually not, and should either stay hot longer or get its own retention policy).

Cross-tier query stitching is a common failure point: a dashboard spanning "last 45 days" crosses the hot/warm boundary and the warm/cold boundary is close by too, and the query planner has to merge full-resolution warm data with 5-minute cold data without a visible discontinuity in the chart. Test this seam explicitly, it is where tiering bugs show up in production, not inside a single tier.

Archive rehydration cost is easy to underestimate: a 1-hour restore-initiation SLA sounds fast until a customer asks for a 3-year lookback across 1,000 series and that pulls thousands of archive blocks, each carrying its own restore latency and egress cost. Expose rehydration as an explicit, rate-limited, customer-visible operation rather than pretending archive queries behave like cold-tier queries.

Direct Answer

The OpenTelemetry Collector is a pipeline with three stages: a receiver takes telemetry in (from an app via OTLP, or by scraping an endpoint), a processor chain transforms it in flight (batching, filtering, enriching), and an exporter sends it out to one or more backends. Deploy it as a per-host agent when you want telemetry to leave the host already batched and lightly processed; deploy it as a central gateway when you want heavier processing (like sampling decisions that need to see traffic from many sources at once) done in one place instead of duplicated on every host.

Structured Elaboration

Pipeline diagram

flowchart LR
  A["Instrumented App"] -->|"OTLP"| R["Receiver"]
  R --> P["Processor Chain: batch, filter, enrich"]
  P --> E["Exporter"]
  E --> B1[("Backend A")]
  E --> B2[("Backend B")]

Receiver
Accepts telemetry in a given format and protocol, most commonly the OTLP receiver (gRPC or HTTP) for data pushed from an instrumented application, or a Prometheus-style receiver that scrapes metrics endpoints on a schedule. A single collector can run multiple receivers at once.

Processor
Sits between receiver and exporter, transforming data without changing where it came from or where it's going. Common processors: batch (groups items to amortize per-request overhead, see the worked example), memory_limiter (sheds load before the process runs out of memory), attributes (adds, renames, or redacts fields), and sampling processors (drop a fraction of data by a configured rule).

Exporter
Sends processed telemetry to one or more backends in whatever protocol that backend expects (Prometheus remote-write, OTLP to a vendor, a file, and so on). A pipeline can fan out to multiple exporters from the same processed stream.

Agent versus gateway
An agent runs on or near every host (often as a DaemonSet), doing initial receipt and light processing close to the source. A gateway is a smaller number of centrally deployed instances that agents forward to, doing heavier fleet-wide processing (tail-based trace sampling needs to see many sources at once to decide, which an agent working alone cannot do) before the final export.

Worked Example

Consider sending spans without the batch processor, one OTLP request per span, versus with it, spans grouped into batches of 500 before a single request.

Assume each request carries a fixed protocol/framing overhead of roughly 1,024 bytes (headers, envelope) and each span payload is roughly 500 bytes (a stated design input, not a measured wire capture).

Without batching, one span per request:

Roughly two-thirds of every byte sent is framing overhead, not span data.

With batching, 500 spans per request:

The batch processor takes the overhead fraction from about 67% down to about 0.4%, which is the concrete reason batching sits in almost every collector pipeline by default rather than being an optional tuning step.

Trade-offs and Pitfalls

Running the collector as an agent everywhere is simple to reason about but limits what any single instance can see: it cannot make a decision (like tail-based trace sampling) that depends on data arriving at a different host. That kind of decision needs a gateway tier the agents forward to.

Too large a batch size trades throughput efficiency for latency: a batch that only flushes every few seconds under low traffic delays every piece of telemetry inside it by that long, even though the framing-overhead math above still favors big batches under high traffic. Most collector configurations flush on whichever comes first, a size threshold or a time threshold, exactly to avoid this under low load.

A processor chain's order matters and is easy to get wrong: a memory_limiter should run early (to shed load before spending CPU on later processors), and a batch processor should generally run late, right before the exporter, so it only batches data that already survived filtering or sampling decisions rather than wasting batch capacity on data about to be dropped.

Direct Answer

Pick the weakest delivery guarantee that still lets you reconstruct correctness at the aggregation layer, rather than paying the latency and complexity cost of strong guarantees everywhere. For most telemetry pipelines that means at-least-once delivery plus idempotent, deduplicated writes: true end-to-end exactly-once delivery across independent systems does not really exist, what you can build is a system that behaves as if it were exactly-once because duplicate deliveries have no effect on the result.

Structured Elaboration

At-most-once
A point may be silently lost, never duplicated. Cheapest and lowest-latency, but every dropped point is gone. Acceptable only for high-volume, low-value telemetry where a small, unbiased loss rate does not meaningfully change an aggregate.

At-least-once
No steady-state data loss, but retries can produce duplicates. This is the practical default for most producers (send, wait for ack, retry on timeout, occasionally re-deliver something that actually did arrive). Correctness at the aggregation layer requires deduplication: an idempotent write keyed by a unique identifier, so a duplicate delivery does not change the result.

Exactly-once
The goal is a metric aggregation that reflects each event's effect precisely once. In practice this is built, not given by any single component: it requires end-to-end idempotency (deterministic dedup keys), transactional or checkpointed writes (a write-ahead log tying the aggregate update and the consumer offset commit together atomically), and durable state. The cost is coordination overhead and added write latency.

Eventual consistency
Aggregates may be temporarily stale or incomplete, converging to correct only after all in-flight and late-arriving events settle. Handled with event-time windowing, watermarks that define how late an event can arrive before it's excluded, and a correction mechanism (recompute or emit a delta) for events that show up after their window closed.

Making a weaker guarantee correct in practice

• Deduplication IDs: a producer-assigned, monotonically increasing sequence number per stream. The consumer tracks a high-water mark per stream and discards anything at or below it.
• Idempotent writes: aggregation writes as upserts keyed by (metric, labels, timestamp window), so replaying the same input overwrites rather than accumulates.
• Write-ahead logs: the consumer writes its aggregate update and its new checkpoint offset in a single atomic step, then acks the upstream offset. On crash and restart, it replays only the un-checkpointed range, which is safe because that replay is itself idempotent.

Checkpoint and dedup flow

flowchart LR
  EVT["Event with sequence id"] --> DEDUP{"seq > high-water mark?"}
  DEDUP -->|"yes, new"| APPLY["Apply to Aggregate"]
  DEDUP -->|"no, duplicate"| DISCARD["Discard"]
  APPLY --> CKPT[("Atomic: Aggregate + Checkpoint")]
  CKPT --> ACK["Ack Offset Upstream"]

Worked Example

Assume 3 producers each sending 100 events into a 1-minute rollup window, for a correct total of 300 events. One producer's network connection times out on 2 of its events, so it retries both, and (because at-least-once delivery does not deduplicate at the transport layer) both the original and the retried copy arrive.

Without deduplication, the consumer counts:

A small number for one window, but a systematic bias that compounds across every window and every retried event at scale, not a one-time rounding error.

With deduplication (each producer assigns a monotonically increasing sequence number per stream, and the consumer tracks a high-water mark per producer, discarding anything at or below the mark it has already seen):

The dedup mechanism restores the exact correct count using only transport-layer at-least-once delivery, no exactly-once broker feature required.

Trade-offs and Pitfalls

Sequence-number-based dedup (a high-water mark per stream) only works correctly for pure retries, redelivery of the same sequence number. It breaks if the transport can reorder events beyond a small window, since a legitimately new event with a lower sequence number than one already seen would be incorrectly discarded as a duplicate. If your transport allows significant reordering, dedup by explicit unique event ID with a time-bounded lookup window instead of a strict high-water mark.

Exactly-once semantics inside a single system (say, a Kafka Streams application with transactional writes) do not extend automatically to an external sink. The moment you write to an external time-series database or object store, you are back to needing idempotent writes at that boundary, transactional guarantees inside the stream processor do not make the external write exactly-once by themselves.

Choosing at-least-once plus dedup as the default is right for the vast majority of telemetry, but not for every case: a billing-critical count (metered usage that directly drives a customer invoice) may warrant paying the extra coordination cost for stronger guarantees specifically at that boundary, while everything else stays on the cheaper default.

Time-series compression works because consecutive samples in a series are usually similar to each other: timestamps arrive at near-regular intervals, and values tend to change by small amounts (or not at all) between samples. Block-chunking, delta-of-delta timestamp encoding, XOR-based float compression, and label dictionary encoding each exploit a different piece of that similarity.

How each technique works

Block-chunking: samples are grouped into fixed time-window or fixed-count blocks (e.g., 2-hour windows), each with its own compressed byte stream and a small metadata header (start/end timestamp, min/max value). This lets a query engine skip whole blocks that fall outside a requested range without decompressing them, and it bounds how much has to be re-encoded when new data arrives (you only ever append to the current open block).

Delta-of-delta timestamp encoding: instead of storing each timestamp, store the delta from the previous timestamp, then the delta of that delta. For a series scraped at a fixed interval, the delta between consecutive timestamps is constant, so the delta-of-delta is exactly zero almost every time, which collapses to a single control bit per sample.

XOR-based float compression (Gorilla): XOR the current value's bit pattern against the previous value's bit pattern. If the value hasn't changed much, most of the leading bits (sign, exponent, high mantissa) and trailing bits (low mantissa) of the XOR result are zero, and only a short "meaningful" middle span needs to be stored explicitly.

Dictionary encoding for labels: label keys and values repeat across enormous numbers of series (env=prod appears on millions of series), so each distinct string is stored once in a dictionary and every series references it by a small integer ID instead of repeating the string.

Worked example: counter vs. sparse gauge

The following is executed Python (struct-level IEEE-754 bit manipulation, no external data) so the numbers are exactly reproducible:

import struct

def bits(f):
    return struct.unpack('>Q', struct.pack('>d', f))[0]

def leading_zeros(x, width=64):
    return width if x == 0 else width - x.bit_length()

def trailing_zeros(x, width=64):
    if x == 0: return width
    c = 0
    while (x & 1) == 0:
        x >>= 1; c += 1
    return c

def xor_meaningful_bits(prev, cur):
    x = bits(prev) ^ bits(cur)
    if x == 0: return 0, 64, 0
    lz, tz = leading_zeros(x), trailing_zeros(x)
    return lz, tz, 64 - lz - tz

# dense monotonic counter: http_requests_total, +1 per 15s scrape
print(xor_meaningful_bits(184320.0, 184321.0))
# sparse gauge, small fluctuation: temperature idling
print(xor_meaningful_bits(21.4, 21.9))
# sparse gauge, large jump: temperature spikes
print(xor_meaningful_bits(21.9, 87.2))

Output:

(28, 35, 1)   # counter +1 step: 1 meaningful bit
(16, 47, 1)   # gauge, small change: also 1 meaningful bit
(9, 0, 55)    # gauge, large jump: 55 meaningful bits (almost the full double)

And for delta-of-delta timestamps on a realistically jittered 15-second scrape, with the full input pinned so the result is exactly reproducible (10 explicit timestamps, which by the $D_i$ formula above yields 8 delta-of-delta values, since each output needs 3 consecutive timestamps):

timestamps = [
    1720000000, 1720000015, 1720000030, 1720000045, 1720000060,
    1720000075, 1720000089, 1720000105, 1720000120, 1720000134,
]
deltas = [timestamps[i] - timestamps[i - 1] for i in range(1, len(timestamps))]
delta_of_delta = [deltas[i] - deltas[i - 1] for i in range(1, len(deltas))]
print("deltas:", deltas)
print("delta-of-delta:", delta_of_delta)

Output:

deltas:          [15, 15, 15, 15, 15, 14, 16, 15, 14]
delta-of-delta:  [0, 0, 0, 0, -1, 2, -1, -1]

4/8 are exactly 0 (single control bit each): the first four scrape gaps land exactly on the 15-second interval, then jitter shows up as a 14s gap, a 16s gap, and two more gaps that miss the interval by one second, each producing a nonzero delta-of-delta.

Which techniques help most for which shape

The counter case shows why Gorilla-style encoding is the default for metrics generally: even though the value itself is always changing (climbing), the bit pattern delta between consecutive floats stays small as long as the relative step size is small, which is true for almost all real counters. The gauge case shows the failure mode: a big jump changes the exponent bits, which wipes out both the leading-zero and trailing-zero runs simultaneously, so the encoding degrades toward storing the value nearly raw.

Trade-offs and pitfalls

• A common wrong turn is assuming XOR compression is "for gauges" and delta-encoding is "for counters" as a rule; the real determinant is how much the bit pattern changes step to step, which correlates with relative magnitude change, not with the metric type label.
• Partial decompression for range queries requires the block-level index (min/max timestamp, min/max value) to be cheap to read without decoding the compressed body; if you skip that index to save space, every range query degrades to a full block scan.
• Block size is a real tuning knob, not a footnote: too small and per-block metadata overhead dominates; too large and a single noisy stretch (like the sparse-gauge jump) drags down the compression ratio for the whole block and increases decompression latency for small range queries.
• Dictionary encoding for labels needs a compaction/garbage-collection story; a dictionary that only grows (never reclaims IDs for labels that stop being used) becomes its own unbounded-cardinality problem over long retention.

Default to logical isolation (shared infrastructure with hard per-tenant quotas and QoS enforcement) for the bulk of tenants, and offer physical isolation (dedicated shards or node pools) as an explicit, metered upgrade path for tenants whose usage or SLA requirements outgrow what shared quotas can safely guarantee. The isolation model and the migration path are two sides of the same design.

Architecture

flowchart LR
  A[Tenant Requests] --> B[Ingress: Auth and RBAC]
  B --> C[Per-Tenant Shard / Queue]
  C --> D[Shared Ingestion Pool]
  C --> E[Dedicated Ingestion Pool]
  D --> F[Query Gateway: QoS Scheduler]
  E --> F
  F --> G[Shared Query Compute]
  F --> H[Dedicated Query Compute]
  I[Billing / Quota Manager] --> B
  I --> F

• Logical isolation: per-tenant partitions/queues on shared compute, enforced with token-bucket rate limits at ingress and query-time concurrency caps; cheapest, and sufficient for the majority of tenants whose usage is well within their quota most of the time.
• Physical isolation: dedicated shard, node pool, or account for a tenant; strongest guarantee, but the operational and cost overhead of running fully separate infrastructure per tenant doesn't scale to hundreds of tenants, so it has to be selective.
• Query-level QoS: priority classes (interactive dashboard queries vs. batch/backfill queries), per-tenant concurrency limits, and admission control that sheds low-priority load before it degrades everyone; this is what actually prevents a noisy tenant's expensive query from starving others on shared compute, since ingestion isolation alone doesn't protect the read path.
• Billing-aware quotas: map each tenant's plan tier to a concrete ingest-rate and query-concurrency quota; soft-limit warnings before hard throttling, and an explicit overdraft/pay-as-you-go path rather than a silent hard cutoff. Retention is part of the same per-tenant contract, not a platform-wide constant: a tenant's plan tier should set its own retention window (e.g., 7 days on a shared/basic tier vs. 90 days on a dedicated tier), enforced as tenant-scoped TTL policy in the storage layer so one tenant's longer retention SLA doesn't force everyone else to pay for the same window.

Sizing the admission-control headroom

The core quantitative question for logical isolation is: how much burst capacity can the shared pool actually absorb before a legitimate burst from one tenant risks starving others? Take a platform with total ingest capacity $C = 2{,}000{,}000$ samples/sec shared across $N = 500$ tenants, where baseline quotas are provisioned to consume a target fraction $u$ of total capacity (leaving headroom for bursts), and tenants are allowed to burst up to $m\times$ their baseline:

If a fraction $f$ of tenants burst simultaneously while the rest sit at baseline, total load must stay under capacity:

Substituting $\text{baseline}_{\text{tenant}} = uC/N$ and simplifying:

With $u = 0.6$ (provision baseline to consume 60% of capacity, leaving 40% headroom) and $m = 5$ (allow a 5x burst):

C, N, u, m = 2_000_000, 500, 0.6, 5
baseline = (u * C) / N          # 2,400 samples/sec/tenant
burst = m * baseline            # 12,000 samples/sec/tenant
f_max = (1/u - 1) / (m - 1)     # 0.1667
max_bursting = f_max * N        # 83.3 tenants

Result: baseline quota is 2,400 samples/sec/tenant, burst allowance is 12,000 samples/sec/tenant, and up to about 16.7% of tenants (roughly 83 of 500) can burst simultaneously at 5x without exceeding total capacity. Plugging $f_{max}$ back into the original inequality confirms it lands exactly at capacity (2,000,000 samples/sec), which is the check that the derivation is self-consistent. This is the number that should actually drive the admission controller's global burst budget, not a guess: if more than ~83 tenants try to burst at once, the controller has to start denying or queuing burst requests rather than granting them all.

Migrating a tenant from shared to dedicated

1. Trigger: sustained usage consistently near quota (not just occasional bursts), or an explicit SLA purchase requiring guaranteed isolation.
2. Provision dedicated shard/node pool ahead of cutover.
3. Dual-write or replicate the tenant's recent data into the new dedicated shard while it's still live on the shared pool.
4. Cut over routing at the control plane (ingress rules keyed on tenant ID) once the dedicated shard is caught up; this should be a routing change, not a data migration event, so it can be near-zero-downtime.
5. Decommission the tenant's shared-pool footprint after a verification window, and keep the cutover reversible in case the dedicated shard has an unexpected issue.

Trade-offs and pitfalls

• Sizing baseline quotas at $u$ close to 1.0 (using nearly all capacity for guaranteed baseline) leaves almost no burst headroom, which defeats the purpose of a shared pool; the $u$ vs. burst-headroom trade-off above should be an explicit, revisited decision, not a default.
• Query-level QoS is often skipped because ingestion isolation feels like "the isolation problem," but an expensive ad-hoc query from one tenant can degrade shared query compute even when every tenant's ingestion is perfectly isolated; both paths need protection independently.
• A migration path that isn't reversible (no fallback if the dedicated shard has a problem post-cutover) turns a capacity upgrade into a risk event; always keep the shared-pool footprint alive through a verification window.
• Billing-aware quotas without a clear soft-limit warning stage turn every quota breach into a support ticket; the graduated response (warn, throttle, then hard-limit) matters as much as the quota number itself.