Observability and Monitoring Architecture Questions

Direct Answer

Put uniform, low-cost collection (node and infra metrics, general logs) on a DaemonSet everywhere, including on the stateful nodes, since a flat per-node cost is easy to reason about and budget for. Reserve sidecars for signal that genuinely requires in-process, per-pod access a node agent cannot get (an application's internal connection pool state, for example), and be deliberate about NOT putting a sidecar on the database pods unless that condition is actually met. Anything needing cross-pod correlation or heavy processing (tail sampling, log parsing, deep query-stats collection) belongs off-node entirely, in a centralized service that polls or receives from the stateful workload remotely, so it never competes with the database for the node's own CPU and memory. This reasoning holds the same way on a managed control plane (EKS, GKE, AKS) as on self-managed Kubernetes, since it is about resource contention on the node, not about who runs the API server.

Structured Elaboration

Placement topology

flowchart LR
  subgraph STATELESS["Stateless Front-End Nodes"]
    FE["Front-end Pods"] --> DS1["DaemonSet Agent"]
  end
  subgraph STATEFUL["Stateful DB Nodes, capacity-constrained"]
    DB[("Database Pod")]
    DS2["DaemonSet Agent: node metrics only"]
  end
  DS1 --> GATEWAY["Central Collector Gateway"]
  DS2 --> GATEWAY
  POLLER["Off-node DB Metrics Poller"] -->|"remote scrape"| DB
  POLLER --> GATEWAY
  GATEWAY --> BACKEND[("Backend")]

Stateless front-end nodes
Lower stakes: DaemonSet for infra metrics and logs, sidecars are affordable here if a particular front-end service wants per-pod tracing detail, since these nodes typically run with more spare headroom.

Stateful database nodes
Higher stakes: these nodes are usually provisioned close to capacity by the database's own resource requests, leaving little headroom for anything else. The DaemonSet's flat, small, predictable overhead is safe here. A sidecar doing anything beyond trivial passthrough is risky, because its resource usage now directly competes with the database's own background work (vacuum, checkpointing, WAL flush) for the same constrained node.

What to centralize instead of running on the stateful node
Heavy or periodic collection against the database (query-stats dumps, slow-query log parsing) should run as an off-node poller hitting the database's metrics endpoint remotely. This adds a network hop and a small amount of latency to seeing that data, but adds zero incremental CPU or memory pressure on the node itself beyond the lightweight DaemonSet.

Worked Example

Assume a stateful node with 16 vCPU total, where the database pod's own resource request is 14 vCPU, leaving 2 vCPU (2,000m) of headroom for everything else on the node.

The baseline DaemonSet agent (node and infra metrics only) requests 200m CPU:

That is a safe, predictable cost. Now add a hypothetical sidecar that periodically runs a heavier in-process collection task, bursting to 500m CPU during its collection window:

A 35% claim on the database's only spare capacity, right when the sidecar's own collection burst happens to run, is exactly the kind of contention that can visibly delay the database's own background maintenance work. Moving that heavier collection off-node (a remote poller scraping the database's metrics endpoint) removes the 500m burst from the node entirely, leaving the node's overhead at the flat 10% DaemonSet baseline regardless of collection frequency.

Trade-offs and Pitfalls

The off-node poller trades node-local safety for a network dependency and slightly staler data (whatever the poll interval is, versus in-process real-time access). For a database, this is almost always the right trade: a few seconds of staleness on query-stats data is a much smaller risk than periodic CPU contention with the database engine itself.

Not every stateful workload has this much headroom to spare. If a database's own resource request already consumes 15.5 of 16 vCPU, even the flat DaemonSet overhead becomes a real constraint, and the answer shifts toward running collection on a dedicated node pool or accepting sparser, lower-frequency node-level metrics on those specific hosts rather than the fleet-wide default cadence.

A common mistake is applying the same sidecar-friendly policy used for stateless front-ends uniformly across the whole cluster, without re-evaluating it against the stateful nodes' actual spare capacity. The placement decision should be driven by measured headroom per node pool, not by a single fleet-wide policy applied blindly.

Direct answer

Emit a small, separate provenance record at the moment each sampling decision is made in the OpenTelemetry collector (which policy fired, its version, the score/trigger, a pointer to the raw trace), route it to its own indexed store rather than attaching it to the span itself, and keep the immutable policy-version history in a content-addressed snapshot store so any past decision can be tied back to the exact policy that produced it. Replay works by using the index to locate the original trace payload and re-running it through a resampling service against either the historical or current policy snapshot.

What to record per decision

{provenance_id, trace_id, span_root_id, policy_id, policy_version, score, trigger (rule id / deterministic / random seed), collector_id, timestamp, payload_pointer}. This is captured by a processor stage in the collector's sampling pipeline, which already has all of this context at decision time; nothing here requires re-deriving the decision after the fact.

Why provenance is a separate store from span data

Attaching this metadata directly onto every span as attributes would grow span size and, worse, several of these fields (policy_id, trigger, score) are exactly the kind of thing that turns into an unbounded-cardinality label if it ever gets promoted from a span attribute into a metric. Keeping provenance in its own indexed, purpose-built store avoids both problems: span size stays bounded, and the provenance store can be indexed on exactly the fields audits actually query (policy_id, time range, collector_id, trigger), independent of how spans themselves are indexed.

Storage layout

• Snapshot store: full policy definitions, content-addressed (hash of the policy config as its ID), append-only. A provenance record references a snapshot ID rather than duplicating the policy content.
• Index store: one row per decision, indexed on policy_id, timestamp, trace_id, collector_id, pointing to both the policy snapshot and the original trace payload location.
• Payload store: the actual trace data (sampled or not, if retained for replay purposes) in cheap object storage, referenced by pointer, not duplicated into the index.

Auditing and replay

• Auditing a policy's history over time is a query against the snapshot store's append-only log plus the index store filtered by policy_id; because snapshots are content-addressed and immutable, there's no ambiguity about what a given policy_version actually did at decision time.
• Replaying or resampling a historical trace: look up its provenance record by trace_id, fetch the referenced payload and policy snapshot, and re-run it through a resampling service using either the original snapshot (to reproduce the original decision exactly) or the current policy (to ask "would this decision be different today").

Worked example

Assume a JSON-encoded provenance record is roughly $300$ bytes (field names plus values; a smaller binary encoding would shrink this further, but JSON is used here for a conservative, worst-case estimate). Assume the fleet is producing $200{,}000$ trace-eligible requests/sec and applying a $2\%$ sampling rate, so decision records are only written for sampled traces (unsampled traffic is covered by an aggregate counter, not a per-trace record, to keep volume bounded):

For a 1-year audit-retention requirement:

Applying a conservative $5{:}1$ compression ratio for this repetitive, structured JSON (typical for columnar or general-purpose compression on highly repetitive records):

This confirms the design instinct in the hints: 3+ TB/month of provenance metadata alone is large enough that it must be a dedicated, purpose-indexed store, not an afterthought bolted onto span storage, and the compression step matters enough to be worth the CPU cost at this volume.

flowchart LR
  Collector[OTel Sampling Processor] --> DecisionEmit[Emit Provenance Record]
  DecisionEmit --> Stream[Provenance Event Stream]
  Stream --> IndexStore[Indexed Metadata Store]
  Stream --> SnapshotStore[Policy Snapshot Store]
  IndexStore --> AuditAPI[Audit Query API]
  SnapshotStore --> AuditAPI
  AuditAPI --> Replay[Replay / Resample Service]
  Replay --> PayloadStore[Trace Payload Object Store]

Trade-offs and pitfalls

• Recording a full per-trace provenance record for every trace, sampled or not, would multiply the volume above by roughly $1/0.02 = 50\times$; capping the per-trace record to sampled traces only, and covering unsampled volume with an aggregate counter, is what keeps this tractable.
• Content-addressing policy snapshots (rather than a mutable "current policy" pointer) is what makes historical audit queries trustworthy; a mutable policy record would make it impossible to prove what a decision from six months ago was actually based on.
• Replay accuracy depends entirely on the payload store retaining the original trace data long enough to replay it; if trace payloads expire (per the operational retention/tiering policy) before the provenance audit-retention window does, replay for older decisions becomes impossible even though the provenance record itself still exists. The two retention windows need to be reconciled explicitly, not assumed to match.
• A JSON-based index store is simple to query ad hoc but the 300-byte/record estimate is generous; a tighter binary encoding could meaningfully cut both storage and index cost at this volume, and is worth revisiting once the audit query patterns are well understood.

Direct answer

Fix this at the architecture level, not by throwing more compute at the same query plan. Build a layer of materialized views (precomputed, stored query results that refresh as new data lands, instead of being recomputed from scratch on every request) that pre-aggregate the group-bys and time windows dashboards actually use, add a query rewriter (a planner step that intercepts an incoming query and swaps it for a cheaper, equivalent one) that transparently substitutes a matching materialized view for the raw scan whenever the rewrite preserves aggregate semantics, and cache the result of the highest-frequency top-K queries with a TTL tied to the refresh cadence. Decide what to precompute with a cost-benefit rule: materialize a query shape when the daily rows it saves scanning outweighs the daily rows its incremental refresh costs to maintain, not by intuition about which dashboards "feel slow."

Structured elaboration

Materialized views (MVs): store pre-grouped, pre-aggregated rows keyed by the dimensions a panel actually displays (for example, top-20 services by error rate), refreshed incrementally as new raw data lands, not recomputed from scratch each time.

Pre-aggregation windows: keep multiple resolutions so a query can pick the coarsest one that still covers its time range: 1-minute rollups for the last hour, 5-minute for the last day, 1-hour for the last month. This bounds how many rows any query has to touch regardless of the underlying cardinality.

Partitioning (the piece the naive scan is missing): partition the raw and rollup tables by time first, then by a bounded set of high-selectivity dimensions (service, region). This lets both the raw fallback path and the MV refresh job skip whole partitions instead of scanning the full high-cardinality series space, which is what turns a query over "recent, high-cardinality metrics" from a full-table scan into a bounded one.

Query rewriting: a planner stage intercepts the incoming query, checks whether its group-by, filter, and time window are a subset of an existing MV's coverage, and rewrites the query to read the MV instead of raw data. Only rewrite when the operation is safe (sums, counts, min/max compose across MVs cleanly; distinct counts and unbounded percentiles generally do not without a sketch-based MV, which needs its own merge logic).

Caching top-K queries: cache the actual result set for the highest-frequency parameterized queries (dashboard panel + time range), keyed by a hash of the query shape and the current rollup epoch, invalidated on the next refresh rather than on a wall-clock timer.

flowchart LR
    Q[Incoming dashboard query] --> P{Query rewriter}
    P -- matches an MV --> MV[(Materialized view / rollup)]
    P -- no safe match --> RAW[(Partitioned raw store)]
    MV --> CACHE{Top-K result cache}
    CACHE -- hit --> R[Response]
    CACHE -- miss --> R
    RAW --> R
    ING[Streaming ingest] -- incremental refresh --> MV

Selection criterion, precisely: precompute a query shape when

where $f$ is how often that shape is queried per day. This is the freshness-versus-speed trade-off made concrete: the left side is what you save on reads, the right side is what you pay to keep the view fresh.

Worked example

A dashboard panel groups by service across $K = 100{,}000$ series matched by its filter, over a $W = 3600$s (1-hour) window, at a 15s scrape interval.

Without a materialized view, every execution scans one row per series per scrape tick in the window:

With a materialized view that stores 1-minute rollups already grouped down to the top 20 series the panel displays:

At $f = 500$ views/day for this panel, the daily rows saved by reading the MV instead of raw:

The MV refreshes incrementally every minute (1,440 refreshes/day), and each refresh only has to process the new minute's raw rows for the panel's series ($K \times 4$ samples/minute):

Savings exceed maintenance cost by roughly 20.8x here, so this panel clears the bar comfortably. Solving the selection inequality for the breakeven view frequency:

So the concrete criterion for this panel shape is: materialize it once it's viewed more than roughly 24 times a day; below that, the refresh overhead isn't earning its keep and the raw fallback path is cheaper.

Trade-offs & pitfalls

Combining all three, with the rewriter deciding per-query which to use, is what actually removes the timeout; any one alone leaves a gap.

Common wrong turns: materializing every group-by combination a dashboard could theoretically ask for (storage and refresh cost grow combinatorially, and most of those shapes are never queried, i.e. $f \approx 0$, which fails the selection criterion outright); rewriting queries onto an MV whose aggregation isn't actually composable for the requested operation (silently wrong distinct-counts or percentiles are worse than a slow correct answer); and caching on a fixed TTL instead of the rollup epoch, which either serves stale data past a refresh or invalidates a cache entry that's still perfectly valid.

The core design decision is a two-level key: shard by a hash of the series identity so writes and label lookups distribute evenly, and chunk by time within each series so both compression and range queries stay efficient. Everything else (indexing, downsampling, compression) hangs off that.

Data model

Each sample is (metric_name, labels, timestamp, value). In practice you don't store metric_name and labels as free strings per sample; you compute a SeriesID once per unique combination:

and every sample after that is just (SeriesID, timestamp, value). The label set is stored once in a separate series-metadata record, not repeated per sample.

Partitioning and chunking

flowchart LR
  A[Sample Ingest] --> B[Hash by SeriesID]
  B --> C[Shard 1]
  B --> D[Shard 2]
  B --> E[Shard N]
  C --> F[Head Block: in-memory chunk]
  F --> G[Flush to Durable Chunk]
  G --> H[Inverted Label Index]
  G --> I[Block Storage]
  H --> J[Query: label lookup]
  I --> J

• Partition key: (time_window, shard_id) where shard_id = hash(SeriesID) mod N. Time-first partitioning means old partitions become immutable and can be compacted/downsampled/expired independently; sharding by SeriesID hash inside each time window spreads a hot series's neighbors across nodes instead of colocating them.
• Chunking: each series appends to an in-memory "head" chunk, flushed to a durable, compressed chunk when it hits either a time bound (e.g., 2 hours) or a size bound. Chunk metadata (SeriesID, start/end timestamp, min/max value) lets a query prune whole chunks without decompressing them.
• Compression: delta-of-delta for timestamps, XOR (Gorilla-style) for values within a chunk, general-purpose compression (LZ4/Snappy) over the chunk byte stream, dictionary-encoded label strings referenced by ID.
• Index structures: a primary LSM-style index maps SeriesID -> chunk pointers for fast range scans of one series; a secondary inverted index maps label_key=value -> [SeriesIDs], typically stored as compressed bitmaps (e.g., Roaring bitmaps) so a query like {job="checkout", env="prod"} becomes a bitmap intersection instead of a full scan.

Sizing the design against a concrete workload

Take 10,000,000 active series, an average of 6 label pairs per series (beyond the metric name), a 2-hour flush window at 15-second scrape interval, and a target of at most 2,000,000 active series per shard for balanced load:

active_series = 10_000_000
avg_labels_per_series = 6
bytes_per_posting_entry = 3          # Roaring-bitmap-compressed 32-bit series ID, moderately dense postings (assumption)
total_postings_entries = active_series * avg_labels_per_series
inverted_index_bytes = total_postings_entries * bytes_per_posting_entry

chunk_window_s = 2 * 3600
scrape_interval_s = 15
samples_per_series_per_chunk = chunk_window_s / scrape_interval_s
compressed_bytes_per_sample = 2      # consistent with the Gorilla-style bit math above, rounded up for a mixed workload (assumption)
head_bytes_per_series = samples_per_series_per_chunk * compressed_bytes_per_sample
head_total_bytes = active_series * head_bytes_per_series

target_series_per_shard = 2_000_000
min_shards = active_series / target_series_per_shard

Result: total_postings_entries = 60,000,000, inverted_index_bytes ≈ 180 MB, samples_per_series_per_chunk = 480, head_bytes_per_series = 960, head_total_bytes ≈ 9.6 GB, min_shards = 5.0 (round up to 8 for power-of-2 hash routing headroom).

That tells you two concrete things: the in-memory working set for the head block across all shards (~9.6 GB) comfortably fits on modern hardware split across 8 shards (about 1.2 GB/shard), and the inverted index itself (~180 MB) is small relative to the chunk data, meaning label-lookup cost is dominated by bitmap intersection speed, not index size. If active series grew 10x to 100M, head memory would grow to ~96 GB total, which is the point where you'd need to either shrink the chunk window (trading write amplification for lower per-shard memory) or add more shards.

Trade-offs

Pitfalls

• Choosing a partition key that's purely hash-based (no time component) makes retention expensive: you can't just drop a partition, you have to scan and delete, which is the mistake most designs make when they optimize only for write balance and forget that data has to expire.
• Sizing the head-block window without doing the arithmetic above (as many designs do) leads to either restart storms (window too large, recovery replays too much) or excessive flush overhead (window too small).
• An inverted index without a query-time cardinality guard turns a single broad label predicate (like an unbounded regex) into a full index scan across all shards; that needs a query-side cost limit independent of the storage design.

Direct answer

Put a durable, replicated buffer (a log like Kafka, not an in-memory queue) between collectors and processors so a failed component can be detected and rerouted around without losing the data that was in flight. Track three pieces of state to do this safely: per-partition consumer offsets (so replay resumes from the right place, not from zero), per-component health signals (lag, error rate, heartbeat) that drive rerouting decisions, and an idempotency key on every event so a replay after a failure doesn't get double-counted downstream. What stops the remediation logic itself from cascading is a hard budget on how many auto-remediation actions any one component can trigger in a window, with a circuit breaker that pages a human once that budget is exhausted instead of retrying forever.

Structured elaboration

State to track:

• Consumer offsets per partition per consumer group, checkpointed durably, so a restarted processor resumes exactly where it left off.
• Health signals per component: consumer lag, processing error rate, heartbeat/liveness. These feed the detection logic, not just an external dashboard.
• Idempotency keys on events (a stable ID derived from source + sequence number, not regenerated on replay), so downstream stores can deduplicate when the same event is delivered twice after a replay.

Detection and reroute:

• Liveness and readiness probes remove an unhealthy processor from its consumer group; the group rebalances so the partitions it owned are picked up by healthy consumers.
• Rising consumer lag or error rate on a partition, even without an outright liveness failure, is itself a detection signal (this catches degraded-but-technically-alive processors, not just crashed ones).
• Collectors buffer locally and retry with backoff when they can't reach a processor, rather than dropping data immediately.

Replay:

• On restart, a processor resumes from its last committed offset; the durable log's retention window bounds how far back replay can reach.
• For a deliberate, controlled replay (not just crash recovery), a replay controller can reset a consumer group's offset to an earlier point and replay into an isolated processing path first, so a bad replay doesn't double-write into the live output the way replaying directly into the primary consumer group would.
• Stateful processors restore their state from a changelog or external state store before resuming, so replay doesn't start from a stale in-memory state.

Autoscaling: scale on consumer lag and error rate together, not CPU alone. A processor that's CPU-idle but falling behind (for example, downstream I/O-bound) needs more replicas even though CPU utilization alone wouldn't trigger a scale-up.

Degraded-mode operation: when the pipeline itself is under load it can't fully process (for example a partial outage reducing available processor capacity), fail toward reduced fidelity rather than data loss: temporarily increase sampling (drop a configured fraction of lower-priority telemetry) to keep the pipeline within capacity while preserving full-fidelity handling for high-priority signals, then return to full sampling once capacity recovers.

A specific, common failure mode worth naming: a collector with a slow memory leak. This doesn't trip a liveness check the way a crash does; it degrades gradually. The remediation here isn't "restart on OOM" (that's just a symptom-fix that fires only after the leak has already caused damage), it's tracking a per-collector memory trend and proactively rotating (drain, restart) a collector whose memory is trending toward its limit before it OOMs and drops whatever was buffered in-process at the time.

flowchart LR
    C[Collector] -- push --> BUF[(Durable log: partitioned, replicated)]
    BUF --> P1[Processor pod]
    P1 -- heartbeat/lag --> HM[Health manager]
    HM -- unhealthy: rebalance --> BUF
    HM -- budget exceeded --> CB[Circuit breaker: page human]
    HM -- lag rising --> AS[Autoscaler]
    AS -- scale replicas --> P1
    RC[Replay controller] -- controlled replay --> BUF
    RC --> ISO[Isolated replay path]

Worked example

Sizing the durable buffer to survive an outage. Assume the pipeline sustains $r = 200{,}000$ events/sec at $300$ bytes/event, and the target is to absorb up to a 15-minute processor outage without dropping data (detect + reroute + recover within that window):

(The 1.5x factor is headroom for more than one component being degraded at once, not just a single clean outage.) With replication factor 3 for durability:

That's the concrete number that goes into sizing the durable log's disk footprint for a 15-minute recovery SLO at this ingestion rate: roughly a quarter-terabyte of replicated buffer, not an arbitrary "keep some retention" guess.

Bounding the remediation loop so it can't cascade. Use a token-bucket budget per component: capacity 3 auto-remediation actions (for example, restarts), refilling 1 token every 10 minutes. In the worst case, a flapping component can trigger:

before the bucket is empty and the circuit breaker opens, escalating to a human instead of continuing to retry. Pair this with capped exponential backoff on reroute retries (base 2s, doubling, capped at 60s):

The cap matters here specifically: without it, a component that's actually gone for good would have retry intervals growing unbounded, delaying the eventual "stop retrying, alert a human" decision far longer than necessary.

Trade-offs & pitfalls

Common wrong turns: replaying directly into the live consumer group after a failure instead of an isolated path first, which can double-process events downstream if the failure that triggered the replay wasn't a clean crash (partial writes already landed); sizing the durable buffer for the average outage duration instead of the target recovery SLO, which quietly fails during the outages that matter most; and building auto-remediation without a hard action budget, on the assumption that "it can only help," when a bad remediation action (for example restarting a processor whose real problem is a poison-pill message) can itself be the thing that turns a single-partition issue into a rebalancing storm across the whole consumer group.