Observability and Monitoring Architecture Questions
Building visibility into infrastructure and services: metrics, logs, and traces, dashboards and alerting, SLIs/SLOs, and the design of an observability stack. Covers instrumenting systems for actionable signal, reducing alert noise, and diagnosing production issues from telemetry. Infrastructure-wide observability, distinct from network-specific monitoring.
Compare Elasticsearch, ClickHouse, and an object-storage-plus-index approach as the backend for log storage at scale. Discuss search performance for ad-hoc queries, ingestion throughput, cost at long retention, schema flexibility, and operational overhead for each.
Sample Answer
For long-retention log storage, the honest default is a tiered combination: ClickHouse (or an object-storage-plus-index approach) for the bulk of retained volume, with Elasticsearch reserved for the recent window where full-text ad-hoc search actually matters. Picking Elasticsearch as the sole backend for everything gets expensive fast at long retention; picking object-storage-only sacrifices interactive search on recent data.
What each backend is actually optimized for
flowchart LR
L[Log Producers] --> P[Ingestion Pipeline]
P --> ES[Elasticsearch: inverted index + shards]
P --> CH[ClickHouse: columnar table]
P --> OSS[Object Storage: Parquet blocks]
OSS --> IDX[Secondary Index]
ES --> Q[Ad-hoc Query]
CH --> Q
IDX --> Q
| Dimension | Elasticsearch | ClickHouse | Object storage + index |
|---|---|---|---|
| Ad-hoc search performance | Best: inverted index gives sub-second full-text and fuzzy search, native relevance scoring | Good for structured/columnar filters and aggregations, weaker for free-text (needs token tables or a bolted-on search layer) | Weakest without a fresh index; latency depends on how current the index is and whether it covers the field you need |
| Ingestion throughput | Moderate; bulk API helps but refresh interval, replication, and inverted-index build cost CPU per document | High: columnar writes, batched inserts, background merges are cheap per row | Highest for raw storage (append/upload is nearly unbounded); the indexing pipeline, not the storage, is the bottleneck |
| Cost at long retention | Highest: replication plus inverted-index/doc-values overhead multiplies stored bytes well beyond raw | Low: strong columnar compression | Lowest: cheapest byte-for-byte, and you can defer indexing entirely for cold data |
| Schema flexibility | Schema-on-write with dynamic mapping; flexible for JSON but mapping explosions are a real failure mode | Schema-on-write, columnar; wide/nested tables work but schema evolution needs explicit migration | Most flexible: store raw JSON as objects, apply schema at query time |
| Operational overhead | High: shard sizing, heap/GC tuning, index lifecycle management | Moderate: merge/partition tuning, but fewer distinct failure modes than an ES cluster at the same scale | Low for the storage tier itself; complexity shifts to whatever indexing/query-engine pipeline you build on top |
Worked cost comparison at 30-day retention
Take 50,000 log events/sec, averaging 1 KB raw per event, retained 30 days:
raw bytes=50,000×1,000×86,400×30=1.296×1014 bytes=129.6 TBFor Elasticsearch, assume a replication factor of 2 (one primary plus one replica, the normal HA baseline) and an inverted-index/doc-values overhead multiplier of roughly 1.3x over the raw JSON (a labeled assumption, not a measured constant, since it depends on mapping and field count):
ES stored=129.6×2×1.3=336.96 TBFor ClickHouse, assume an 8x columnar compression ratio versus raw JSON (a labeled assumption; typical for structured log fields with LZ4/ZSTD codecs) and the same replication factor of 2 for HA:
CH stored=8129.6×2=32.4 TBFor object storage plus index, assume 10x compression from columnar Parquet with ZSTD (labeled assumption) and no replication multiplier (the object store's own erasure coding already provides durability), plus a 5% secondary-index overhead:
OSS+IDX stored=10129.6×1.05=13.61 TBSo at this volume and retention, Elasticsearch stores roughly 10.4x more data than ClickHouse and about 24.8x more than object-storage-plus-index for the identical raw input, purely from replication and index overhead, before you've even paid for the compute to keep those indices hot.
Recommendation and trade-offs
- Use Elasticsearch for a short hot window (days, not months) where engineers are actively doing free-text incident search; keep index count and field mapping disciplined to avoid mapping explosions.
- Use ClickHouse (or a managed equivalent) as the default long-retention backend when most queries are structured filters plus aggregation (by service, level, status code) rather than free-text search; this is the common case for "how many 5xxs did service X throw last Tuesday."
- Use object-storage-plus-index only when queries are rare, tolerate higher latency, and you want the lowest possible cost floor for compliance-driven retention (e.g., 1-year audit logs nobody expects to search interactively).
- The trap senior candidates catch and juniors miss: this isn't a single-backend decision. A hot/warm/cold tiering strategy (ES or a fast index for the last few days, ClickHouse for weeks to months, object storage for the long tail) captures the strengths of each without paying any one system's worst-case cost or latency for the whole retention window.
- Schema flexibility cuts both ways: ES's dynamic mapping feels convenient until an unbounded field (a stray high-cardinality tag) blows up your mapping and cluster health; object storage defers that cost to query time, which is safer for ingestion but slower for discovery.
Define SLOs for the observability pipeline itself: ingestion availability, storage durability, query freshness, and end-to-end latency for a dashboard query. For each one, propose a concrete target, how you'd measure it, and what action fires when it's breached.
Sample Answer
Direct answer
Define one SLI (service level indicator, the measured signal used to judge whether a promise is being kept) per user-facing promise the observability pipeline makes: ingestion availability (did the write succeed), storage durability (did an acknowledged write survive), query freshness (how long until an ingested event is queryable), and end-to-end query latency (how fast a dashboard panel renders). Set a concrete target and measurement method for each, and wire alerting to error-budget burn rate rather than a raw threshold, so a brief severe outage and a sustained mild degradation both page appropriately instead of only one of them tripping an alert.
Structured elaboration
For each SLI: target, how it's measured, and what fires on breach.
| SLI | Target (30-day SLO) | Measurement | Breach action |
|---|---|---|---|
| Ingestion availability | 99.9% of write requests accepted | Synthetic canary producers every 10s + real gateway accept/reject counters | Auto-scale ingestion frontends; page if burn rate crosses fast-window threshold |
| Storage durability | 99.999% of acknowledged writes recoverable | Canary writes read back on a schedule; replication-lag monitoring | Fail over reads to a healthy replica; escalate if canary reads fail |
| Query freshness | p99 within 5 minutes of event time | A freshness consumer that tracks time-to-visible per event ID | Scale indexing workers; apply ingest backpressure to protect freshness for higher-priority signals |
| End-to-end query latency | p95 ≤ 500ms for canonical dashboard queries | Synthetic dashboard runs every minute against the canonical query set + real user latency traces | Enable/expand query result caching; shed non-critical query load |
Why error-budget burn rate, not a flat threshold: a flat "alert if availability drops below 99.9% this hour" either pages on noise (one bad minute in an otherwise fine hour) or misses a slow, sustained degradation that never dips low enough in any single short window to trip the threshold, while still burning through the monthly budget. A burn-rate alert asks a different question: "at this rate, how much of the month's error budget will this consume, and how fast?"
flowchart LR
G[Ingestion gateway] -- accept/reject --> M1[Availability SLI]
ST[(Storage)] -- canary read-back --> M2[Durability SLI]
IDX[Indexer] -- time-to-visible --> M3[Freshness SLI]
QE[Query engine] -- synthetic + real latency --> M4[Latency SLI]
M1 & M2 & M3 & M4 --> BR{Burn-rate calc}
BR -- fast window, high burn --> PAGE[Page on-call]
BR -- slow window, moderate burn --> TICKET[Ticket, review]
Worked example
Error budget for the ingestion availability SLO. At 99.9% over a 30-day period:
allowedErrorRate=1−0.999=0.001 errorBudgetMinutes=0.001×(30×24×60)=0.001×43,200=43.2 minutes/monthDeriving the multi-window burn-rate thresholds from first principles. A burn rate of b means the pipeline is failing b times faster than the allowed rate. If we want to alert fast when a burn rate would consume some fraction of the full 30-day (720-hour) budget within a short window:
burnRateThreshold=fractionOfBudget×windowHoursperiodHoursFor a fast window (page if 2% of the monthly budget would burn in 1 hour):
bfast=0.02×1720=14.4For a slower window (ticket if 5% would burn in 6 hours):
bslow=0.05×6720=6.0Applying it to an actual incident. Suppose the ingestion gateway starts rejecting 5% of writes (an actual observed error rate of 0.05 against an allowed rate of 0.001):
observedBurnRate=0.0010.05=50×Since 50>14.4, this immediately trips the fast-window page. It also tells you exactly how much runway remains before the entire month's error budget is gone at this rate:
hoursToExhaustBudget=50720=14.4 hoursThat's the number that should drive urgency in the room: at a 5% failure rate, this SLO's entire 30-day error budget is gone in 14.4 hours if nothing changes, which is why it pages immediately rather than waiting for a longer observation window.
Trade-offs & pitfalls
Common wrong turns: setting every SLI's target based on what current performance already achieves rather than what downstream consumers actually need (an SLO that's already being met with zero margin gives you no error budget to spend on legitimate risk, like a risky deploy); alerting on a single-window threshold instead of the fast/slow burn-rate pair, which either misses slow-burn incidents or pages too eagerly on noise; and treating "storage durability" and "ingestion availability" as the same SLI, when a write can be accepted (available) and still be lost before it's durable, so they need independent measurement even though they're both about "did the write succeed" in casual conversation. Also worth stating plainly in the room: these SLOs are for the observability platform's own reliability, which is a different (and often tighter) bar than the SLOs of the services it's monitoring, since a degraded observability platform blinds you during exactly the incidents where you need it most.
Design a self-healing telemetry ingestion pipeline: it should detect a failed collector or processor, reroute telemetry to a healthy instance, replay buffered data after a failure, and auto-scale under load, all while exposing its own health so platform engineers can tell when the observability system itself is degraded. What state would you need to track to do this safely, and what stops the remediation logic itself from causing a cascading failure?
Sample Answer
Direct answer
Put a durable, replicated buffer (a log like Kafka, not an in-memory queue) between collectors and processors so a failed component can be detected and rerouted around without losing the data that was in flight. Track three pieces of state to do this safely: per-partition consumer offsets (so replay resumes from the right place, not from zero), per-component health signals (lag, error rate, heartbeat) that drive rerouting decisions, and an idempotency key on every event so a replay after a failure doesn't get double-counted downstream. What stops the remediation logic itself from cascading is a hard budget on how many auto-remediation actions any one component can trigger in a window, with a circuit breaker that pages a human once that budget is exhausted instead of retrying forever.
Structured elaboration
State to track:
- Consumer offsets per partition per consumer group, checkpointed durably, so a restarted processor resumes exactly where it left off.
- Health signals per component: consumer lag, processing error rate, heartbeat/liveness. These feed the detection logic, not just an external dashboard.
- Idempotency keys on events (a stable ID derived from source + sequence number, not regenerated on replay), so downstream stores can deduplicate when the same event is delivered twice after a replay.
Detection and reroute:
- Liveness and readiness probes remove an unhealthy processor from its consumer group; the group rebalances so the partitions it owned are picked up by healthy consumers.
- Rising consumer lag or error rate on a partition, even without an outright liveness failure, is itself a detection signal (this catches degraded-but-technically-alive processors, not just crashed ones).
- Collectors buffer locally and retry with backoff when they can't reach a processor, rather than dropping data immediately.
Replay:
- On restart, a processor resumes from its last committed offset; the durable log's retention window bounds how far back replay can reach.
- For a deliberate, controlled replay (not just crash recovery), a replay controller can reset a consumer group's offset to an earlier point and replay into an isolated processing path first, so a bad replay doesn't double-write into the live output the way replaying directly into the primary consumer group would.
- Stateful processors restore their state from a changelog or external state store before resuming, so replay doesn't start from a stale in-memory state.
Autoscaling: scale on consumer lag and error rate together, not CPU alone. A processor that's CPU-idle but falling behind (for example, downstream I/O-bound) needs more replicas even though CPU utilization alone wouldn't trigger a scale-up.
Degraded-mode operation: when the pipeline itself is under load it can't fully process (for example a partial outage reducing available processor capacity), fail toward reduced fidelity rather than data loss: temporarily increase sampling (drop a configured fraction of lower-priority telemetry) to keep the pipeline within capacity while preserving full-fidelity handling for high-priority signals, then return to full sampling once capacity recovers.
A specific, common failure mode worth naming: a collector with a slow memory leak. This doesn't trip a liveness check the way a crash does; it degrades gradually. The remediation here isn't "restart on OOM" (that's just a symptom-fix that fires only after the leak has already caused damage), it's tracking a per-collector memory trend and proactively rotating (drain, restart) a collector whose memory is trending toward its limit before it OOMs and drops whatever was buffered in-process at the time.
flowchart LR
C[Collector] -- push --> BUF[(Durable log: partitioned, replicated)]
BUF --> P1[Processor pod]
P1 -- heartbeat/lag --> HM[Health manager]
HM -- unhealthy: rebalance --> BUF
HM -- budget exceeded --> CB[Circuit breaker: page human]
HM -- lag rising --> AS[Autoscaler]
AS -- scale replicas --> P1
RC[Replay controller] -- controlled replay --> BUF
RC --> ISO[Isolated replay path]
Worked example
Sizing the durable buffer to survive an outage. Assume the pipeline sustains r=200,000 events/sec at 300 bytes/event, and the target is to absorb up to a 15-minute processor outage without dropping data (detect + reroute + recover within that window):
bufferBytesNeeded=r×(15×60)×300×1.5=200,000×900×300×1.5=81 GB(The 1.5x factor is headroom for more than one component being degraded at once, not just a single clean outage.) With replication factor 3 for durability:
bufferBytesWithRF=81×3=243 GBThat's the concrete number that goes into sizing the durable log's disk footprint for a 15-minute recovery SLO at this ingestion rate: roughly a quarter-terabyte of replicated buffer, not an arbitrary "keep some retention" guess.
Bounding the remediation loop so it can't cascade. Use a token-bucket budget per component: capacity 3 auto-remediation actions (for example, restarts), refilling 1 token every 10 minutes. In the worst case, a flapping component can trigger:
worstCaseActionsPerHour=3+⌊1060⌋=3+6=9 actions/hourbefore the bucket is empty and the circuit breaker opens, escalating to a human instead of continuing to retry. Pair this with capped exponential backoff on reroute retries (base 2s, doubling, capped at 60s):
2,4,8,16,32,60,60,60(sum=242s before giving up on a single reroute attempt)The cap matters here specifically: without it, a component that's actually gone for good would have retry intervals growing unbounded, delaying the eventual "stop retrying, alert a human" decision far longer than necessary.
Trade-offs & pitfalls
| Design choice | Prevents | Costs |
|---|---|---|
| Durable buffer with 15-min absorption (243GB, RF=3) | Data loss during outages up to the target window | Disk and replication cost scale with the target outage window; longer targets get expensive fast |
| Remediation token bucket (9 actions/hr worst case) | Cascading restart storms | A genuinely flapping component still gets 9 restart attempts before paging, which is 9 chances to make things worse if the restart itself is the problem |
| Degraded-mode sampling | Total pipeline overload during partial capacity loss | Silently dropping lower-priority telemetry is only safe if "lower-priority" was actually decided in advance, not improvised during the incident |
Common wrong turns: replaying directly into the live consumer group after a failure instead of an isolated path first, which can double-process events downstream if the failure that triggered the replay wasn't a clean crash (partial writes already landed); sizing the durable buffer for the average outage duration instead of the target recovery SLO, which quietly fails during the outages that matter most; and building auto-remediation without a hard action budget, on the assumption that "it can only help," when a bad remediation action (for example restarting a processor whose real problem is a poison-pill message) can itself be the thing that turns a single-partition issue into a rebalancing storm across the whole consumer group.
Design a DaemonSet-based collection agent for a shared Kubernetes cluster: it needs to gather logs, metrics, and traces from every node, handle backpressure gracefully, support dynamic configuration (for example via CRDs), and remain safe to upgrade without dropping telemetry. What would you build in for multi-tenant isolation and failure handling?
Sample Answer
Direct answer
Run one agent pod per node via a DaemonSet, with a small in-memory ring buffer for burst absorption and a bounded on-disk queue for surviving a network partition, both sized from the node's actual telemetry volume rather than a guess. Handle dynamic configuration through a CRD that a controller validates and pushes to agents for hot-reload (no pod restart, no dropped telemetry mid-reload). For multi-tenant isolation, tag every event with the owning tenant at collection time, enforce per-tenant rate limits so one noisy tenant can't starve another's telemetry, and never let two tenants' data share an in-memory buffer.
Structured elaboration
Buffering and backpressure: an in-memory ring buffer absorbs short bursts (for example a log storm from a crash-looping pod); once it's full, the agent spills to a bounded on-disk queue rather than blocking the node's workloads or dropping data outright. A token-bucket rate limiter per tenant caps how fast any one tenant's telemetry can consume shared agent resources, with system/infra telemetry prioritized over tenant traffic when the two compete.
Dynamic configuration via CRDs: a cluster-level controller watches TenantConfig and AgentPolicy custom resources, validates them, and pushes signed configuration to each agent (via a local endpoint on the node, not a full pod restart). Agents apply the new config without dropping in-flight telemetry, which is the actual requirement behind "dynamic configuration" here: a config change that requires a restart isn't dynamic, it's just a faster redeploy.
Multi-tenant isolation:
- Data: every collected event is tagged with tenant identity at the point of collection; per-tenant buffers, not a shared one, so one tenant's backpressure can't block another's.
- Compute/resource: standard Kubernetes resource requests/limits on the DaemonSet pod bound how much CPU/memory the agent itself can consume on a shared node.
- Network: network policies restrict the agent's egress to only the tenant-authorized destinations, and short-lived, tenant-scoped credentials (issued by the controller) authenticate the agent's connection to each tenant's ingest endpoint.
Safe upgrades: rolling DaemonSet update with maxUnavailable bounded per zone, and a preStop hook that drains the on-disk queue (flushes buffered telemetry) before the old pod terminates, so an upgrade doesn't silently lose whatever was queued at that moment.
Failure handling: liveness probes restart a crashed agent, which then replays from its on-disk queue on startup; a corrupted on-disk queue file is rotated into a quarantine location and alerted on rather than silently discarded.
flowchart LR
N[Node: pods + kubelet] --> A[Agent, one per node]
A --> RB[In-memory ring buffer]
RB -- full --> DQ[(On-disk bounded queue)]
DQ -- network restored --> OUT[Tenant ingest endpoint]
CRD[TenantConfig / AgentPolicy CRD] --> CTRL[Controller: validate + sign]
CTRL -- push config, hot-reload --> A
PS[preStop hook] -- drain before upgrade --> DQ
Worked example
Sizing the on-disk buffer for a network partition. Assume 110 pods/node (a common Kubernetes per-node pod ceiling), each emitting 5 log lines/sec averaging 250 bytes/line:
nodeBytesPerSec=(110×5)×250=550×250=137,500 bytes/secFor a target of surviving a 10-minute network partition to the ingest endpoint without dropping data:
diskBufferBytes=137,500×(10×60)=82.5 MBWith 4x headroom for longer partitions or bursty traffic during the partition itself:
provisionedDisk=82.5×4=330 MBThat's a small, concrete disk request per node, roughly a third of a gigabyte, not the multi-gigabyte allocation teams often over-provision by default, because it's derived from the actual per-node telemetry rate rather than a round number.
In-memory burst buffer. If traffic spikes to 3x baseline for short periods (a log storm from a restart loop) and the agent needs 2 seconds of headroom before spilling to disk:
memBurstBytes=137,500×3×2=825,000 bytes≈825 KBA sub-megabyte in-memory buffer is enough to absorb a realistic burst at this node's telemetry rate before the disk-spill path takes over, which keeps the agent's steady-state memory footprint small and predictable across a large fleet of nodes.
Trade-offs & pitfalls
| Choice | Protects against | Cost |
|---|---|---|
| Per-node agent (DaemonSet) vs. per-pod sidecar | Lower resource overhead per node, one collector to operate | Less isolation than a sidecar; a single agent's bug affects every pod on that node |
| Bounded disk queue (330MB, 10-min target) | Data loss during short partitions | A partition longer than the target still drops data once the queue fills, so the target needs to match real observed outage durations, not a guess |
| Per-tenant rate limiting | Noisy-tenant starvation | A legitimately high-volume tenant gets throttled the same as a misbehaving one unless limits are tuned per tenant, not applied uniformly |
Common wrong turns: sizing the on-disk buffer as a fixed number (like "10GB, should be plenty") instead of deriving it from the node's actual telemetry rate and a stated outage-tolerance target, which either wastes disk on quiet nodes or under-provisions busy ones; treating a DaemonSet config change as safe to apply via full pod restart because "it's just a redeploy," which momentarily drops whatever was in the in-memory ring buffer at restart time, exactly the data loss the hot-reload CRD path exists to avoid; and sharing one rate-limit bucket across all tenants on a node instead of per-tenant buckets, which means a single noisy tenant can still starve every other tenant's telemetry even though "backpressure" was implemented.
Time-series databases lean on a handful of compression techniques: block-chunking, delta-of-delta timestamp encoding, XOR-based float compression (as in Facebook's Gorilla), and dictionary encoding for labels. Explain how each works and how it affects write throughput and query performance, and contrast a dense, monotonically-increasing counter against a sparse gauge: which techniques help most for each, and why?
Sample Answer
Time-series compression works because consecutive samples in a series are usually similar to each other: timestamps arrive at near-regular intervals, and values tend to change by small amounts (or not at all) between samples. Block-chunking, delta-of-delta timestamp encoding, XOR-based float compression, and label dictionary encoding each exploit a different piece of that similarity.
How each technique works
Block-chunking: samples are grouped into fixed time-window or fixed-count blocks (e.g., 2-hour windows), each with its own compressed byte stream and a small metadata header (start/end timestamp, min/max value). This lets a query engine skip whole blocks that fall outside a requested range without decompressing them, and it bounds how much has to be re-encoded when new data arrives (you only ever append to the current open block).
Delta-of-delta timestamp encoding: instead of storing each timestamp, store the delta from the previous timestamp, then the delta of that delta. For a series scraped at a fixed interval, the delta between consecutive timestamps is constant, so the delta-of-delta is exactly zero almost every time, which collapses to a single control bit per sample.
Di=(ti−ti−1)−(ti−1−ti−2)XOR-based float compression (Gorilla): XOR the current value's bit pattern against the previous value's bit pattern. If the value hasn't changed much, most of the leading bits (sign, exponent, high mantissa) and trailing bits (low mantissa) of the XOR result are zero, and only a short "meaningful" middle span needs to be stored explicitly.
meaningful_bits=64−leading_zeros(vi⊕vi−1)−trailing_zeros(vi⊕vi−1)Dictionary encoding for labels: label keys and values repeat across enormous numbers of series (env=prod appears on millions of series), so each distinct string is stored once in a dictionary and every series references it by a small integer ID instead of repeating the string.
Worked example: counter vs. sparse gauge
The following is executed Python (struct-level IEEE-754 bit manipulation, no external data) so the numbers are exactly reproducible:
import struct
def bits(f):
return struct.unpack('>Q', struct.pack('>d', f))[0]
def leading_zeros(x, width=64):
return width if x == 0 else width - x.bit_length()
def trailing_zeros(x, width=64):
if x == 0: return width
c = 0
while (x & 1) == 0:
x >>= 1; c += 1
return c
def xor_meaningful_bits(prev, cur):
x = bits(prev) ^ bits(cur)
if x == 0: return 0, 64, 0
lz, tz = leading_zeros(x), trailing_zeros(x)
return lz, tz, 64 - lz - tz
# dense monotonic counter: http_requests_total, +1 per 15s scrape
print(xor_meaningful_bits(184320.0, 184321.0))
# sparse gauge, small fluctuation: temperature idling
print(xor_meaningful_bits(21.4, 21.9))
# sparse gauge, large jump: temperature spikes
print(xor_meaningful_bits(21.9, 87.2))
Output:
(28, 35, 1) # counter +1 step: 1 meaningful bit
(16, 47, 1) # gauge, small change: also 1 meaningful bit
(9, 0, 55) # gauge, large jump: 55 meaningful bits (almost the full double)
And for delta-of-delta timestamps on a realistically jittered 15-second scrape, with the full input pinned so the result is exactly reproducible (10 explicit timestamps, which by the Di formula above yields 8 delta-of-delta values, since each output needs 3 consecutive timestamps):
timestamps = [
1720000000, 1720000015, 1720000030, 1720000045, 1720000060,
1720000075, 1720000089, 1720000105, 1720000120, 1720000134,
]
deltas = [timestamps[i] - timestamps[i - 1] for i in range(1, len(timestamps))]
delta_of_delta = [deltas[i] - deltas[i - 1] for i in range(1, len(deltas))]
print("deltas:", deltas)
print("delta-of-delta:", delta_of_delta)
Output:
deltas: [15, 15, 15, 15, 15, 14, 16, 15, 14]
delta-of-delta: [0, 0, 0, 0, -1, 2, -1, -1]
4/8 are exactly 0 (single control bit each): the first four scrape gaps land exactly on the 15-second interval, then jitter shows up as a 14s gap, a 16s gap, and two more gaps that miss the interval by one second, each producing a nonzero delta-of-delta.
Which techniques help most for which shape
| Series shape | Timestamp behavior | Value behavior | Best-fit techniques |
|---|---|---|---|
| Dense, monotonically-increasing counter (e.g., request count) | Very regular interval; delta-of-delta is 0 almost always | Small, steady XOR distance between consecutive floats even though the value keeps climbing, because the relative change per step is tiny | Delta-of-delta on timestamps (near 1 bit/sample), XOR compression on values (near 1 meaningful bit/sample as shown above); block-chunking with larger blocks since data is smooth and compresses uniformly well |
| Sparse gauge (e.g., a temperature or queue-depth sensor with irregular scrape gaps and occasional large jumps) | Irregular gaps mean delta-of-delta is frequently nonzero and needs the wider-value fallback encoding | Large jumps between samples XOR into far fewer leading/trailing zero bits (55 meaningful bits in the example above, near the 64-bit ceiling) | Smaller blocks so a poorly-compressing stretch doesn't drag down a whole block's average; per-block adaptive codec fallback (e.g., general-purpose compression like Snappy/LZ4 on top when Gorilla's assumptions don't hold); dictionary encoding still helps regardless since it targets labels, not values |
The counter case shows why Gorilla-style encoding is the default for metrics generally: even though the value itself is always changing (climbing), the bit pattern delta between consecutive floats stays small as long as the relative step size is small, which is true for almost all real counters. The gauge case shows the failure mode: a big jump changes the exponent bits, which wipes out both the leading-zero and trailing-zero runs simultaneously, so the encoding degrades toward storing the value nearly raw.
Trade-offs and pitfalls
- A common wrong turn is assuming XOR compression is "for gauges" and delta-encoding is "for counters" as a rule; the real determinant is how much the bit pattern changes step to step, which correlates with relative magnitude change, not with the metric type label.
- Partial decompression for range queries requires the block-level index (min/max timestamp, min/max value) to be cheap to read without decoding the compressed body; if you skip that index to save space, every range query degrades to a full block scan.
- Block size is a real tuning knob, not a footnote: too small and per-block metadata overhead dominates; too large and a single noisy stretch (like the sparse-gauge jump) drags down the compression ratio for the whole block and increases decompression latency for small range queries.
- Dictionary encoding for labels needs a compaction/garbage-collection story; a dictionary that only grows (never reclaims IDs for labels that stop being used) becomes its own unbounded-cardinality problem over long retention.
Unlock Full Question Bank
Get access to all 47 Observability and Monitoring Architecture interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.