Observability and Monitoring Architecture Questions

Direct Answer

Route queries by data age: hot is optimized for write and point-lookup latency at full resolution, warm keeps full resolution but trades write-path speed for better compression, and cold and archive downsample aggressively and lean on cheap object storage, accepting slower, block-oriented reads. The tiering only pays off if you also downsample, keeping full resolution for 7 years costs an order of magnitude more than the numbers below show.

Structured Elaboration

Rehydration is bounded by making every tier block-addressable by time range and label hash: a query for a specific series over a specific window only pulls the blocks that could contain it, using the manifest's Bloom filters (compact per-block structures that can say "this block definitely doesn't contain that series" without opening it) to skip the rest, rather than scanning a tier wholesale.

Tier flow

flowchart LR
  Q["Query Router"] --> HOT[("Hot: 1 day")]
  Q --> WARM[("Warm: 30 days")]
  Q --> COLD[("Cold: 1 year, downsampled")]
  Q --> ARC[("Archive: 7 years, downsampled")]
  HOT -->|"compaction"| WARM
  WARM -->|"downsample to 5min"| COLD
  COLD -->|"downsample to 1hr"| ARC
  ARC -.->|"on-demand rehydrate"| Q

Worked Example

Assume 1,000,000 active series, each scraped every 15 seconds (a stated design input). Raw points per series per day:

Total raw points/day across the fleet: $1{,}000{,}000 \times 5{,}760 = 5.76\times10^{9}$.

Hot (1 day). Assume time-series-optimized compression achieves roughly 2 bytes/point (a stated assumption for delta-of-delta timestamp and XOR value encoding, not a cited benchmark), plus 30% index overhead:

Warm (30 days). Larger compaction blocks improve the ratio slightly to 1.5 bytes/point, still at full resolution:

Cold (1 year). Downsample to 5-minute resolution, a 300s/15s = 20x reduction in point count, at 2 bytes/point:

Had the cold tier kept raw resolution instead, the year would cost $5.76\times10^{9} \times 365 \times 2\ \text{B} \approx 4{,}204.8\ \text{GB}$, about 20x more, which is the whole reason the downsample step exists.

Archive (years 2 to 7, 6 years). Downsample further to 1-hour resolution (24 samples/day), at 1.5 bytes/point since bulk cold storage compresses better:

Total 7-year footprint (before replication): $15.0 + 259.2 + 210.2 + 78.8 \approx 563.2\ \text{GB}$. Kept at raw resolution for all 7 years instead, the footprint would be roughly $4{,}204.8 \times 7 \approx 29{,}434\ \text{GB} \approx 29.4\ \text{TB}$, about 52x larger. That gap is the entire economic argument for tiering with downsampling rather than tiering on storage class alone.

Trade-offs and Pitfalls

Downsampling is lossy by design: once a cold-tier block collapses 20 raw points into one 5-minute aggregate, you cannot recover the original spikes inside that window. Decide per metric type whether that is acceptable (aggregate SLI counters, usually fine) or not (a metric feeding an anomaly-detection model that needs the raw shape, usually not, and should either stay hot longer or get its own retention policy).

Cross-tier query stitching is a common failure point: a dashboard spanning "last 45 days" crosses the hot/warm boundary and the warm/cold boundary is close by too, and the query planner has to merge full-resolution warm data with 5-minute cold data without a visible discontinuity in the chart. Test this seam explicitly, it is where tiering bugs show up in production, not inside a single tier.

Archive rehydration cost is easy to underestimate: a 1-hour restore-initiation SLA sounds fast until a customer asks for a 3-year lookback across 1,000 series and that pulls thousands of archive blocks, each carrying its own restore latency and egress cost. Expose rehydration as an explicit, rate-limited, customer-visible operation rather than pretending archive queries behave like cold-tier queries.

Direct Answer

Split ingestion by region so no single path crosses a WAN in the hot loop: collectors batch and buffer locally, hand off to a partitioned durable log keyed by series identity, and a stream layer deduplicates and rolls up before anything touches long-term storage. The three levers that make one million points per second tractable are partition count (parallelism), batch size (write amplification), and pre-aggregation (what actually needs to survive at full resolution).

Structured Elaboration

Pipeline topology

flowchart LR
  subgraph REGION["Per-Region Tier (x3)"]
    APP[Service Instances] --> AGENT[Collector Agent]
    AGENT --> WAL[("Local WAL Buffer")]
    WAL --> KAFKA[["Kafka: 32 partitions by tenant+series key"]]
  end
  KAFKA --> SP["Stream Processor: dedup + rollup"]
  SP --> TSDB[("Hot TSDB")]
  SP --> OBJ[("Object Storage: rollups")]

Collector and agent placement
Run a lightweight collection tier per region (behind a regional load balancer, autoscaled in Kubernetes) so no metric point leaves its region before being durably buffered. Cross-region replication happens downstream, at the storage layer, never in the write-critical path, so a WAN blip in one region does not add latency to the other two.

Buffering and batching
Each collector holds a local disk-backed buffer (a WAL, RocksDB-backed queue, or equivalent) so a restart or a downstream stall does not drop in-flight data. Batch by size, not purely by time: a size trigger keeps latency proportional to actual load instead of always waiting out a fixed window.

Message broker and partitioning keys
Use a partitioned durable log (Kafka or equivalent) per region. Partition key: hash of tenant_id + metric_name. This keeps every point for a given series in the same partition, which is what makes per-series ordering and local windowed aggregation possible downstream, at the cost of potential hot partitions for very high-cardinality single tenants.

Deduplication
Give every point an idempotency key (source_id + monotonic sequence number). The stream processor keeps a rolling window of seen keys; anything already seen in that window is a duplicate produced by a retry, not new data.

Backpressure handling
The collector never blocks its callers. When the local buffer approaches capacity, it sheds lowest-priority series first (a stated priority tier, not silent random drop) and raises an explicit metric so the shedding is visible, not a silent gap in a dashboard three weeks later.

Fault tolerance
Replicate the log (replication factor 3) across brokers in multiple availability zones. Collectors are stateless except for the local buffer, so a lost collector instance loses only unflushed buffer content, not history. Stream processors checkpoint offsets so a crash resumes from the last committed point, not from zero.

Pre-aggregation and rollups
Do windowed rollups (count, sum, min, max, and a percentile sketch) in the stream layer before the write to the hot store. Keep raw resolution for a short window (operators debugging an active incident need seconds-level data); roll everything older than that window up to coarser resolution, since almost no dashboard or alert needs one-second granularity on data from an hour ago.

Worked Example

Assume 1,000,000 points/sec split across 3 regions and an average encoded point size of 150 bytes (timestamp, value, and label set after protobuf encoding, a stated design input, not a benchmark). Regional steady load is then:

Plan for regional skew: if one region fails, its traffic can fail over to the nearest healthy region, so size for up to 1.5x the average, 500,000 pts/s = 75 MB/s peak.

Partition count. Choose a conservative per-partition write budget of 10 MB/s (accounts for replication-factor-3 fsync overhead on commodity brokers, a stated assumption, not a vendor benchmark):

Round up to 32 partitions per regional topic: this gives 4x headroom over the peak-throughput floor of 8, so future traffic growth or a temporary partition hot-spot does not require an emergency repartition, and it splits evenly across, say, 8 stream-processor instances at 4 partitions each.

Batch fill time. At the peak-derived floor of 8 partitions, each carries roughly 50 MB/s / 8 = 6.25 MB/s. A 1 MB size-triggered batch fills in:

That is well under a reasonable 500 ms linger cap, so the size trigger (not the timeout) governs flush cadence under normal load, and the timeout only matters for low-traffic partitions.

Deduplication memory. With a 2-minute (120 s) dedup window at the regional steady rate, the number of distinct keys the stream processor must track at once is:

Sizing a Bloom filter (a compact structure that answers "have I possibly seen this key before," with a small, tunable false-positive rate but no false negatives) for these keys at a 0.1% false-positive rate ($p = 0.001$):

A roughly 72 MB Bloom filter per region is cheap enough to keep fully in memory and gives a bounded, known false-positive rate for the dedup layer, versus an exact hash-set which would need far more memory to track 40 million live keys.

Local buffer disk sizing. To survive a 10-minute (600 s) broker outage at the regional steady rate without dropping data:

Provisioning roughly 30 GB of local disk per regional collector tier is the concrete number that backs the "collectors survive a broker outage" claim, not just an assertion that buffering exists.

Trade-offs and Pitfalls

A common alternative is writing directly from collectors to the time-series store, skipping the durable log entirely. That removes a hop and its operational cost, but couples ingest availability directly to storage availability: any storage hiccup now blocks collectors instead of just delaying a downstream consumer. The durable log is worth its cost specifically because it decouples those failure domains.

Pre-aggregation trades ingest-time compute for downstream storage and query cost: computing rollups at 1,000,000 points/sec needs real CPU budget in the stream layer, and if the rollup logic has a bug, it is much harder to recompute correct history than if raw data were simply sitting untouched in cheap storage. Keep raw data for a bounded window specifically so a bad rollup is recoverable.

The partition key of tenant_id + metric_name preserves per-series locality but can create a hot partition if one tenant emits a disproportionate share of traffic; watch for this and consider adding a bucket suffix to the key for known outlier tenants rather than repartitioning the whole topic reactively.

Default to logical isolation (shared infrastructure with hard per-tenant quotas and QoS enforcement) for the bulk of tenants, and offer physical isolation (dedicated shards or node pools) as an explicit, metered upgrade path for tenants whose usage or SLA requirements outgrow what shared quotas can safely guarantee. The isolation model and the migration path are two sides of the same design.

Architecture

flowchart LR
  A[Tenant Requests] --> B[Ingress: Auth and RBAC]
  B --> C[Per-Tenant Shard / Queue]
  C --> D[Shared Ingestion Pool]
  C --> E[Dedicated Ingestion Pool]
  D --> F[Query Gateway: QoS Scheduler]
  E --> F
  F --> G[Shared Query Compute]
  F --> H[Dedicated Query Compute]
  I[Billing / Quota Manager] --> B
  I --> F

• Logical isolation: per-tenant partitions/queues on shared compute, enforced with token-bucket rate limits at ingress and query-time concurrency caps; cheapest, and sufficient for the majority of tenants whose usage is well within their quota most of the time.
• Physical isolation: dedicated shard, node pool, or account for a tenant; strongest guarantee, but the operational and cost overhead of running fully separate infrastructure per tenant doesn't scale to hundreds of tenants, so it has to be selective.
• Query-level QoS: priority classes (interactive dashboard queries vs. batch/backfill queries), per-tenant concurrency limits, and admission control that sheds low-priority load before it degrades everyone; this is what actually prevents a noisy tenant's expensive query from starving others on shared compute, since ingestion isolation alone doesn't protect the read path.
• Billing-aware quotas: map each tenant's plan tier to a concrete ingest-rate and query-concurrency quota; soft-limit warnings before hard throttling, and an explicit overdraft/pay-as-you-go path rather than a silent hard cutoff. Retention is part of the same per-tenant contract, not a platform-wide constant: a tenant's plan tier should set its own retention window (e.g., 7 days on a shared/basic tier vs. 90 days on a dedicated tier), enforced as tenant-scoped TTL policy in the storage layer so one tenant's longer retention SLA doesn't force everyone else to pay for the same window.

Sizing the admission-control headroom

The core quantitative question for logical isolation is: how much burst capacity can the shared pool actually absorb before a legitimate burst from one tenant risks starving others? Take a platform with total ingest capacity $C = 2{,}000{,}000$ samples/sec shared across $N = 500$ tenants, where baseline quotas are provisioned to consume a target fraction $u$ of total capacity (leaving headroom for bursts), and tenants are allowed to burst up to $m\times$ their baseline:

If a fraction $f$ of tenants burst simultaneously while the rest sit at baseline, total load must stay under capacity:

Substituting $\text{baseline}_{\text{tenant}} = uC/N$ and simplifying:

With $u = 0.6$ (provision baseline to consume 60% of capacity, leaving 40% headroom) and $m = 5$ (allow a 5x burst):

C, N, u, m = 2_000_000, 500, 0.6, 5
baseline = (u * C) / N          # 2,400 samples/sec/tenant
burst = m * baseline            # 12,000 samples/sec/tenant
f_max = (1/u - 1) / (m - 1)     # 0.1667
max_bursting = f_max * N        # 83.3 tenants

Result: baseline quota is 2,400 samples/sec/tenant, burst allowance is 12,000 samples/sec/tenant, and up to about 16.7% of tenants (roughly 83 of 500) can burst simultaneously at 5x without exceeding total capacity. Plugging $f_{max}$ back into the original inequality confirms it lands exactly at capacity (2,000,000 samples/sec), which is the check that the derivation is self-consistent. This is the number that should actually drive the admission controller's global burst budget, not a guess: if more than ~83 tenants try to burst at once, the controller has to start denying or queuing burst requests rather than granting them all.

Migrating a tenant from shared to dedicated

1. Trigger: sustained usage consistently near quota (not just occasional bursts), or an explicit SLA purchase requiring guaranteed isolation.
2. Provision dedicated shard/node pool ahead of cutover.
3. Dual-write or replicate the tenant's recent data into the new dedicated shard while it's still live on the shared pool.
4. Cut over routing at the control plane (ingress rules keyed on tenant ID) once the dedicated shard is caught up; this should be a routing change, not a data migration event, so it can be near-zero-downtime.
5. Decommission the tenant's shared-pool footprint after a verification window, and keep the cutover reversible in case the dedicated shard has an unexpected issue.

Trade-offs and pitfalls

• Sizing baseline quotas at $u$ close to 1.0 (using nearly all capacity for guaranteed baseline) leaves almost no burst headroom, which defeats the purpose of a shared pool; the $u$ vs. burst-headroom trade-off above should be an explicit, revisited decision, not a default.
• Query-level QoS is often skipped because ingestion isolation feels like "the isolation problem," but an expensive ad-hoc query from one tenant can degrade shared query compute even when every tenant's ingestion is perfectly isolated; both paths need protection independently.
• A migration path that isn't reversible (no fallback if the dedicated shard has a problem post-cutover) turns a capacity upgrade into a risk event; always keep the shared-pool footprint alive through a verification window.
• Billing-aware quotas without a clear soft-limit warning stage turn every quota breach into a support ticket; the graduated response (warn, throttle, then hard-limit) matters as much as the quota number itself.

Direct answer

Adopt the W3C Trace Context standard (traceparent/tracestate headers) as the single canonical propagation format, translate legacy headers to it at the edge (gateways and service-mesh sidecars) during migration so every hop sees a standard header regardless of what the originating service still emits, and enforce adoption with automated propagation-continuity tests in CI rather than a one-time audit. A request that shows up with missing or partial context gets a freshly minted root trace ID at the first trusted boundary, tagged so it's visibly distinguishable from a properly-propagated trace during debugging.

Migration and enforcement approach

• Translate at the edge first, not the leaves. API gateways, load balancers, and service-mesh sidecars are a small, centrally-controlled set of chokepoints compared to 1,000 individual services. Teaching them to read a legacy header (e.g. X-Trace-Id) and emit a standard traceparent alongside it (marking origin in tracestate) gets standard propagation working end-to-end immediately, without waiting on every service team.
• Provide a thin, zero-dependency propagation library per language, not a full tracing SDK. Services only need the ability to read/attach context and forward it through HTTP, gRPC, and message-queue headers; that's a much smaller adoption ask than "instrument your whole service."
• Dual-write during the transition: once a service is updated, it reads both legacy and standard headers (preferring standard) and writes both, so downstream services that haven't migrated yet still get what they expect.
• Cut over legacy emission once adoption crosses a high-confidence threshold (e.g. 95%+), then remove the compatibility shim from the edge translators; keeping compatibility code indefinitely is itself a long-term maintenance and correctness liability.

Enforcement, not just adoption

• CI test that fails a build if outgoing requests from an instrumented service don't carry a valid traceparent: propagation loss is caught at merge time, not in a production incident three weeks later.
• A live metric, propagation_loss_total, incremented whenever a service receives a request with no trace context on a path where the caller was known to be migrated; this turns "context got dropped somewhere" from an anecdote into an alertable, per-service signal during rollout.

Handling missing or partial context

• Fully missing (traceparent absent): the first trusted boundary (edge gateway or service-mesh ingress) mints a new 128-bit trace ID and marks tracestate with origin=synthesized so anyone debugging later immediately knows this trace didn't start where they'd expect.
• Partial (trace ID present, span ID missing or malformed): create a new span with the given trace ID as parent context where possible, and record the same partial=true flag; don't silently drop the partial trace ID information, since even a broken parent link is more useful for correlation than starting fresh.
• Sampling decisions: if a sampler hint is present in tracestate, honor it; if absent, apply deterministic (hash-of-trace-ID) sampling at the edge so downstream services don't each make an independent, inconsistent sampling call for the same trace.

Worked example

Header overhead. A traceparent header (00-<32 hex trace id>-<16 hex parent id>-<2 hex flags>) is $2+1+32+1+16+1+2 = 55$ bytes. Assume an average tracestate payload of 40 bytes, for $55+40=95$ bytes of propagation overhead added per request.

At an assumed average of $r=200$ requests/sec per service across $n=1{,}000$ services:

That's a small, easily-budgeted fixed cost across the whole fleet, confirming the "minimal runtime overhead" requirement is satisfiable by the header format choice itself.

Migration cadence. With edge translation already giving end-to-end propagation on day one, the remaining work is migrating each service off legacy-only emission. At a rollout cadence of $B = 50$ services/week (a process/scheduling parameter the team sets, driven by how many services can be safely canaried per week):

The compatibility shim at the edge stays in place through week 19-20 and is removed only after the 95% threshold and a grace period, which is what makes the cutover safe rather than a hard deadline that breaks the long tail of stragglers.

flowchart LR
  Request[Incoming Request] --> Gateway[Edge Gateway]
  Gateway -->|has traceparent| Propagate[Forward W3C Context]
  Gateway -->|missing or partial| Mint[Mint Root ID + partial flag]
  Propagate --> LegacySvc[Not-yet-migrated Service]
  Mint --> LegacySvc
  LegacySvc --> Shim[Compat Shim: legacy to W3C]
  Shim --> Collector[OTel Collector]
  Collector --> Backend[Trace Backend]

Trade-offs and pitfalls

• Edge-only translation gets end-to-end propagation working fast, but it means internal, service-to-service context (span-level parent/child relationships between two not-yet-migrated services) is still lossy until those specific services adopt the standard library; edge translation solves the trace-ID-continuity problem, not the full-fidelity-span-tree problem.
• Dual-writing both header formats during transition roughly doubles header overhead temporarily; that's an accepted, bounded cost (the 19 MB/s figure above would briefly be closer to double) in exchange for a safe rollout with no hard cutover date.
• Removing the compatibility shim too early, before the long tail of stragglers has actually migrated, silently breaks propagation for exactly the services least likely to have good test coverage; the 95% threshold plus a grace period exists specifically to avoid that failure mode.
• Marking synthesized root traces (origin=synthesized) is easy to skip under time pressure but is what prevents a debugging engineer from wasting time trying to find a "missing" parent span that never existed.

Direct answer

Fix this at the architecture level, not by throwing more compute at the same query plan. Build a layer of materialized views (precomputed, stored query results that refresh as new data lands, instead of being recomputed from scratch on every request) that pre-aggregate the group-bys and time windows dashboards actually use, add a query rewriter (a planner step that intercepts an incoming query and swaps it for a cheaper, equivalent one) that transparently substitutes a matching materialized view for the raw scan whenever the rewrite preserves aggregate semantics, and cache the result of the highest-frequency top-K queries with a TTL tied to the refresh cadence. Decide what to precompute with a cost-benefit rule: materialize a query shape when the daily rows it saves scanning outweighs the daily rows its incremental refresh costs to maintain, not by intuition about which dashboards "feel slow."

Structured elaboration

Materialized views (MVs): store pre-grouped, pre-aggregated rows keyed by the dimensions a panel actually displays (for example, top-20 services by error rate), refreshed incrementally as new raw data lands, not recomputed from scratch each time.

Pre-aggregation windows: keep multiple resolutions so a query can pick the coarsest one that still covers its time range: 1-minute rollups for the last hour, 5-minute for the last day, 1-hour for the last month. This bounds how many rows any query has to touch regardless of the underlying cardinality.

Partitioning (the piece the naive scan is missing): partition the raw and rollup tables by time first, then by a bounded set of high-selectivity dimensions (service, region). This lets both the raw fallback path and the MV refresh job skip whole partitions instead of scanning the full high-cardinality series space, which is what turns a query over "recent, high-cardinality metrics" from a full-table scan into a bounded one.

Query rewriting: a planner stage intercepts the incoming query, checks whether its group-by, filter, and time window are a subset of an existing MV's coverage, and rewrites the query to read the MV instead of raw data. Only rewrite when the operation is safe (sums, counts, min/max compose across MVs cleanly; distinct counts and unbounded percentiles generally do not without a sketch-based MV, which needs its own merge logic).

Caching top-K queries: cache the actual result set for the highest-frequency parameterized queries (dashboard panel + time range), keyed by a hash of the query shape and the current rollup epoch, invalidated on the next refresh rather than on a wall-clock timer.

flowchart LR
    Q[Incoming dashboard query] --> P{Query rewriter}
    P -- matches an MV --> MV[(Materialized view / rollup)]
    P -- no safe match --> RAW[(Partitioned raw store)]
    MV --> CACHE{Top-K result cache}
    CACHE -- hit --> R[Response]
    CACHE -- miss --> R
    RAW --> R
    ING[Streaming ingest] -- incremental refresh --> MV

Selection criterion, precisely: precompute a query shape when

where $f$ is how often that shape is queried per day. This is the freshness-versus-speed trade-off made concrete: the left side is what you save on reads, the right side is what you pay to keep the view fresh.

Worked example

A dashboard panel groups by service across $K = 100{,}000$ series matched by its filter, over a $W = 3600$s (1-hour) window, at a 15s scrape interval.

Without a materialized view, every execution scans one row per series per scrape tick in the window:

With a materialized view that stores 1-minute rollups already grouped down to the top 20 series the panel displays:

At $f = 500$ views/day for this panel, the daily rows saved by reading the MV instead of raw:

The MV refreshes incrementally every minute (1,440 refreshes/day), and each refresh only has to process the new minute's raw rows for the panel's series ($K \times 4$ samples/minute):

Savings exceed maintenance cost by roughly 20.8x here, so this panel clears the bar comfortably. Solving the selection inequality for the breakeven view frequency:

So the concrete criterion for this panel shape is: materialize it once it's viewed more than roughly 24 times a day; below that, the refresh overhead isn't earning its keep and the raw fallback path is cheaper.

Trade-offs & pitfalls

Combining all three, with the rewriter deciding per-query which to use, is what actually removes the timeout; any one alone leaves a gap.

Common wrong turns: materializing every group-by combination a dashboard could theoretically ask for (storage and refresh cost grow combinatorially, and most of those shapes are never queried, i.e. $f \approx 0$, which fails the selection criterion outright); rewriting queries onto an MV whose aggregation isn't actually composable for the requested operation (silently wrong distinct-counts or percentiles are worse than a slow correct answer); and caching on a fixed TTL instead of the rollup epoch, which either serves stale data past a refresh or invalidates a cache entry that's still perfectly valid.