Observability and Monitoring Architecture Questions

Default to logical isolation (shared infrastructure with hard per-tenant quotas and QoS enforcement) for the bulk of tenants, and offer physical isolation (dedicated shards or node pools) as an explicit, metered upgrade path for tenants whose usage or SLA requirements outgrow what shared quotas can safely guarantee. The isolation model and the migration path are two sides of the same design.

Architecture

flowchart LR
  A[Tenant Requests] --> B[Ingress: Auth and RBAC]
  B --> C[Per-Tenant Shard / Queue]
  C --> D[Shared Ingestion Pool]
  C --> E[Dedicated Ingestion Pool]
  D --> F[Query Gateway: QoS Scheduler]
  E --> F
  F --> G[Shared Query Compute]
  F --> H[Dedicated Query Compute]
  I[Billing / Quota Manager] --> B
  I --> F

• Logical isolation: per-tenant partitions/queues on shared compute, enforced with token-bucket rate limits at ingress and query-time concurrency caps; cheapest, and sufficient for the majority of tenants whose usage is well within their quota most of the time.
• Physical isolation: dedicated shard, node pool, or account for a tenant; strongest guarantee, but the operational and cost overhead of running fully separate infrastructure per tenant doesn't scale to hundreds of tenants, so it has to be selective.
• Query-level QoS: priority classes (interactive dashboard queries vs. batch/backfill queries), per-tenant concurrency limits, and admission control that sheds low-priority load before it degrades everyone; this is what actually prevents a noisy tenant's expensive query from starving others on shared compute, since ingestion isolation alone doesn't protect the read path.
• Billing-aware quotas: map each tenant's plan tier to a concrete ingest-rate and query-concurrency quota; soft-limit warnings before hard throttling, and an explicit overdraft/pay-as-you-go path rather than a silent hard cutoff. Retention is part of the same per-tenant contract, not a platform-wide constant: a tenant's plan tier should set its own retention window (e.g., 7 days on a shared/basic tier vs. 90 days on a dedicated tier), enforced as tenant-scoped TTL policy in the storage layer so one tenant's longer retention SLA doesn't force everyone else to pay for the same window.

Sizing the admission-control headroom

The core quantitative question for logical isolation is: how much burst capacity can the shared pool actually absorb before a legitimate burst from one tenant risks starving others? Take a platform with total ingest capacity $C = 2{,}000{,}000$ samples/sec shared across $N = 500$ tenants, where baseline quotas are provisioned to consume a target fraction $u$ of total capacity (leaving headroom for bursts), and tenants are allowed to burst up to $m\times$ their baseline:

If a fraction $f$ of tenants burst simultaneously while the rest sit at baseline, total load must stay under capacity:

Substituting $\text{baseline}_{\text{tenant}} = uC/N$ and simplifying:

With $u = 0.6$ (provision baseline to consume 60% of capacity, leaving 40% headroom) and $m = 5$ (allow a 5x burst):

C, N, u, m = 2_000_000, 500, 0.6, 5
baseline = (u * C) / N          # 2,400 samples/sec/tenant
burst = m * baseline            # 12,000 samples/sec/tenant
f_max = (1/u - 1) / (m - 1)     # 0.1667
max_bursting = f_max * N        # 83.3 tenants

Result: baseline quota is 2,400 samples/sec/tenant, burst allowance is 12,000 samples/sec/tenant, and up to about 16.7% of tenants (roughly 83 of 500) can burst simultaneously at 5x without exceeding total capacity. Plugging $f_{max}$ back into the original inequality confirms it lands exactly at capacity (2,000,000 samples/sec), which is the check that the derivation is self-consistent. This is the number that should actually drive the admission controller's global burst budget, not a guess: if more than ~83 tenants try to burst at once, the controller has to start denying or queuing burst requests rather than granting them all.

Migrating a tenant from shared to dedicated

1. Trigger: sustained usage consistently near quota (not just occasional bursts), or an explicit SLA purchase requiring guaranteed isolation.
2. Provision dedicated shard/node pool ahead of cutover.
3. Dual-write or replicate the tenant's recent data into the new dedicated shard while it's still live on the shared pool.
4. Cut over routing at the control plane (ingress rules keyed on tenant ID) once the dedicated shard is caught up; this should be a routing change, not a data migration event, so it can be near-zero-downtime.
5. Decommission the tenant's shared-pool footprint after a verification window, and keep the cutover reversible in case the dedicated shard has an unexpected issue.

Trade-offs and pitfalls

• Sizing baseline quotas at $u$ close to 1.0 (using nearly all capacity for guaranteed baseline) leaves almost no burst headroom, which defeats the purpose of a shared pool; the $u$ vs. burst-headroom trade-off above should be an explicit, revisited decision, not a default.
• Query-level QoS is often skipped because ingestion isolation feels like "the isolation problem," but an expensive ad-hoc query from one tenant can degrade shared query compute even when every tenant's ingestion is perfectly isolated; both paths need protection independently.
• A migration path that isn't reversible (no fallback if the dedicated shard has a problem post-cutover) turns a capacity upgrade into a risk event; always keep the shared-pool footprint alive through a verification window.
• Billing-aware quotas without a clear soft-limit warning stage turn every quota breach into a support ticket; the graduated response (warn, throttle, then hard-limit) matters as much as the quota number itself.

Direct Answer

Route queries by data age: hot is optimized for write and point-lookup latency at full resolution, warm keeps full resolution but trades write-path speed for better compression, and cold and archive downsample aggressively and lean on cheap object storage, accepting slower, block-oriented reads. The tiering only pays off if you also downsample, keeping full resolution for 7 years costs an order of magnitude more than the numbers below show.

Structured Elaboration

Rehydration is bounded by making every tier block-addressable by time range and label hash: a query for a specific series over a specific window only pulls the blocks that could contain it, using the manifest's Bloom filters (compact per-block structures that can say "this block definitely doesn't contain that series" without opening it) to skip the rest, rather than scanning a tier wholesale.

Tier flow

flowchart LR
  Q["Query Router"] --> HOT[("Hot: 1 day")]
  Q --> WARM[("Warm: 30 days")]
  Q --> COLD[("Cold: 1 year, downsampled")]
  Q --> ARC[("Archive: 7 years, downsampled")]
  HOT -->|"compaction"| WARM
  WARM -->|"downsample to 5min"| COLD
  COLD -->|"downsample to 1hr"| ARC
  ARC -.->|"on-demand rehydrate"| Q

Worked Example

Assume 1,000,000 active series, each scraped every 15 seconds (a stated design input). Raw points per series per day:

Total raw points/day across the fleet: $1{,}000{,}000 \times 5{,}760 = 5.76\times10^{9}$.

Hot (1 day). Assume time-series-optimized compression achieves roughly 2 bytes/point (a stated assumption for delta-of-delta timestamp and XOR value encoding, not a cited benchmark), plus 30% index overhead:

Warm (30 days). Larger compaction blocks improve the ratio slightly to 1.5 bytes/point, still at full resolution:

Cold (1 year). Downsample to 5-minute resolution, a 300s/15s = 20x reduction in point count, at 2 bytes/point:

Had the cold tier kept raw resolution instead, the year would cost $5.76\times10^{9} \times 365 \times 2\ \text{B} \approx 4{,}204.8\ \text{GB}$, about 20x more, which is the whole reason the downsample step exists.

Archive (years 2 to 7, 6 years). Downsample further to 1-hour resolution (24 samples/day), at 1.5 bytes/point since bulk cold storage compresses better:

Total 7-year footprint (before replication): $15.0 + 259.2 + 210.2 + 78.8 \approx 563.2\ \text{GB}$. Kept at raw resolution for all 7 years instead, the footprint would be roughly $4{,}204.8 \times 7 \approx 29{,}434\ \text{GB} \approx 29.4\ \text{TB}$, about 52x larger. That gap is the entire economic argument for tiering with downsampling rather than tiering on storage class alone.

Trade-offs and Pitfalls

Downsampling is lossy by design: once a cold-tier block collapses 20 raw points into one 5-minute aggregate, you cannot recover the original spikes inside that window. Decide per metric type whether that is acceptable (aggregate SLI counters, usually fine) or not (a metric feeding an anomaly-detection model that needs the raw shape, usually not, and should either stay hot longer or get its own retention policy).

Cross-tier query stitching is a common failure point: a dashboard spanning "last 45 days" crosses the hot/warm boundary and the warm/cold boundary is close by too, and the query planner has to merge full-resolution warm data with 5-minute cold data without a visible discontinuity in the chart. Test this seam explicitly, it is where tiering bugs show up in production, not inside a single tier.

Archive rehydration cost is easy to underestimate: a 1-hour restore-initiation SLA sounds fast until a customer asks for a 3-year lookback across 1,000 series and that pulls thousands of archive blocks, each carrying its own restore latency and egress cost. Expose rehydration as an explicit, rate-limited, customer-visible operation rather than pretending archive queries behave like cold-tier queries.

Direct Answer

Use tail-based sampling with a short per-trace buffering window, so the sampling decision can see whether an error or rare condition appeared anywhere in the trace before deciding to keep or drop it. Split the policy in two: always retain traces carrying an error or rare-severity flag (unconditionally, outside the budget), then probabilistically sample the remaining normal traffic at whatever rate fits what budget is left. The buffer window is the dial between decision correctness (catching a late-arriving error span) and cost (memory held per in-flight trace, plus decision latency).

Structured Elaboration

Sampling pipeline

flowchart LR
  SPANS["Incoming Spans"] --> BUF[("Per-Trace Buffer, N sec window")]
  BUF --> DEC{"Trace complete or window expired"}
  DEC -->|"error / rare signal"| KEEP["Always Retain"]
  DEC -->|"normal"| SAMP["Budget-Aware Sampler"]
  SAMP -->|"sampled in"| KEEP
  SAMP -->|"sampled out"| DROP["Discard"]
  KEEP --> STORE[("Trace Storage")]

Buffering approach
Spans for a given trace ID accumulate in a short-lived per-trace buffer as they arrive out of order from different services. The buffer closes and a decision is made either when the trace looks complete (a root span closes) or when a maximum window expires, whichever comes first, so a trace that never completes cleanly (a dropped span, a crashed service) does not hold the buffer open indefinitely.

Signals feeding the sampling decision

• Any span carrying an error status or exception.
• A latency signal: a span exceeding a defined threshold for its operation.
• A rare, high-severity condition flag emitted explicitly by the service (a specific error code, a business-critical operation type).
• For the remaining normal traffic, a running measure of budget consumed so far in the current period, feeding a dynamic sampling probability.

Decision and retention policy

1. If any signal above fires: retain unconditionally, outside the budget calculation.
2. Otherwise: sample at a probability computed to hit the remaining budget for the period (see worked example), using a consistent, deterministic hash of the trace ID so the decision is reproducible if re-evaluated.

Latency versus correctness trade-off
A longer buffer window catches more late-arriving error signals (higher correctness) but holds more traces in memory longer and delays the retain/drop decision (higher latency and cost). A shorter window decides faster and cheaper but risks finalizing a trace as "normal" just before its error span arrives from a slow downstream hop.

Worked Example

Assume a fleet-wide rate of 50,000 traces/sec, an average trace size of 5,000 bytes (spans, tags, and metadata combined, a stated design input), and a measured error rate of 0.5% of traces (a stated assumption feeding the calculation, not a claim about any specific system).

Total traces/day: $50{,}000 \times 86{,}400 = 4.32\times10^{9}$.

Error traces/day (always kept): $0.005 \times 4.32\times10^{9} = 2.16\times10^{7}$, costing $2.16\times10^{7} \times 5{,}000\ \text{B} \approx 108\ \text{GB/day}$.

Given a fixed storage budget of 500 GB/day, the remaining budget for non-error traces is $500 - 108 = 392\ \text{GB/day}$.

Non-error traces/day: $4.32\times10^{9} - 2.16\times10^{7} = 4{,}298{,}400{,}000$.

Required sampling rate on non-error traffic to spend exactly the remaining budget:

So the system retains 100% of error traces and roughly 1.82% of normal traces, for a blended retention rate of about $500/21{,}600 \approx 2.31\%$ of all traces by volume (total unsampled data would be $4.32\times10^9 \times 5{,}000\ \text{B} \approx 21{,}600\ \text{GB/day}$).

Buffer memory. Holding a 5-second decision window at the full incoming rate: $50{,}000 \times 5{,}000\ \text{B} \times 5\ \text{s} = 1.25\times10^{9}\ \text{B} \approx 1.25\ \text{GB}$ of in-flight trace buffer needed per collector-tier aggregate, a concrete, provisionable memory figure rather than an open-ended "keep enough buffer" statement.

Trade-offs and Pitfalls

Head-based sampling (deciding at the very first span, before the rest of the trace exists) is cheaper and adds no buffering latency, but structurally cannot guarantee error retention: the decision happens before the error span, if any, has even occurred. Tail-based sampling is the only way to honor "always keep errors," at the cost of the buffer described above.

A fixed buffer window creates a silent correctness gap for the tail of the trace-duration distribution: if the window is 5 seconds but a small fraction of traces (the slowest, often the most interesting) take 8+ seconds end to end, those traces get finalized and possibly dropped before their late error span arrives. Mitigate by extending the window specifically for traces that have already crossed a latency threshold mid-flight, rather than using one fixed window for everything.

Dynamic budget-based sampling can create a feedback loop under a real incident: an incident produces more errors, which consumes more of the always-keep budget, which is fine, but if the incident also produces more overall traffic, the non-error sampling rate has to drop to compensate, right when operators most want visibility into the surrounding normal traffic for comparison. Consider reserving a small, fixed floor sampling rate for normal traffic that the budget calculation cannot squeeze to zero.

Direct answer

Model sampling rules as immutable, monotonically-versioned objects in a central rule store; distribute them to collectors as an ordered delta stream (not independent pushes) so every client applies changes in the same order; make client-side apply atomic (swap-the-whole-ruleset, never patch-in-place) so a mid-apply crash can't leave a client on a mixed rule state; and gate every rollout behind a canary with an automated, metrics-driven rollback that republishes the last-known-good version the moment the canary's sampled-span rate deviates outside a statistically defensible band.

Rule distribution protocol

• Rule object: {rule_id, scope (service/env/instance selector), sampling_policy, rate_limit, generation, created_by, timestamp}. generation is a single, server-assigned, strictly increasing integer; it is the only source of ordering truth.
• Delta stream: the control plane publishes {base_generation, ops: [add/update/remove], new_generation} over a long-lived stream (gRPC bidi stream, or HTTP/2 push with a resumable cursor). Clients also support a full-snapshot pull (conditional GET keyed on generation) as a recovery path.
• Client apply: a client only accepts a delta if base_generation matches its current local generation; on any mismatch it discards the delta and requests a full snapshot instead of trying to reconcile partial state. This turns "did the client miss an update" into a cheap, self-healing check instead of a distributed-consistency problem.

Avoiding races between clients on different rule versions

Because rules are immutable and versioned, two clients on different generations are not in conflict with each other: each is internally consistent, just stale relative to the newest version. The risk is entirely local: a client applying a delta concurrently with in-flight sampling decisions. That's solved by atomic pointer swap: the client builds the new ruleset object off to the side, then swaps a single pointer once it's fully constructed. In-flight trace decisions read whichever ruleset was current at the moment they started; nothing reads a half-updated struct.

Operator-side races (two admins editing the same rule) are handled with optimistic concurrency: an edit must specify the generation it's based on, and the server rejects it with a conflict if that generation is no longer current, forcing a re-fetch-and-retry rather than a silent overwrite.

Dynamic per-service throttling

Rate limits are expressed as {tokens_per_second, burst} inside the same versioned rule object, so throttle changes go through the identical distribution and atomic-apply path as sampling-policy changes; there's no separate, less-safe channel for "urgent" throttle updates. Clients enforce locally with a token bucket; the server periodically rebalances per-service quotas based on reported sampled-span counts, and any resulting rate change ramps over a stated duration (start_rate, target_rate, ramp_seconds) rather than jumping, so a sudden 10x throttle loosening doesn't spike ingestion the instant it lands.

Canary rollout with an automated rollback trigger

A new rule is applied to a small canary slice of clients first. The guardrail needs a detection threshold that won't false-positive on normal noise but will catch a real regression fast. Treating the sampled-span count in the canary window as a Poisson-ish count process (a count of independent, randomly-timed events, where the natural run-to-run variability scales with the count itself), the standard error of a count $N$ (how much that count would naturally jitter from one measurement window to the next, just from randomness, not a real change) is approximately $\sqrt{N}$. To reliably detect a relative deviation $\Delta$ in the sampling rate at a $z$-sigma confidence level (how many standard errors of margin to require before trusting a deviation is real rather than noise), you need the deviation's signal ($\Delta N$) to clear $z$ standard errors ($z\sqrt{N}$):

For a conservative 3-sigma guardrail ($z=3$) tuned to catch a 20% deviation ($\Delta = 0.2$) in sampled-span rate:

So the guardrail is configured to require at least 225 sampled spans observed from the canary group before it will trust a deviation reading, which sets a minimum canary observation window rather than a fixed wall-clock delay. If the canary group is producing spans at rate $r$ spans/sec, the minimum window is $T_{min} = N / r$. For a canary receiving $r = 50$ spans/sec (1% of a 5,000 spans/sec service): $T_{min} = 225 / 50 = 4.5\text{ s}$. For a lower-traffic canary at $r = 2$ spans/sec: $T_{min} = 225 / 2 = 112.5\text{ s}$. The guardrail configuration should use the lower-bound rate across the canary's expected traffic, not the average, so it doesn't fire prematurely during a quiet period.

Rollback mechanics

Rollback is not a special code path, it's the same delta-publish mechanism republishing the previous generation as a new, higher-numbered generation (never reusing an old generation number, which would break the atomic-apply invariant). Clients apply it exactly like any other update. Every client's ack stream feeds a guardrail monitor; once the canary group's deviation crosses the threshold above, the control plane auto-publishes the rollback without waiting for a human, and pages the on-call with what fired.

flowchart LR
  Operator[Operator/Automation] --> RuleStore[Versioned Rule Store]
  RuleStore --> Publisher[Delta Publisher]
  Publisher --> Collectors[Collector/Client Fleet]
  Collectors --> AckAgg[Ack + Metrics Aggregator]
  AckAgg --> Guardrail[Guardrail Monitor]
  Guardrail -->|breach: N greater than 225, deviation greater than 20%| Rollback[Auto-publish prior generation]
  Rollback --> Publisher
  Guardrail --> Dashboard[Operator Dashboard]

Trade-offs and pitfalls

• Patch-in-place rule updates (mutating fields on an existing ruleset object) are the classic bug source here: they create a window where a concurrent read sees a half-updated rule. Immutable objects plus pointer swap eliminate that class of bug entirely.
• A too-sensitive guardrail (small $N$, small $\Delta$) will false-positive on ordinary Poisson noise and roll back good rules; a too-loose one won't catch real regressions until real damage is done. The $N \geq (z/\Delta)^2$ relationship makes that trade-off explicit and tunable rather than guessed.
• Full-snapshot fallback on generation mismatch is intentionally wasteful (a whole ruleset transfer instead of a delta) but it trades a little bandwidth for never needing distributed reconciliation logic, which is the right trade for a system whose main job is safety, not efficiency.
• Ramping rate-limit changes instead of applying them instantly avoids a self-inflicted ingestion spike, but it also means an urgent "stop the bleeding" throttle takes longer to take full effect; the ramp duration itself should be a tunable, not hardcoded.
• Clients offline for extended periods will fail every delta's base_generation check on reconnect and correctly fall back to a full snapshot; if the control plane doesn't retain enough history that's fine, since snapshot pull doesn't need history, only the current generation.

Set the policy by working backward from what each query pattern actually needs, then verify the storage savings with the arithmetic rather than picking round numbers and hoping. A reasonable concrete policy: raw metrics at native resolution for 14 days, 5-minute rollups for 1 year, hourly rollups for years 2 through 5; full traces for 30 days, then 1% sampled for the following 11 months.

Rationale by window

flowchart LR
  A[Raw ingest: native res] -->|14 days| B[Raw tier: hot]
  B -->|downsample| C[5-min tier: 1 year]
  C -->|downsample| D[Hourly tier: years 2-5]
  D -->|expire| E[Deleted]
  F[Trace ingest] -->|30 days full| G[Full trace tier]
  G -->|sample 1%| H[Sampled trace tier: 11 months]

• 14 days raw: covers essentially all incident debugging, since almost every retro or root-cause investigation happens within two weeks of the event, and alerting needs full resolution on recent data to avoid missing short spikes.
• 1 year at 5-minute rollups: supports capacity planning and seasonal comparisons (week-over-week, month-over-month) without needing per-second precision; 5 minutes is short enough to still show diurnal patterns clearly.
• Years 2-5 at hourly rollups: supports long-term trend and year-over-year growth analysis; anything finer than hourly at this age is rarely queried and expensive to keep.
• 30 days full traces: matches the raw-metrics window for the same reason, full-fidelity root cause work happens fast, and traces are the most expensive telemetry type per unit.
• 1% sampled for 11 more months: preserves enough statistical signal for "did this class of error exist a few months ago" investigations without paying for full trace volume; always retain 100% of traces tied to errors or SLO breaches regardless of the sampling rate (a fixed-percentage sample can otherwise miss the rare traces investigators actually want).

Verifying the storage savings

For 1,000,000 active series, using the same 2-bytes/compressed-raw-sample and 8-bytes/downsampled-point (4 aggregates: min, max, sum, count, at roughly 2 bytes each) assumptions used elsewhere in TSDB capacity planning:

series = 1_000_000
compressed_bytes_per_raw_sample = 2
agg_bytes_per_downsampled_point = 8

def samples(days, interval_s):
    return series * (days * 86400 / interval_s)

raw_bytes = samples(14, 15) * compressed_bytes_per_raw_sample
ds_bytes = samples(365, 300) * agg_bytes_per_downsampled_point
hourly_bytes = samples(1460, 3600) * agg_bytes_per_downsampled_point
total_tiered_bytes = raw_bytes + ds_bytes + hourly_bytes

allraw_bytes = samples(14 + 365 + 1460, 15) * compressed_bytes_per_raw_sample

Result: raw tier = 161.28 GB, 5-min tier = 840.96 GB, hourly tier = 280.32 GB, total tiered storage over the full 5-year window ≈ 1.283 TB, versus an all-raw-forever equivalent of ≈ 21.19 TB for the same window, a 16.5x reduction. The 5-minute tier dominates total storage (840 GB of the 1.28 TB) precisely because it covers the most time (1 year) at the finest surviving resolution; that's useful to know when deciding whether to push the raw window shorter or the 5-minute window's resolution coarser if the budget gets tighter.

For traces, at 10,000 traces/sec with an assumed 4 KB compressed size per trace: full 30-day retention stores about 103.68 TB, while the following 11 months at 1% sampling adds roughly 11.40 TB, so the sampled tail costs about 11% as much as the initial 30-day full window despite covering over 10x the time span.

What you can and can't still answer after each window closes

Trade-offs and pitfalls

• Percentile aggregates (p95, p99) do not survive naive downsampling: averaging five 1-minute p99 values is not the same number as the true p99 across that 5-minute window. If percentile fidelity matters at the rollup tier, you need to store enough of a histogram or sketch (not just min/max/sum/count) to recompute percentiles, which raises the per-point byte cost above the 8-byte assumption used here.
• A fixed sampling percentage for traces (1% flat) will statistically under-represent rare-but-important request types unless it's stratified or combined with the always-keep-errors/SLO-breach rule; a pure random sample optimizes for "typical" traffic, which is exactly what you don't need for debugging.
• Retention policy that isn't enforced automatically (a manual cleanup job, or "we'll get to it") tends to silently become the accidental real retention policy; tie expiry to the storage engine's native TTL/compaction mechanism rather than a side script.
• Communicating the policy to teams matters as much as setting it: if engineers don't know that a 3-week-old spike is only visible at 5-minute resolution, they'll draw wrong conclusions from a smoothed-out graph without realizing detail was lost.