Observability and Monitoring Architecture Questions

Direct answer

Break the cost into four buckets tied to concrete volumes rather than one blended number: storage (split by hot/warm/cold tier), ingest compute, egress, and query compute. Once each bucket is a formula in terms of raw ingest volume, retention days, and downsample ratio, every proposed knob (sampling, retention, downsampling, precomputing queries) is just a parameter change you can re-run through the same formulas, which is what makes the model defensible to finance: they're not trusting your intuition, they're trusting arithmetic they can re-check.

Structured elaboration

The four cost buckets and what drives each:

Retention tiers as the main storage lever: hot (short window, full resolution, most expensive per byte), warm (downsampled, medium retention, mid-cost), cold (heavily downsampled, long retention, cheapest per byte, e.g. object storage archive class). Storage cost is the sum of the three tiers, each computed from its own retention window and downsample ratio, not one blended "storage cost per byte."

Which knobs move the number, and by how much: sampling rate reduces raw ingest volume directly, which cascades into every downstream bucket. Retention shortening only affects the tier whose window changed. Downsampling more aggressively in a tier only affects that tier and everything colder than it. Query precomputation (materialized views, discussed in depth as its own system-design question in this topic) reduces query compute without touching storage or ingest at all. These don't all move the same line item, which is exactly the point when explaining trade-offs to a non-architecture stakeholder: "cut retention" and "sample harder" save different money for different reasons.

Worked example

Baseline: 1 PB/month raw ingest ($I = 1{,}000$ TB/month), three retention tiers.

Illustrative unit costs (chosen only to demonstrate how the model moves, not a vendor quote, since real pricing varies by provider, region, and contract):

• Hot storage: $0.023/GB-month. Warm: $0.010/GB-month. Cold archive: $0.0018/GB-month.
• Ingest compute: $2/TB processed. Egress: $0.05/GB. Query compute: $0.005/GB scanned.

Retention plan: hot 7 days (full resolution), warm 90 days (10x downsampled), cold 730 days (100x downsampled from raw).

At 5% of raw volume egressing to external dashboards/consumers:

At 200,000 dashboard queries/day averaging 50MB scanned each (already benefiting from the materialized-view design, not a raw scan):

Storage is 59.5% of the bill, egress 16.9%, ingest 13.5%, query 10.1%. Storage dominates, which tells the stakeholder conversation where to start.

Sensitivity: which knobs actually move the number.

(Trace sampling: assuming traces are 40% of the 1PB raw volume and are sampled down to keeping 10%, raw ingest drops to $1{,}000 - 0.4 \times 1{,}000 \times 0.9 = 640$ TB, which scales storage, ingest, and egress together since all three are proportional to $I$; query cost is held fixed because it depends on materialized-view cardinality, not raw ingest volume.)

The trace-sampling knob moves the most money here because it's the only one that reduces the ingest-side volume driving three of the four buckets at once, not just one storage tier.

Trade-offs & pitfalls

Presenting this to a stakeholder who won't read the architecture diagram: lead with the four-bucket table and the "storage is 60% of the bill" headline, then show the sensitivity table as a menu: "shorten hot retention and you save about $3,067/month but lose the ability to debug anything past 3 days at full resolution; sample traces harder and you save more, about $4,790/month, but rare-event tracing gets less reliable." Frame every knob as a cost-versus-capability trade the business is choosing, not an engineering decision made in isolation.

Common wrong turns: reporting a single blended "$/GB" number across all tiers, which hides that hot storage is over 12x more expensive per byte than cold in this model and makes every optimization conversation vaguer than it needs to be; sizing the model off total data stored ever, rather than steady-state storage per tier (a tier's storage is bounded by its own retention window, not by cumulative ingest since the platform launched); and treating cross-cloud portability as free optionality when discussing a "buy versus build" version of this trade-off: egress pricing and archive-tier retrieval fees differ enough between providers that a cost model built on one provider's numbers doesn't transfer without re-deriving the tier costs.

Default to logical isolation (shared infrastructure with hard per-tenant quotas and QoS enforcement) for the bulk of tenants, and offer physical isolation (dedicated shards or node pools) as an explicit, metered upgrade path for tenants whose usage or SLA requirements outgrow what shared quotas can safely guarantee. The isolation model and the migration path are two sides of the same design.

Architecture

flowchart LR
  A[Tenant Requests] --> B[Ingress: Auth and RBAC]
  B --> C[Per-Tenant Shard / Queue]
  C --> D[Shared Ingestion Pool]
  C --> E[Dedicated Ingestion Pool]
  D --> F[Query Gateway: QoS Scheduler]
  E --> F
  F --> G[Shared Query Compute]
  F --> H[Dedicated Query Compute]
  I[Billing / Quota Manager] --> B
  I --> F

• Logical isolation: per-tenant partitions/queues on shared compute, enforced with token-bucket rate limits at ingress and query-time concurrency caps; cheapest, and sufficient for the majority of tenants whose usage is well within their quota most of the time.
• Physical isolation: dedicated shard, node pool, or account for a tenant; strongest guarantee, but the operational and cost overhead of running fully separate infrastructure per tenant doesn't scale to hundreds of tenants, so it has to be selective.
• Query-level QoS: priority classes (interactive dashboard queries vs. batch/backfill queries), per-tenant concurrency limits, and admission control that sheds low-priority load before it degrades everyone; this is what actually prevents a noisy tenant's expensive query from starving others on shared compute, since ingestion isolation alone doesn't protect the read path.
• Billing-aware quotas: map each tenant's plan tier to a concrete ingest-rate and query-concurrency quota; soft-limit warnings before hard throttling, and an explicit overdraft/pay-as-you-go path rather than a silent hard cutoff. Retention is part of the same per-tenant contract, not a platform-wide constant: a tenant's plan tier should set its own retention window (e.g., 7 days on a shared/basic tier vs. 90 days on a dedicated tier), enforced as tenant-scoped TTL policy in the storage layer so one tenant's longer retention SLA doesn't force everyone else to pay for the same window.

Sizing the admission-control headroom

The core quantitative question for logical isolation is: how much burst capacity can the shared pool actually absorb before a legitimate burst from one tenant risks starving others? Take a platform with total ingest capacity $C = 2{,}000{,}000$ samples/sec shared across $N = 500$ tenants, where baseline quotas are provisioned to consume a target fraction $u$ of total capacity (leaving headroom for bursts), and tenants are allowed to burst up to $m\times$ their baseline:

If a fraction $f$ of tenants burst simultaneously while the rest sit at baseline, total load must stay under capacity:

Substituting $\text{baseline}_{\text{tenant}} = uC/N$ and simplifying:

With $u = 0.6$ (provision baseline to consume 60% of capacity, leaving 40% headroom) and $m = 5$ (allow a 5x burst):

C, N, u, m = 2_000_000, 500, 0.6, 5
baseline = (u * C) / N          # 2,400 samples/sec/tenant
burst = m * baseline            # 12,000 samples/sec/tenant
f_max = (1/u - 1) / (m - 1)     # 0.1667
max_bursting = f_max * N        # 83.3 tenants

Result: baseline quota is 2,400 samples/sec/tenant, burst allowance is 12,000 samples/sec/tenant, and up to about 16.7% of tenants (roughly 83 of 500) can burst simultaneously at 5x without exceeding total capacity. Plugging $f_{max}$ back into the original inequality confirms it lands exactly at capacity (2,000,000 samples/sec), which is the check that the derivation is self-consistent. This is the number that should actually drive the admission controller's global burst budget, not a guess: if more than ~83 tenants try to burst at once, the controller has to start denying or queuing burst requests rather than granting them all.

Migrating a tenant from shared to dedicated

1. Trigger: sustained usage consistently near quota (not just occasional bursts), or an explicit SLA purchase requiring guaranteed isolation.
2. Provision dedicated shard/node pool ahead of cutover.
3. Dual-write or replicate the tenant's recent data into the new dedicated shard while it's still live on the shared pool.
4. Cut over routing at the control plane (ingress rules keyed on tenant ID) once the dedicated shard is caught up; this should be a routing change, not a data migration event, so it can be near-zero-downtime.
5. Decommission the tenant's shared-pool footprint after a verification window, and keep the cutover reversible in case the dedicated shard has an unexpected issue.

Trade-offs and pitfalls

• Sizing baseline quotas at $u$ close to 1.0 (using nearly all capacity for guaranteed baseline) leaves almost no burst headroom, which defeats the purpose of a shared pool; the $u$ vs. burst-headroom trade-off above should be an explicit, revisited decision, not a default.
• Query-level QoS is often skipped because ingestion isolation feels like "the isolation problem," but an expensive ad-hoc query from one tenant can degrade shared query compute even when every tenant's ingestion is perfectly isolated; both paths need protection independently.
• A migration path that isn't reversible (no fallback if the dedicated shard has a problem post-cutover) turns a capacity upgrade into a risk event; always keep the shared-pool footprint alive through a verification window.
• Billing-aware quotas without a clear soft-limit warning stage turn every quota breach into a support ticket; the graduated response (warn, throttle, then hard-limit) matters as much as the quota number itself.

Direct Answer

DaemonSet agents are the default for anything needed uniformly across every node (host and container runtime metrics, general log collection): one process per node, flat overhead, and centrally managed config. Sidecars earn their much higher per-instance cost only when a specific pod needs isolated, per-app processing a shared node agent cannot provide. Centralized collectors minimize total resource overhead but add a network hop and a central point of configuration and, if not run with real replication, a central point of failure.

Structured Elaboration

Topology comparison

flowchart LR
  subgraph DS["DaemonSet Pattern"]
    NODE["Node"] --> AGENTD["DaemonSet Agent"]
  end
  subgraph SC["Sidecar Pattern"]
    POD["Pod"] --> SIDE["Sidecar Container"]
  end
  subgraph CC["Centralized Pattern"]
    APPS["App Pods"] --> GATEWAY["Central Collector"]
  end
  AGENTD --> BACKEND[("Backend")]
  SIDE --> BACKEND
  GATEWAY --> BACKEND

Worked Example

Assume a 200-node cluster averaging 15 pods/node, so 3,000 pods total.

DaemonSet: one agent pod per node, each requesting 100m CPU / 128Mi memory (a stated sizing assumption):

Sidecar: one lean sidecar per pod, each requesting half the DaemonSet's per-instance footprint, 50m CPU / 64Mi memory (an assumed leaner-per-instance sizing, still multiplied by far more instances):

That is 7.5x the DaemonSet's CPU and memory footprint, for the same cluster:

(the ratio matches exactly because per-instance overhead was assumed at half the DaemonSet's, times 15 pods/node, giving $15 \times 0.5 = 7.5$).

Centralized collector: 5 replicas, each provisioned heavier since it aggregates fleet-wide (2 vCPU / 4Gi each, a stated assumption):

Half the DaemonSet's CPU and 1/15th the sidecar's, at the cost of a network hop and, with only 5 replicas, meaningfully more disruption if two of them go down at once than losing two of 200 DaemonSet pods.

Trade-offs and Pitfalls

For log collection specifically (folded in from the log-focused variant of this comparison), sidecar and DaemonSet differ in a way the metrics comparison above does not capture: a DaemonSet log shipper reads container log files from the node's filesystem, and a very short-lived pod (a fast-completing Job) can be garbage-collected and its logs rotated away before the node-level shipper gets to them. A sidecar, tied to the same pod lifecycle, can use a termination hook to flush its buffer before the pod actually exits, which is a real advantage for ephemeral workloads even though it costs far more in steady-state resource overhead.

The 7.5x resource multiplier for sidecars is not a fixed law, it is a direct consequence of pods-per-node (15 in this example). A cluster running fewer, larger pods per node (say 4 pods/node) would show a much smaller sidecar penalty (2x instead of 7.5x at the same per-instance assumption), so this trade-off should be recalculated against the actual cluster's pod density, not assumed to generalize.

Centralized collectors concentrate risk: with only 5 replicas instead of 200 independent DaemonSet pods, losing 2 replicas to a bad node or a bad deploy is a much bigger fraction of total capacity. Run centralized collectors with genuine multi-AZ spread and a PodDisruptionBudget, not just multiple replicas on the same failure domain.

Direct answer

Collect a time series of used capacity per volume, for example daily df samples or whatever your monitoring system already stores, fit a trend line to it, and project forward to when it crosses your action threshold. You need enough history to smooth out noise (weekly patterns, one-off cleanups) but recent enough to reflect current growth, and the output should be a date, not just a percentage, because a date is what triggers a purchase order.

Structured elaboration

Collect at least daily df-style samples of used GB per volume, not just percent, since percent alone hides how many GB a jump actually represents on a large disk. Fit a trend line (ordinary least squares is enough for this) to get a growth rate in GB per day, then project forward to find the day the fit crosses your capacity ceiling.

What triggers the purchase order: pick a lead time longer than your procurement cycle. If buying and provisioning new storage takes 3 weeks, trigger the order when the trend crosses "90% full in 5 weeks," not when the disk is already at 90%.

Worked example

Executed example using 14 days of sampled disk usage on a 500 GB volume, fitting a least-squares line and projecting forward:

measured samples (GB used): [350.5, 350.4, 355.1, 357.4, 355.1, 358.0, 362.2,
                              363.5, 366.0, 366.7, 371.3, 373.2, 374.1, 377.7]
fitted growth rate: 2.105 GB/day (true underlying rate in this simulation was 2.0 GB/day)
fitted intercept (day 0 est.): 349.3 GB
90% full (450 GB) projected at day 47.9 from day 0
days remaining from today (day 13) to the 90% mark: 34.9 days

The fit (ordinary least squares) recovers the true 2.0 GB/day growth rate closely, 2.105 estimated, even with daily measurement noise. That is the point: a single day's jump or dip should not be read as a trend change, the line across many days is what you act on.

Trade-offs and pitfalls

• A straight-line fit assumes linear growth; a service about to onboard a large new customer or double its retention window will blow past a linear projection, so pair the trend with awareness of planned changes, not just historical data.
• Too little history (a few days) makes the slope noisy and unreliable; too much history (a year) can hide a recent acceleration by averaging it away. Recompute on a rolling window, for example the trailing 30 days, and re-evaluate weekly.
• An aggregate trend across 200 servers hides the one server about to fill up next week; you want both a fleet-wide summary for planning and a per-host projection for the "who pages tonight" question.

What the interviewer probes next

They will usually push on whether the fit is really linear or whether a single unusual day, a big import, a bulk cleanup, is quietly steering it, and on how you would roll a whole fleet of these projections into something a non-technical stakeholder can act on without reading a chart.

Direct Answer

Standardize on the official OpenTelemetry SDK for each language, default to auto-instrumentation everywhere for fast, uniform baseline coverage, and add manual spans only where auto-instrumentation genuinely can't see the operation that matters (a business-critical internal function call, not just "more detail everywhere"). Stage the rollout service by service behind a validation gate that measures actual trace completeness, not a fixed timeline, so you cut over only once you can show cross-service context propagation is actually working.

Structured Elaboration

Rollout topology

flowchart LR
  SDK["Language SDK + Auto-Instrumentation"] --> AGENT["Collector Agent"]
  AGENT --> GATEWAY["Central Collector"]
  GATEWAY -->|"shadow period"| LEGACY[("Legacy Tracer Backend")]
  GATEWAY -->|"shadow period"| NEW[("OTel Backend")]
  VALIDATOR["Completeness Validator"] --> NEW
  VALIDATOR -->|">=99% gate"| CUTOVER{"Cutover Decision"}
  CUTOVER -->|"pass"| NEWONLY["Remove Legacy Tracer"]

SDK selection
Use the official OpenTelemetry SDK per language (opentelemetry-java, opentelemetry-js, opentelemetry-python), pinned to a stable release, with a shared, agreed set of resource attributes (service.name, environment, version) so traces from different languages line up consistently in the backend.

Auto-instrumentation versus manual spans
Auto-instrumentation (the Java agent, Node's and Python's official auto-instrumentation packages) covers standard frameworks (HTTP servers and clients, common database drivers, messaging clients) with zero code changes, and should be the default everywhere. Add manual spans only around business-specific operations the auto-instrumentation has no way to know about, an internal pricing calculation, a multi-step order-fulfillment sequence, where the span boundary itself carries meaning auto-instrumentation cannot infer from a generic library call.

Collector configuration
Route all languages' output to a common collector layer (agent plus gateway, per the topology reasoning covered elsewhere in this topic) so the sampling policy, enrichment, and export destination are configured once, centrally, rather than per-language.

Initial sampling policy
Start conservative: a head-based baseline rate (5% is a reasonable starting point, a stated design choice, not a fixed rule) plus an always-sample-on-error override, so early rollout gets enough volume for trend visibility without full ingestion cost, while never missing an actual failure.

Staging the rollout with a validation gate
Roll out non-critical services first, run old and new tracing in parallel (shadow mode) rather than cutting over immediately, and measure context-propagation completeness: the fraction of requests where a trace's spans correctly link across every service hop it touched. Only cut a service over, removing the legacy tracer, once completeness clears an explicit threshold.

Worked Example

Sampling volume. At a 5% head-based rate on a service handling 2,000 requests/sec:

Enough for meaningful latency-trend and error-rate visibility without paying to ingest the full 2,000/sec.

Completeness gate. Define the cutover gate as at least 99% context-propagation completeness. During a canary window, a 3-hop request path (Java gateway to Node BFF to Python backend) generates 10,000 sampled requests, and the validator finds that 9,830 of them have a fully linked trace across all 3 hops:

That is below the 99% gate by 0.7 percentage points ($99\% - 98.3\% = 0.7\text{pp}$), so this service stays in shadow mode rather than cutting over. In absolute terms, 170 requests lack a fully linked trace ($10{,}000 - 9{,}830 = 170$); expressed as a share of the whole sample that is 1.7% ($170 / 10{,}000 = 1.7\%$), a different quantity from the 0.7 percentage-point gap to the gate itself, since the 1.7% figure measures the shortfall from 100% completeness, not from the 99% threshold. The next step is investigating those specific 170 requests for where propagation broke, commonly a specific async boundary (a queue hop, a background job) that isn't carrying trace headers, rather than assuming the whole pipeline is unreliable.

Trade-offs and Pitfalls

Auto-instrumentation is fast to roll out but noisy by default: it instruments every framework call it recognizes, which can produce spans nobody asked for and drive up cardinality and cost. Pair the rollout with an explicit review of what auto-instrumentation is producing per service before declaring it fully adopted, not just after.

A fixed rollout timeline (cut over service X by date Y) creates pressure to declare success before completeness is actually validated. Gating cutover on a measured completeness threshold, as above, is slower up front but avoids silently shipping broken traces that only get noticed the first time someone needs a trace during an actual incident and it's missing a hop.

Context propagation across async boundaries (message queues, background jobs, batch processing) is the most common place this rollout finds gaps, since HTTP and gRPC auto-instrumentation handle synchronous propagation well, but a queue message needs its trace context explicitly attached to the message and re-extracted on the consumer side, something auto-instrumentation for the queue client library may or may not do out of the box depending on the library.