Observability and Monitoring Architecture Questions

Direct answer

Break the cost into four buckets tied to concrete volumes rather than one blended number: storage (split by hot/warm/cold tier), ingest compute, egress, and query compute. Once each bucket is a formula in terms of raw ingest volume, retention days, and downsample ratio, every proposed knob (sampling, retention, downsampling, precomputing queries) is just a parameter change you can re-run through the same formulas, which is what makes the model defensible to finance: they're not trusting your intuition, they're trusting arithmetic they can re-check.

Structured elaboration

The four cost buckets and what drives each:

Retention tiers as the main storage lever: hot (short window, full resolution, most expensive per byte), warm (downsampled, medium retention, mid-cost), cold (heavily downsampled, long retention, cheapest per byte, e.g. object storage archive class). Storage cost is the sum of the three tiers, each computed from its own retention window and downsample ratio, not one blended "storage cost per byte."

Which knobs move the number, and by how much: sampling rate reduces raw ingest volume directly, which cascades into every downstream bucket. Retention shortening only affects the tier whose window changed. Downsampling more aggressively in a tier only affects that tier and everything colder than it. Query precomputation (materialized views, discussed in depth as its own system-design question in this topic) reduces query compute without touching storage or ingest at all. These don't all move the same line item, which is exactly the point when explaining trade-offs to a non-architecture stakeholder: "cut retention" and "sample harder" save different money for different reasons.

Worked example

Baseline: 1 PB/month raw ingest ($I = 1{,}000$ TB/month), three retention tiers.

Illustrative unit costs (chosen only to demonstrate how the model moves, not a vendor quote, since real pricing varies by provider, region, and contract):

• Hot storage: $0.023/GB-month. Warm: $0.010/GB-month. Cold archive: $0.0018/GB-month.
• Ingest compute: $2/TB processed. Egress: $0.05/GB. Query compute: $0.005/GB scanned.

Retention plan: hot 7 days (full resolution), warm 90 days (10x downsampled), cold 730 days (100x downsampled from raw).

$$\text{hotStorageTB} = I \cdot \frac{7}{30} = 1{,}000 \times \frac{7}{30} = 233.33 \text{ TB}$$ $$\text{warmStorageTB} = \frac{I}{10} \cdot \frac{90}{30} = 100 \times 3 = 300 \text{ TB}$$ $$\text{coldStorageTB} = \frac{I}{100} \cdot \frac{730}{30} = 10 \times 24.33 = 243.3 \text{ TB}$$ $$\text{storageCost} = 233{,}330 \times 0.023 + 300{,}000 \times 0.010 + 243{,}300 \times 0.0018 = \$8{,}804.67/\text{month}$$ $$\text{ingestCost} = 1{,}000 \times \$2 = \$2{,}000/\text{month}$$

At 5% of raw volume egressing to external dashboards/consumers:

$$\text{egressCost} = (1{,}000 \times 0.05) \times 1{,}000 \times \$0.05 = \$2{,}500/\text{month}$$

At 200,000 dashboard queries/day averaging 50MB scanned each (already benefiting from the materialized-view design, not a raw scan):

$$\text{queryCost} = 200{,}000 \times 0.05 \times 30 \times \$0.005 = \$1{,}500/\text{month}$$ $$\text{TOTAL} = \$8{,}804.67 + \$2{,}000 + \$2{,}500 + \$1{,}500 = \$14{,}804.67/\text{month}$$

Storage is 59.5% of the bill, egress 16.9%, ingest 13.5%, query 10.1%. Storage dominates, which tells the stakeholder conversation where to start.

Sensitivity: which knobs actually move the number.

(Trace sampling: assuming traces are 40% of the 1PB raw volume and are sampled down to keeping 10%, raw ingest drops to $1{,}000 - 0.4 \times 1{,}000 \times 0.9 = 640$ TB, which scales storage, ingest, and egress together since all three are proportional to $I$; query cost is held fixed because it depends on materialized-view cardinality, not raw ingest volume.)

The trace-sampling knob moves the most money here because it's the only one that reduces the ingest-side volume driving three of the four buckets at once, not just one storage tier.

Trade-offs & pitfalls

Presenting this to a stakeholder who won't read the architecture diagram: lead with the four-bucket table and the "storage is 60% of the bill" headline, then show the sensitivity table as a menu: "shorten hot retention and you save about $3,067/month but lose the ability to debug anything past 3 days at full resolution; sample traces harder and you save more, about $4,790/month, but rare-event tracing gets less reliable." Frame every knob as a cost-versus-capability trade the business is choosing, not an engineering decision made in isolation.

Common wrong turns: reporting a single blended "$/GB" number across all tiers, which hides that hot storage is over 12x more expensive per byte than cold in this model and makes every optimization conversation vaguer than it needs to be; sizing the model off total data stored ever, rather than steady-state storage per tier (a tier's storage is bounded by its own retention window, not by cumulative ingest since the platform launched); and treating cross-cloud portability as free optionality when discussing a "buy versus build" version of this trade-off: egress pricing and archive-tier retrieval fees differ enough between providers that a cost model built on one provider's numbers doesn't transfer without re-deriving the tier costs.