Estimation and Capacity Sizing Questions

Assumptions to state explicitly

• Raw daily volume: 5 TB/day (ingest before compression/filtering).
• Compression ratio: e.g., 3:1 for columnar Parquet/Snappy — affects on-disk storage and network egress.
• Ingest duplication/overhead: +10% for metadata, partition files, headers.
• Retention window: hot (30 days), warm (1 year), cold (3–7 years) — determines tiered storage sizing and cost.
• Data format & schema: JSON vs AVRO/Parquet — impacts compression and parse/compute cost.
• Query patterns: ad-hoc SQL vs scheduled batch jobs vs real-time low-latency — affects compute sizing and indexing.
• Processing model: batch micro-batch or streaming (Spark/Flink) — affects cluster sizing and autoscaling behavior.
• SLA/availability: 99.9% vs 99.99% — influences redundancy and cost.
• Growth rate: 20% monthly — projects future capacity.
• Retention of derived datasets/aggregations: whether materialized views are stored.

How each assumption affects estimates

• Compression & format directly reduce disk and storage cost; worst-case (no compression) increases storage and I/O.
• Retention window multiplies stored TB; tiering reduces cost but adds retrieval latency/cost.
• Query patterns determine VM/cluster size, memory, CPU, and whether caching is needed — ad-hoc analytics require larger interactive clusters.
• Streaming vs batch dictates sustained compute (streaming) vs spiky (batch) and influences autoscaling design.
• SLA and redundancy increase compute and storage replication costs.
• Growth rate compounds future cost and capacity planning.

Communicating uncertainty & contingency

• Present range estimates: pessimistic (no compression), expected (3:1), optimistic (5:1) with associated monthly costs.
• Call out confidence level for each assumption (high/medium/low) and key risks.
• Propose short-term contingency: enable autoscaling, use cheaper cold storage for older data, and set quota alerts.
• Propose measurement plan: run a 2-week pilot ingest to validate compression, file sizes, query latency, and update estimates.
• Recommend governance triggers: if growth or costs exceed X% of plan, execute mitigation (increase retention tiering, add sampling, enforce schema normalization).
• Provide a one-page decision table: assumption → impact → mitigation → next-check date.