For long-retention log storage, the honest default is a tiered combination: ClickHouse (or an object-storage-plus-index approach) for the bulk of retained volume, with Elasticsearch reserved for the recent window where full-text ad-hoc search actually matters. Picking Elasticsearch as the sole backend for everything gets expensive fast at long retention; picking object-storage-only sacrifices interactive search on recent data.
What each backend is actually optimized for
mermaid
flowchart LR
L[Log Producers] --> P[Ingestion Pipeline]
P --> ES[Elasticsearch: inverted index + shards]
P --> CH[ClickHouse: columnar table]
P --> OSS[Object Storage: Parquet blocks]
OSS --> IDX[Secondary Index]
ES --> Q[Ad-hoc Query]
CH --> Q
IDX --> Q
| Dimension | Elasticsearch | ClickHouse | Object storage + index |
|---|
| Ad-hoc search performance | Best: inverted index gives sub-second full-text and fuzzy search, native relevance scoring | Good for structured/columnar filters and aggregations, weaker for free-text (needs token tables or a bolted-on search layer) | Weakest without a fresh index; latency depends on how current the index is and whether it covers the field you need |
| Ingestion throughput | Moderate; bulk API helps but refresh interval, replication, and inverted-index build cost CPU per document | High: columnar writes, batched inserts, background merges are cheap per row | Highest for raw storage (append/upload is nearly unbounded); the indexing pipeline, not the storage, is the bottleneck |
| Cost at long retention | Highest: replication plus inverted-index/doc-values overhead multiplies stored bytes well beyond raw | Low: strong columnar compression | Lowest: cheapest byte-for-byte, and you can defer indexing entirely for cold data |
| Schema flexibility | Schema-on-write with dynamic mapping; flexible for JSON but mapping explosions are a real failure mode | Schema-on-write, columnar; wide/nested tables work but schema evolution needs explicit migration | Most flexible: store raw JSON as objects, apply schema at query time |
| Operational overhead | High: shard sizing, heap/GC tuning, index lifecycle management | Moderate: merge/partition tuning, but fewer distinct failure modes than an ES cluster at the same scale | Low for the storage tier itself; complexity shifts to whatever indexing/query-engine pipeline you build on top |
Worked cost comparison at 30-day retention
Take 50,000 log events/sec, averaging 1 KB raw per event, retained 30 days:
raw bytes=50,000×1,000×86,400×30=1.296×1014 bytes=129.6 TB
For Elasticsearch, assume a replication factor of 2 (one primary plus one replica, the normal HA baseline) and an inverted-index/doc-values overhead multiplier of roughly 1.3x over the raw JSON (a labeled assumption, not a measured constant, since it depends on mapping and field count):
ES stored=129.6×2×1.3=336.96 TB
For ClickHouse, assume an 8x columnar compression ratio versus raw JSON (a labeled assumption; typical for structured log fields with LZ4/ZSTD codecs) and the same replication factor of 2 for HA:
CH stored=8129.6×2=32.4 TB
For object storage plus index, assume 10x compression from columnar Parquet with ZSTD (labeled assumption) and no replication multiplier (the object store's own erasure coding already provides durability), plus a 5% secondary-index overhead:
OSS+IDX stored=10129.6×1.05=13.61 TB
So at this volume and retention, Elasticsearch stores roughly 10.4x more data than ClickHouse and about 24.8x more than object-storage-plus-index for the identical raw input, purely from replication and index overhead, before you've even paid for the compute to keep those indices hot.
Recommendation and trade-offs
- Use Elasticsearch for a short hot window (days, not months) where engineers are actively doing free-text incident search; keep index count and field mapping disciplined to avoid mapping explosions.
- Use ClickHouse (or a managed equivalent) as the default long-retention backend when most queries are structured filters plus aggregation (by service, level, status code) rather than free-text search; this is the common case for "how many 5xxs did service X throw last Tuesday."
- Use object-storage-plus-index only when queries are rare, tolerate higher latency, and you want the lowest possible cost floor for compliance-driven retention (e.g., 1-year audit logs nobody expects to search interactively).
- The trap senior candidates catch and juniors miss: this isn't a single-backend decision. A hot/warm/cold tiering strategy (ES or a fast index for the last few days, ClickHouse for weeks to months, object storage for the long tail) captures the strengths of each without paying any one system's worst-case cost or latency for the whole retention window.
- Schema flexibility cuts both ways: ES's dynamic mapping feels convenient until an unbounded field (a stray high-cardinality tag) blows up your mapping and cluster health; object storage defers that cost to query time, which is safer for ingestion but slower for discovery.
