Observability and Monitoring Architecture Questions

Designing and architecting end to end observability and monitoring systems that scale, remain reliable under load, and do not become single points of failure. Topics include deciding which telemetry to collect and why including metrics logs traces and events, instrumentation strategies, collection models such as push versus pull, high throughput telemetry ingestion and pipeline design, time series storage and compression, aggregation and partitioning strategies, metric cardinality and retention tradeoffs, distributed tracing propagation and sampling strategies, log aggregation and secure storage, selection of storage backends and time series databases, storage tiering and cost optimization, query and dashboard performance considerations, access control and multi tenancy, integration with deployment pipelines and tooling, and design patterns for self healing telemetry pipelines. Senior level assessments include designing scalable ingestion and aggregation architectures, storage tiering and query performance optimization, cost and operational tradeoffs, and organizational impacts of observability data.

MediumTechnical

37 practiced

Explain common time-series compression techniques used by modern TSDBs (delta-of-delta, Gorilla encoding, run-length encoding, dictionary encoding). For each technique, describe when it is most effective, how it affects write and read performance, and implications for storage and CPU trade-offs.

Sample Answer

** Delta-of-delta (timestamps)**- What: store initial timestamp, then differences between deltas (t2−t1, t3−t2), often variable-length.- Most effective when: regular or near-regular sampling (IoT, monitoring metrics).- Write/read perf: cheap to encode on write (simple arithmetic), fast decode on read sequentially; random seeks require reconstructing prefix.- Storage/CPU trade-off: very high compression for stable intervals with minimal CPU overhead; less effective for irregular timestamps.

** Gorilla (value + timestamp bit-packing)**- What: uses XOR of successive values, leading-zero/length encoding; combined with delta/delta for timestamps.- Most effective when: floating-point numeric metrics with temporal locality (CPU, latency).- Write/read perf: low CPU at write, very compact blocks speed I/O; decoding slightly more CPU but cache-friendly for sequential scans.- Trade-offs: excellent storage reduction and read throughput; complex implementation for partial reads.

** Run-length encoding (RLE)**- What: collapse consecutive identical values into (value, count).- Most effective when: long stretches of unchanged values (status flags, sparse metrics).- Write/read perf: minimal write CPU; reads that can operate on runs are faster; random access less efficient because need run boundaries.- Trade-offs: trivial CPU cost, huge storage gains for plateaus; useless for high-cardinality or noisy series.

** Dictionary encoding**- What: map recurring values to short IDs; store IDs instead of full values.- Most effective when: categorical labels or repeated strings (logs, tags).- Write/read perf: encoding requires lookup/update (some CPU), very fast reads once dictionary cached.- Trade-offs: memory for dictionary, benefits depend on repeat rate; dictionary maintenance across segments adds complexity.

Operational guidance for systems engineers:- Combine techniques (Gorilla + RLE + dict) per column. Balance CPU vs I/O: more aggressive encoding reduces storage/I/O but increases encode/decode CPU—tune based on disk bandwidth, CPU headroom, and query patterns (sequential scans vs random point reads).

EasyTechnical

35 practiced

Explain push vs pull telemetry collection models. Describe three operational trade-offs (service discovery, firewall/network concerns, resource overhead) and give concrete examples (Prometheus pull, StatsD/Pushgateway push). For a Kubernetes environment with many short-lived pods, which model do you prefer and why?

HardTechnical

28 practiced

Design logging and audit storage to meet regulatory requirements: encryption at rest and in transit, append-only tamper-evidence, retention/deletion workflows, role-based access, and audit trails. Describe specific controls (KMS keys, WORM storage, signed manifests) and how to demonstrate these controls during SOC/ISO audits.

Sample Answer

**High-level approach**Design logs/audit storage as encrypted, immutable, access-controlled, and auditable. Use layered controls: transport encryption, KMS-managed keys, WORM storage for immutability, RBAC + separation of duties, signed manifests and hash chains for tamper-evidence, plus documented retention/deletion workflows and monitoring.

**Controls (specifics)**- Encryption in transit: TLS 1.2+ for agents -> collectors -> storage; mTLS for collector endpoints.- Encryption at rest: Customer-managed KMS CMKs (HSM-backed). Use envelope encryption; enable key rotation and deny plaintext decrypt without CMK policy.- Immutability: S3 Object Lock in Governance/Compliance mode or vendor WORM appliance; enable retention periods and Legal Hold.- Tamper-evidence: Generate daily signed manifests of log bundles containing SHA-256 hashes; sign manifests with an HSM-held signing key (KMS asymmetric key) and publish signature to separate integrity store and CI/CD artifact repo.- Append-only ingestion: Use write-once streams (e.g., Kafka with immutable topics or append-only S3 put-only pattern) and disable object overwrite.- RBAC & separation: IAM roles with least privilege, ABAC tags, MFA for privileged ops, dedicated "log-admin" vs "security-analyst" roles; require approval workflows for key usage or delete.- Retention/deletion workflows: Automated lifecycle policies tied to retention windows; deletion requires dual-approval and documented justification; Legal Hold overrides automated deletion.- Monitoring & alerting: CloudTrail for key usage, storage access, and config changes; SIEM rules detect unauthorized delete/modify attempts.

**Demonstrating controls in audits (SOC/ISO)**- Provide system design diagrams, policy documents (KMS key policy, retention policy, access control matrix), runbooks.- Produce evidence artifacts: KMS access logs, CloudTrail entries showing decrypt/sign operations, key rotation schedule and reports, S3 Object Lock configuration screenshots, signed manifest files and verification script outputs (show signature verification and hash chain), SIEM alerts and incident tickets.- Perform and record integrity tests: periodic checksum verification, restore tests from immutable store, and documented change-control approvals for any retention change.- Map controls to SOC/ISO clauses in a control matrix (e.g., KMS+CloudTrail -> CC6.1/ISO 27001 A.10/A.12) and supply auditor-friendly queries/queries scripts to reproduce evidence.

**Why this works**Combines cryptographic protection, immutable storage, procedural controls, and demonstrable logs—meeting regulatory expectations for confidentiality, integrity, availability, and evidentiary proof during audits.

MediumSystem Design

32 practiced

Design a telemetry ingestion pipeline that accepts metrics, logs, and traces from a global fleet: 100k metrics/sec, 50k traces/sec, 50k log lines/sec. Outline components (agents/collectors, ingress gateways, queueing/brokers, batching, transformers), fault tolerance, buffering/backpressure strategies, and how you would provide multi-region ingestion with durable storage and eventual global queryability.

Sample Answer

**High-level approach**Design a resilient, scalable pipeline with lightweight agents at edge, regionally distributed ingress/collector clusters, durable brokered buffering (Kafka), per-signal transformers & batchers, regional durable object stores, and global query federation (Thanos/Tempo-style).

**Components**- Edge agents: Fluent Bit / OpenTelemetry collector sidecars on hosts/k8s DaemonSets — do local aggregation, sampling, debit-based batching, per-stream backpressure.- Regional ingress gateways: Envoy/nginx + autoscaled OTel collectors terminating TLS, performing auth/rate-limiting.- Brokers/queueing: Kafka clusters per-region (RF=3) for durable write-ahead, partitioned by customer/tenant + signal.- Transformers & batchers: Consumer groups for metrics/logs/traces that batch, transform, enrich, then write to long-term stores.- Storage/Databases: - Metrics: Cortex/Thanos remote-write pipeline storing compacted chunks in S3-compatible object store. - Traces: Tempo/Jaeger-style ingest storing compressed trace chunks in object store. - Logs: Append to object store + index in OpenSearch for hot queries.

**Backpressure & buffering**- Agent-side buffering on disk (configurable size) + token-bucket rate limiter per-tenant.- Ingress applies token/circuit-breaker and returns 429 with Retry-After; agents implement exponential backoff and durable on-disk queues.- Kafka provides durable buffer; consumer lag metrics drive autoscaling.- Batch sizes adapt by latency/volume; e.g., metrics: 1-5s batches, traces larger batches tuned to size.

**Fault tolerance**- Regionally replicated Kafka (RF=3), ZK/kraft HA, monitor lag/ISR.- Stateless collectors + autoscaling + health checks.- Write-through to S3-compatible storage (multi-AZ); background compactor/repair jobs.- Cross-region disaster: async replication of object-store buckets + Kafka MirrorMaker 2 for topic-level replication (or storage-tiered replication).

**Multi-region & global queryability**- Ingest primarily to nearest-region Kafka + local durable object store.- Periodic async replication: - For long-term: replicate object-store buckets to a Global bucket or cross-region copies. - For stream-level: MirrorMaker 2 replicates critical topics to a central region.- Query federation: - Metrics: Thanos Querier with StoreAPI pointing to regional Thanos Store gateways + global bucket for historical data. - Traces: Tempo queriers federating across regional indexers/object stores. - Logs: Global search index built from replicated indexes or query fan-out to regional OpenSearch clusters with rollups.- Consistency: eventual; metadata timestamps + region tags allow de-duplication and deterministic conflict resolution.

**Operational considerations**- Multi-tenant isolation via topic/tenant shards, quota enforcement, per-tenant retention.- Observability: monitor consumer lag, ingress latency, agent queue sizes, error rates.- Security: mTLS between agents/gateways, KMS-encrypted storage, IAM-based topic access.- Cost/Trade-offs: Nearline replication increases bandwidth/cost; choose selective replication for high-value tenants/signals.

MediumTechnical

28 practiced

Given a service that produces high-cardinality metrics and a business requirement to retain 13 months of data at decreasing resolution, compare trade-offs and fit for Prometheus+Thanos/Cortex, VictoriaMetrics, and TimescaleDB. Discuss compression, query latency for long-range queries, operational overhead, tenant isolation, and cost implications.

Sample Answer

**Approach — high level**I’d compare each option across the requested dimensions and give a recommendation for a high-cardinality service needing 13‑month retention with downsampling.

**Prometheus + Thanos/Cortex**- Compression: Prometheus TSDB + Thanos blocks use chunked WAL and TSDB compression (good for metrics), Cortex similar; storage efficient but not as dense as columnar DBs.- Long-range query latency: Thanos Querier is optimized for downsampled blocks but cross-cluster queries and compacted block scans can be slower for very wide cardinality.- Operational overhead: Moderate–high. Requires HA Prometheus, object storage, compactor, querier, store-gateway; Cortex increases complexity.- Tenant isolation: Strong in Cortex (multi-tenant); Thanos less multi-tenant by default, needs extra isolation.- Cost: Moderate storage cost (object store) + operational engineer cost.

**VictoriaMetrics**- Compression: Excellent single-node and clustered compression optimized for high-cardinality.- Long-range query latency: Very good for long-range aggregated queries due to efficient storage layout and downsampling.- Operational overhead: Lower than Cortex/Thanos (simpler architecture), easier scaling with VMInsert/VMSelect.- Tenant isolation: Built-in multi-tenancy in enterprise; OSS supports labels but isolation is weaker.- Cost: Lower total cost for storage and infra; less ops burden.

**TimescaleDB**- Compression: Very strong columnar compression for historical data; hypertables + compressed chunks work well.- Long-range query latency: Great for aggregated queries with proper indices and continuous aggregates; can struggle with extremely high cardinality unless pre-aggregated.- Operational overhead: Moderate — PostgreSQL ops, backups, tuning; schema management required.- Tenant isolation: Good via DB roles/schemas; multi-tenant at app layer recommended.- Cost: Higher CPU/IO cost for writes at scale; storage cost low due to compression but compute may increase.

**Recommendation**For very high-cardinality metrics with long retention and need for low ops overhead: VictoriaMetrics (clustered/VM) often gives best cost/perf trade-off. If strict multi-tenancy and PromQL compatibility across many teams is required and you already run Prometheus at scale, Cortex (or Thanos with careful isolation) is viable but costs more to operate. Choose TimescaleDB if you need complex SQL joins, relational context, and best compression for analytical queries, but plan for pre-aggregation and larger write/compute resources.

Unlock Full Question Bank

Get access to hundreds of Observability and Monitoring Architecture interview questions and detailed answers.

Join thousands of developers preparing for their dream job.