Observability and Monitoring Architecture Questions

Designing and architecting end to end observability and monitoring systems that scale, remain reliable under load, and do not become single points of failure. Topics include deciding which telemetry to collect and why including metrics logs traces and events, instrumentation strategies, collection models such as push versus pull, high throughput telemetry ingestion and pipeline design, time series storage and compression, aggregation and partitioning strategies, metric cardinality and retention tradeoffs, distributed tracing propagation and sampling strategies, log aggregation and secure storage, selection of storage backends and time series databases, storage tiering and cost optimization, query and dashboard performance considerations, access control and multi tenancy, integration with deployment pipelines and tooling, and design patterns for self healing telemetry pipelines. Senior level assessments include designing scalable ingestion and aggregation architectures, storage tiering and query performance optimization, cost and operational tradeoffs, and organizational impacts of observability data.

MediumSystem Design

27 practiced

Design a metrics ingestion pipeline for a SaaS with an expected ingest rate of 200k metrics/second, retention 30 days, and interactive dashboards that require low-latency reads for the last 12 hours. Describe the components (agents, brokers, buffering, ingestion workers), partitioning strategy, aggregation points, and how you would handle spikes and backpressure.

Sample Answer

Requirements:- Ingest 200k metrics/sec, 30-day retention, low-latency reads for last 12 hours (interactive dashboards), handle bursts and backpressure.

High-level architecture:- Agents/SDKs at customers -> edge aggregator -> durable broker (Kafka) -> ingestion workers -> hot-store (TSDB for 12h) + cold-store (object store / long-term TSDB) -> query layer + aggregation service.

Components:- Agents: batch and compress metrics, add tenant ID, metric name, tags, timestamp; enforce client-side rate-limits and jitter.- Edge aggregators: run near edge to coalesce, dedupe, apply sampling/tenant quotas, and produce to Kafka.- Broker: Kafka cluster with topic per tenancy-shard or metric-type; retention short (48h) for hot path; also mirror to cold-topic for long-term storage.- Buffering: Kafka + S3 for overflow. Use Kafka retention + disk buffer on aggregators if Kafka unavailable.- Ingestion workers: consumer groups partitioned to ensure ordering per (tenant, metric) key; write to hot TSDB (e.g., Prometheus remote-write-compatible store or M3/ClickHouse) with in-memory or SSD-backed write-ahead buffers.- Aggregation points: downstream real-time aggregator service computes rollups (1s/10s/1m) into hot-store; materialized rollups for dashboards.- Query layer: API/graph layer routes recent 12h queries to hot-store (in-memory/SSD low-latency), older queries to cold-store with async compaction.

Partitioning strategy:- Partition Kafka by hash(tenant_id, metric_name) to keep related series on same partition for ordering and local aggregation. Use enough partitions to support throughput (e.g., 200k/sec, 1000 partitions -> 200 msgs/sec/partition assuming batching).

Handling spikes & backpressure:- Client-side and edge rate-limits + token-bucket; aggregators buffer and downsample non-critical series.- Kafka acts as durable buffer; scale partitions and consumers horizontally.- If Kafka is saturated, shed non-essential metrics (low-priority) and return 429 back to agents; expose quota telemetry.- Ingestion workers use backpressure to slow consumption (consumer pause), autoscale workers based on lag; use SLOs to drop oldest buffered data when disk limits reached.

Trade-offs:- Strong ordering per series vs partition skew — mitigate with rebalancing and consistent hashing.- Hot vs cold store cost: keep 12h hot for latency, cold for cost-effective 30-day retention.

Metrics & observability:- Monitor ingestion rate, consumer lag, disk usage, error rates, per-tenant quotas, and tail latencies; run chaos tests for bursts.

This design provides durable buffering, scalable partitioned ingestion, real-time aggregation for low-latency dashboards, and clear backpressure/overflow policies.

HardSystem Design

31 practiced

Design architecture for multi-tenant RBAC and data isolation in an observability backend that must allow an admin to run aggregated billing queries across tenants but prevent raw cross-tenant data access. Discuss encryption strategies, per-tenant keys, a query proxy approach, audit logging, and performance trade-offs.

Sample Answer

Requirements & constraints:- Strong tenant isolation: no raw cross-tenant reads except controlled aggregated billing.- RBAC: admins can run aggregation across tenants but must not decrypt or access raw tenant payloads.- Auditability, low-latency query performance, scalable for many tenants, and key rotation.

High-level architecture:- Ingest layer → Per-tenant processing & encrypted storage → Query proxy/service → Aggregate-only billing service → Key management & audit/logging services.

Core components:1. Per-tenant encryption at rest - Each tenant has a unique Data Encryption Key (DEK). Data (logs/metrics/traces) is encrypted with tenant DEK before persistent storage (object store / columnar DB). - DEKs are wrapped by a Key Encryption Key (KEK) stored/managed in KMS (cloud HSM or vault). KEKs are service-level keys; per-tenant DEKs allow rotation without re-encrypting global KEK.2. Key management - Use a centralized KMS (or HashiCorp Vault) to generate, rotate, and audit DEKs. Store wrapped DEKs in metadata DB; unwrapping occurs only within a short-lived secure environment (see query proxy).3. Query proxy (authorization & transformation) - All queries go through a query proxy that enforces RBAC and data-access policies. - For regular tenant users: proxy only allows queries scoped to their tenant ID and supplies the DEK-unwrapped to a trusted execution environment (TEE) or uses server-side crypto primitives to decrypt rows for that tenant. - For cross-tenant admin billing queries: proxy forbids returning raw tenant-level records. Instead it runs pre-defined aggregation primitives (counts, sums, histograms) within a trusted execution environment where the DEKs are unwrapped transiently and data is aggregated in-memory; only aggregated, non-identifiable results are returned. Optionally use homomorphic or deterministic encryption for specific fields to allow some operations without decryption.4. Trusted Execution - Use TEEs (e.g., AWS Nitro enclaves) or isolated compute nodes with strict IAM to unwrap DEKs, perform decryption, and compute aggregations. TEEs reduce the blast radius and keep plaintext out of general memory/persistent stores.5. Audit logging & provenance - Every key unwrap, query, and data access is logged immutably (append-only log, write-once storage). Logs include requester identity, tenant IDs involved, query plan, time, and whether plaintext was used. Logs are signed and stored separately (different KMS key).6. Policy & RBAC enforcement - Central policy engine (OPA) in the proxy enforces that admin roles can trigger only pre-approved aggregation types and cannot request raw records or filters that leak tenant identities.7. Performance & caching - Precompute commonly-used billing aggregates per tenant (daily/hourly rollups) in encrypted form. For cross-tenant aggregates, compute from rollups without decrypting raw events where possible. - Use columnar storage with encrypted columns for heavy aggregation fields, enabling vectorized decrypt+aggregate in TEEs. - Consider storing deterministic encrypted indices for fields needed for grouping to avoid full decryption for group-by keys (trade-off: weaker confidentiality).Trade-offs & notes:- Security vs. performance: TEEs and per-tenant decrypt-on-aggregate adds CPU and latency; mitigated by pre-aggregation and cache. Homomorphic or SSE reduces decrypt need but increases complexity and limits operations.- Key management complexity: per-tenant keys improve isolation but increase KMS operations and rotation complexity. Use lazy rotation and rewrap strategies.- Audit & compliance: Immutable, signed logs are essential for proving no raw cross-tenant access.- Developer ergonomics: Provide safe SQL surface—disallow arbitrary UDFs in aggregation path to prevent exfiltration via crafted functions.

Conclusion:Use per-tenant DEKs wrapped by KMS, enforce RBAC via a query proxy + policy engine, perform any cross-tenant aggregation inside TEEs (or tightly-controlled compute) so plaintext never leaves trusted boundaries, keep detailed signed audit logs, and optimize with precomputed rollups and encrypted indices to reduce runtime decryption. This balances isolation, verifiability, and scalable billing query performance.

MediumTechnical

25 practiced

Design liveness/readiness probes and monitoring metrics for a stateful service that may be alive but unable to perform useful work (e.g., blocked DB connection). Explain probe endpoints, probe semantics, metrics to detect partial failures, and how to avoid false positives while ensuring degraded states trigger remediation.

Sample Answer

Clarify goals: liveness should detect processes that are dead/hung and require restart; readiness should detect whether the instance can serve correct, useful work (e.g., DB access, leader-only duties). Monitoring should surface partial failures (alive but degraded) without noisy false positives.

Probe endpoints and semantics- GET /health/liveness — very cheap: process heartbeat, event loop responsiveness (e.g., mutex stall detect), memory pressure check. Should NOT do network/DB calls. If fails → orchestrator (k8s) restarts pod.- GET /health/readiness — heavier: verifies ability to perform user-facing work: - DB connection smoke-check: lightweight ping or prepared-statement execution (<50ms) - Cache connectivity and freshness - Dependent service endpoints up (fast health endpoints) - Application-specific invariants (e.g., write quorum, leader status) - Return non-ready (HTTP 503) if any critical dependency fails → orchestrator removes from service discovery / load balancer (no restart).

Probe implementation notes- Keep liveness deterministic and cheap to avoid false restarts.- Readiness should run at lower frequency or with short timeout (e.g., every 5–10s, timeout 200–500ms).- Use separate paths so orchestrator actions differ (restart vs drain traffic).

Metrics to detect partial failures- DB connection pool metrics: active, idle, wait_count, wait_time, connection_errors- DB latency and error rate (p99, p95, 5xx rates)- Request success rate and error rate (per endpoint)- Request latency histograms (p50/95/99)- Queue/backlog sizes (consumer lag)- Replication lag / leader election events- Health-check failure counts and durations- Recovery time and restart frequency

Avoiding false positives- Require N consecutive failures or sliding-window error rate (e.g., 3 failures in 6 attempts within 30s) before marking not-ready; use exponential backoff for probe retries.- Differentiate transient errors vs sustained (use short retry within probe logic for known transient issues).- Use graceful transition: when readiness fails, trigger graceful drain (stop accepting new requests, finish inflight within timeout) instead of immediate kill.- For liveness, detect real process deadlocks by monitoring event loop latency or watchdog threads; ignore transient GC pauses below configured threshold.

Remediation and automation- Automated: on readiness fail → drain + send alert; if readiness doesn’t recover within SLA window (e.g., 2 min), escalate to restart.- For DB-specific problems: if high DB error/latency, reduce concurrency (circuit breaker), shed load, switch to read-only mode, or route to standby service.- Alerts: different severities — transient (warning), sustained degraded (P1) with runbook links (restart pod, failover DB, increase pool).- Use autoscaling informed by healthy instance count and error budgets, not raw CPU.

Example thresholds (tunable)- DB ping latency >200ms or error rate >1% over 1m → readiness fail- Connection pool wait_time >100ms or wait_count high → alert- Readiness requires 2 consecutive probe failures in 30s; liveness restarts after 1 failure of watchdog interval >10s.

Summary- Keep liveness minimal and focused on process viability.- Make readiness reflect the ability to do correct work (including critical external deps).- Combine probe semantics, robust metrics, sliding-window/aggregation, and staged remediation (drain → alert → restart) to avoid false positives while ensuring degraded states trigger corrective action.

MediumTechnical

53 practiced

Implement reservoir sampling to uniformly sample k items from a very large or unbounded stream. In Python, implement a class ReservoirSampler with methods process(item) and get_sample() that maintain a uniform sample of size k using O(k) memory. Include any assumptions and expected time complexity.

Sample Answer

To maintain a uniform random sample of size k from an unbounded stream, use reservoir sampling (Algorithm R). Keep the first k items in the reservoir. For the i-th item (1-indexed) where i > k, include it in the reservoir with probability k/i by replacing a uniformly chosen existing reservoir element.

python

import random

class ReservoirSampler:
    def __init__(self, k):
        """
        k: desired sample size (int >= 1)
        """
        if k <= 0:
            raise ValueError("k must be >= 1")
        self.k = k
        self.reservoir = []
        self.n = 0  # number of items seen

    def process(self, item):
        """
        Process a new item from the stream.
        O(1) expected time per item.
        """
        self.n += 1
        if len(self.reservoir) < self.k:
            # fill initial reservoir
            self.reservoir.append(item)
            return
        # for item index n > k, include with prob k/n
        r = random.randrange(self.n)  # integer in [0, n-1]
        if r < self.k:
            # replace random index in reservoir
            self.reservoir[r] = item

    def get_sample(self):
        """
        Return current reservoir (shallow copy to avoid external mutation).
        """
        return list(self.reservoir)

Key points:- Correctness: Each seen item has equal probability k/n of being in the reservoir after n items.- Time complexity: O(1) expected per processed item; get_sample O(k).- Space complexity: O(k) memory.- Assumptions: stream items are processed one at a time; random.randrange is uniform; enough randomness available.- Edge cases: when total items seen n < k, sample size is n; ensure k>0. For reproducible tests, seed random externally (random.seed).

HardTechnical

31 practiced

For a metrics store with very high cardinality, propose indexing, partitioning and precomputation strategies that allow sub-second dashboard queries for last-24h views. Discuss materialized views, time-window partitioning, columnar storage, bloom filters, and caching at multiple layers. Explain trade-offs in storage and write amplification.

Sample Answer

Clarify target: sub-second dashboards for rolling last-24h queries over very high-cardinality metrics (many unique series, tags). Key levers: partition by time, index/tag, precompute common rollups, use columnar storage for fast scans, and multi-layer caching. Below is a practical architecture and trade-offs.

Partitioning & storage- Time-window partitioning: partition data by fixed windows (hourly or 10–15min files) so a last-24h query touches at most 24 (or 96) partitions. Smaller windows reduce read amplification for short windows but increase metadata and compaction work.- Columnar storage: store metric tuples (timestamp, metric_id, value, tags) in columnar format (Parquet/Arrow/ClickHouse MergeTree). Columnar reads only timestamp+value+selected tag columns for faster IO and vectorized aggregation.- Tag/series dictionary encoding: encode high-cardinality tag values to integers; store dictionaries per partition for space and faster joins.

Indexing- Lightweight per-partition inverted indexes on hot tag keys (e.g., tenant, metric_name). Use sparse sorted indexes or min/max statistics to skip partitions.- Bloom filters per partition for tag-value existence checks to avoid scanning partitions with no relevant series; extremely cheap memory/IO trade-off.- Secondary indexes (e.g., ClickHouse materialized indices or RocksDB SST indexes) for very selective tag queries; beware write cost.

Precomputation & materialized views- Maintain materialized aggregates for common rollups (per-minute or per-5s aggregates per series or per-tag group) written into a time-series-optimized store. Last-24h dashboards can read these aggregates rather than raw points.- Use a two-tier approach: nearline stream processing (Flink/Beam) to update real-time MV for last few hours and batch compaction for longer retention.- Pre-aggregate frequently queried dimensions (top-k, percentiles using t-digest/CKMS sketches) to reduce computation on query.

Caching at multiple layers- Client-side / edge cache for identical dashboard queries (short TTL ~10–30s).- Application/in-memory cache (LRU with eviction by time and cardinality) for compiled query results or partial aggregates.- Query-result cache in a fast store (Redis/Memcached) for heavy dashboards; invalidate/update via streaming writes.- Storage-level page/block cache and Bloom filters reduce disk seeks.

Trade-offs and write amplification- Precomputation & materialized views increase storage (multiple representations of same data) and write amplification: every raw write can trigger writes to aggregates, indexes, dictionaries, and MV sinks.- Finer partitions and more indexes reduce read latency but increase metadata, compaction overhead, and background IO.- Bloom filters and dictionaries add modest memory overhead but save large scan costs.- Real-time MV (streaming) gives low-latency reads at cost of more CPU and I/O; batching reduces write amplification but increases staleness.- Choose configurations by query patterns: if last-24h read-heavy and patternable, favor aggressive pre-agg+cache; if writes are dominant and ad-hoc queries common, favor lighter indexing and on-read aggregation.

Operational notes- Monitor write amplification metrics (IOps, bytes written/second) and storage growth.- Automate compaction and re-encoding of hot partitions.- Provide fallback: when cache miss and pre-agg missing, run bounded-time compute (parallel columnar scan with vectorized aggregation) with query timeouts.

This mix—time-window partitions, columnar storage, per-partition Bloom filters, selective inverted indexes, materialized rollups for common dimensions, and layered caching—achieves sub-second dashboards for last-24h while trading increased storage and write amplification for read latency and predictable performance.

Unlock Full Question Bank

Get access to hundreds of Observability and Monitoring Architecture interview questions and detailed answers.

Join thousands of developers preparing for their dream job.