Observability and Monitoring Architecture Questions
Designing and architecting end to end observability and monitoring systems that scale, remain reliable under load, and do not become single points of failure. Topics include deciding which telemetry to collect and why including metrics logs traces and events, instrumentation strategies, collection models such as push versus pull, high throughput telemetry ingestion and pipeline design, time series storage and compression, aggregation and partitioning strategies, metric cardinality and retention tradeoffs, distributed tracing propagation and sampling strategies, log aggregation and secure storage, selection of storage backends and time series databases, storage tiering and cost optimization, query and dashboard performance considerations, access control and multi tenancy, integration with deployment pipelines and tooling, and design patterns for self healing telemetry pipelines. Senior level assessments include designing scalable ingestion and aggregation architectures, storage tiering and query performance optimization, cost and operational tradeoffs, and organizational impacts of observability data.
EasyTechnical
35 practiced
List and justify which telemetry types (metrics, logs, traces, events) you would collect for a real-time ML inference service serving 10k requests per second. Provide concrete metric names, suggested structured log fields, typical trace spans, and explain why each telemetry type is useful for debugging, SLO measurement, and capacity planning.
Sample Answer
For a real-time ML inference service at 10k RPS I would collect all four telemetry types—metrics, structured logs, traces, and events—each serving distinct purposes: metrics for SLOs/alerting/capacity, traces for request-level latency/root-cause, logs for rich debugging/context, and events for deployments/incidents.Metrics (high-cardinality + aggregated):- service.requests_total, service.requests_per_second- service.latency.p50/p90/p99 (ms)- service.error_rate (4xx/5xx), service.model_confidence.low_ratio- gpu_utilization, cpu_utilization, memory_usage_bytes- queue_length, inflight_requests, throughput_per_replica- model.version_requests{version=v1}Why: SLO measurement (p99 latency, error_rate), capacity planning (utilization, queue_length), trending and alerting.Structured logs (JSON fields):- timestamp, request_id, trace_id, user_id, model_version- input_summary_hash, request_size_bytes, response_size_bytes- decision/confidence, prediction_latency_ms, backend_host, shard_id- error_type, error_message, stack_trace (when applicable)Why: Deep debugging of individual failures, correlating with traces/metrics.Traces / typical spans:- client → API Gateway (span: http.request)- auth/validation (span: request.validate)- routing/load-balancer (span: router.dispatch)- model.prepare/preprocess (span: preprocess)- model.inference (span: model.forward) — include model_version, GPU id, batch_id- postprocess/serialize (span: postprocess)- downstream call(s) if any (span: db.lookup, span: feature_store.fetch)Why: Pinpoint latency contributors, detect tail latency sources, per-request causality for complex paths.Events (audit & lifecycle):- deployment.started/completed {version, user, timestamp}- config.change, model.rollout_begin/end, canary/rollback- incident.opened/closed, capacity.scale_up/downWhy: Correlate SLO regressions to deployments or config changes for RCA.How to use together:- SLOs: compute p99 latency and error_rate from metrics; generate alerts when thresholds breach.- Debugging: when a p99 spike occurs, find affected trace_ids from metrics → sample traces to identify slow spans → inspect structured logs for input/context and stack traces.- Capacity planning: use long-term metrics (utilization, queue_length, throughput_per_replica) and events (deployment history) to size instances, set autoscale rules, and plan model batching or sharding.Practical tips:- Tag metrics/logs/traces with model_version, region, instance_type to slice by release and infra.- Sample traces intelligently (always sample errors + a small fraction of success).- Emit cardinality-controlled attributes (avoid raw inputs) and use hashes for input fingerprints.
HardTechnical
28 practiced
Create a migration plan to move from a legacy in-house observability stack to a modern open-source stack built on OpenTelemetry, Prometheus, Thanos and Loki. Include dual-writing strategies, data mapping and name translation, exporters, backward compatibility, testing, staged cutover steps, and rollback considerations to minimize production disruption.
Sample Answer
Requirements & constraints:- Preserve existing alerting, SLAs, and historical metrics/logs access.- Zero/near-zero production disruption; allow rollback.- Support ML-specific telemetry: feature drift, inference latency, model inputs, custom metrics.- Compliance and retention policies for logs/metrics.High-level plan:1. Design & mapping- Inventory all legacy signals: metrics (names, units, tags), logs (structures), traces.- Create canonical schema mapping to OpenTelemetry/Prometheus/Loki: - Metrics: map legacy metric -> Prometheus metric name, type (gauge/counter/histogram), label translation rules (rename keys, normalize values). - Logs: map legacy fields -> structured labels in Loki and JSON body. - Traces: identify span conventions and map to OTLP semantic conventions.- Document name-translation table and a reversible mapping function to preserve backward lookups.2. Dual-writing & exporters- Implement dual-write at ingestion layer: - Instrumentation: update application/serving infra to emit OpenTelemetry (OTel) SDK spans/metrics/logs while continuing to send legacy payloads to in-house collector. - Sidecar/agent: deploy OTel Collector as sidecar/daemonset configured with exporters to Prometheus remote_write (or Prometheus scrape via instrumentation), Thanos (object store via Prometheus + Thanos sidecar), and Loki for logs. - Use exporters/transform processors to apply name translations and enrich ML context (model_id, version, dataset_hash).- Ensure idempotence and non-blocking writes (buffering, backpressure control).3. Backward compatibility- Keep legacy pipeline running; replicate new data back to legacy query APIs where feasible (adapter service that answers legacy queries by translating to Thanos/Loki queries).- Maintain historical access by backfilling important metrics/logs from legacy store into Thanos/Loki (export/transform and bulk import), with tags indicating provenance.4. Testing & validation- Unit tests for translation logic.- Integration tests: synthetic workloads reproduce real ML traffic; assert parity of key aggregates (counts, percentiles) within tolerance.- Canary metrics: run comparison jobs to compute difference between legacy and new pipelines for selected metrics, expose dashboards showing drift.- End-to-end: run end-to-end alerts on both systems in “shadow” mode, compare firing rates.5. Staged cutover- Stage 0: Prep — deploy OTel Collectors, exporters, mapping service, adapters; backfill small historical window.- Stage 1: Dual-write enabled for non-critical services and batch jobs; validate parity for 48–72 hours.- Stage 2: Expand dual-write to canary model-serving endpoints and realtime inference paths; monitor error budgets and latency impact.- Stage 3: Switch alert consumers (on-call dashboards) to read from new stack but keep legacy as fallback.- Stage 4: Gradual traffic shift: route small percentage of read queries to new stack, increase as confidence rises.- Stage 5: Decommission legacy ingest after full parity and stakeholders sign-off; keep read-only access for archival.6. Rollback considerations- Dual-writing enables instant rollback: stop consumers pointing to new stack; continue legacy-only reads.- Keep data retention in both systems until decommissioned.- Automate rollback scripts: update routing, revoke permissions, re-enable legacy alerting endpoints.Operational & ML-specific notes:- Ensure model observability labels (model_version, feature_set) are standardized and injected at serving layer.- Use Prometheus histograms for latency-sensitive metrics to compute SLOs; Thanos for global querying and long-term retention.- Correlate logs (Loki) and traces (OTel/Tempo if added) for root-cause of model drift or inference errors.Monitoring & runbook:- Create parity dashboards, automated anomaly detectors comparing pipelines.- Define cutover runbook with success criteria, rollback triggers, and post-mortem template.This approach minimizes disruption via dual-writing, transparent mapping, thorough testing, staged cutover, and clear rollback procedures tailored to ML production needs.
HardTechnical
52 practiced
Given an existing Prometheus + Thanos stack where dashboards query three months of data and panels render slowly, propose concrete strategies to improve query and dashboard performance. Consider result caching, downsampling, query rewrite, precomputed aggregates, and Grafana best practices and justify trade-offs.
Sample Answer
Situation: Dashboards querying three months of raw Prometheus samples via Thanos store result in heavy, slow queries and long render times. Goal: reduce latency while preserving actionable fidelity for ML monitoring.Concrete strategies:1) Downsampling (Thanos side)- Use Thanos compact with downsample.enabled to create 5m/1h downsampled blocks.- Query older ranges against lower-resolution data by time: in Grafana use PromQL that selects max_resolution series or route via Thanos Querier selectors.Trade-off: reduced temporal fidelity for older data but huge reduction in bytes scanned and CPU.2) Precomputed aggregates / recording rules- Create Prometheus recording rules that compute rollups (e.g., job:latency:avg_5m = avg_over_time(request_latency[5m])) and high-level counters (sums, percentiles via histogram_quantile on pre-aggregated histograms).- Use these in dashboards instead of raw expressions.Trade-off: more write-time CPU and storage, but queries become simple instant vectors.Example:recording rule: - expr: avg_over_time(http_request_duration_seconds_sum[5m]) / avg_over_time(http_request_duration_seconds_count[5m]) - name: job:http_request_duration_seconds:avg5m3) Result caching- Enable Thanos Query frontend cache (memcached or built-in caching) to cache query results and split queries into fronted + store shards.- Use Grafana’s query caching (dashboard-level cache or Transform caching plugin) for very-frequently viewed panels.Trade-off: Stale data risk; tune TTL per panel (e.g., 1–5m) to balance freshness.4) Query rewrite / optimization- Replace expensive range-vector joins and rate() over long windows with recording rules.- Limit label-cardinality in queries (use label_join sparingly), and avoid topk on very large cardinality without pre-aggregation.- Use subqueries sparingly; prefer aggregated recordings.5) Grafana best practices- Default dashboard time range to shorter windows (1d) with an explicit “3M” option.- Use templated variables scoped to reduce series (e.g., top 10 services).- Panel options: lower max data points, enable “Min step” to avoid tiny step sizes, and set sensible auto-refresh intervals.- Use dashboard links to deep-dive pages that query full 3M with different UX.Operational considerations & trade-offs:- Storage vs query cost: downsampling + retention rules lower storage and query cost but lose fine-grain historic detail—keep raw recent, downsample older.- Complexity: recording rules and Thanos compaction add operational overhead (CI for rule correctness).- Freshness vs latency: caching reduces latency at the cost of staleness; use short TTLs for ML alerting signals, longer for reporting.Recommendation: Implement recording rules for all heavy aggregations, enable Thanos downsampling and compact older blocks, add Thanos Query Frontend cache with short TTLs, and update Grafana panels (defaults, variables, step) to reduce scanned data. Measure: track query latency, CPU on Thanos Querier/Store, and dashboard render times before/after; iterate on TTLs and retention/downsample windows.
EasyTechnical
34 practiced
Explain metric cardinality and why it can blow up cost and query performance in monitoring systems. Provide ML-specific examples that cause high cardinality (per-user labels, full feature hashes, request ids) and propose at least three concrete strategies to control cardinality while preserving useful observability signals.
Sample Answer
Metric cardinality = number of distinct time series produced by a metric, determined by unique combinations of label keys/values. In systems like Prometheus, each unique label-set spawns a series; high cardinality multiplies series count, increasing storage, ingestion CPU, index size, and query fan‑out — slowing queries and driving cost.ML-specific causes:- per-user labels (user_id) → one series per active user- full feature hashes as labels (high-cardinality features) → explosion with feature combinations- request_ids / trace ids / session ids included as labels → effectively unbounded seriesConcrete strategies to control cardinality while preserving observability:1. Drop or relabel at scrape/ingest: use relabel_configs or an ingest pipeline to remove problematic keys (request_id, session_id) or replace values with coarser buckets (hash_prefix(8) or "user_segment=A/B/C").2. Aggregate before emitting: compute and emit aggregated metrics (percentiles, counts by user_segment/model_version) rather than per-entity counters. Emit histograms or summaries instead of many per-user gauges.3. Sampling and rate-limiting: emit full-detail metrics only for a sampled subset of requests (e.g., 1%) or during known incidents; otherwise emit aggregated rollups.4. Cardinality caps + labels whitelist: enforce max label cardinality per metric and maintain a whitelist of allowed label keys (model_version, region, outcome) to prevent accidental spikes.5. Use logs/traces for high-cardinality IDs: keep request_id/session_id in traces/logs (where they belong) and link them to metric exemplars when needed, avoiding them as labels.These approaches reduce series count, keep queries fast and costs predictable, while preserving actionable signals (model-level, cohort-level, and sampled-request diagnostics).
HardSystem Design
32 practiced
Design an immutable, auditable provenance trail linking deployed model artifacts to the telemetry recorded at inference time, enabling reproducible investigations. Describe model fingerprinting, metadata to store (dataset snapshot, commit hash, config), how to attach metadata to traces and logs, and storage and retention policies for provenance data.
Sample Answer
Requirements:- Immutable, tamper-evident provenance linking deployed model artifact → inference telemetry- Must support reproducible investigations (re-run with same artifact + input)- Low overhead at inference, scalable, searchable, retention/compliance controlsHigh-level architecture:- Model Registry + Artifact Storage (e.g., S3 + object immutability + registry DB)- Fingerprinting service (generates artifact fingerprint, stores metadata)- Inference sidecar / middleware (attaches provenance to traces/logs)- Audit Ledger (append-only store: write-once WORM store or blockchain-style ledger like Kafka+immutable storage or ledger DB)- Search & Reconstruction API and secure UIModel fingerprinting:- Compute multi-part fingerprint: - Binary artifact hash: SHA-256 of model file (weights + serialized graph) - Build fingerprint: container image digest (OCI sha256), dependency hashes (pip/conda lock), protobuf/schema versions - Source fingerprint: git commit hash, repo subpath, patch ID for uncommitted changes - Training fingerprint: training run id, random seeds, training pipeline DAG hash- Store composite fingerprint (JSON) and make it the canonical model_id (immutable)Metadata to store:- Dataset snapshot: dataset_id, manifest (file paths + checksums), sampling seed, versioned feature store snapshot IDs- Commit hash: git SHA, repo URL, branch, patch/PR id- Config: hyperparameters, preprocessing pipeline spec, model signature (input/output schema), framework & versions- Training environment: Docker image digest, CUDA/driver versions- Evaluation metrics: validation/test metrics, thresholds used for promotion- Deployment context: serving config, hardware class, endpoint id, canary rollout metadata- Legal/compliance tags: PII flags, retention policy pointerAttaching metadata to traces and logs:- At deploy time, register model composite-fingerprint → audit ledger entry; deployment returns a short model_version_token.- Inference middleware attaches model_version_token and minimal fingerprint fields (sha256, model_id) to every trace/span as attributes and to structured logs (JSON) alongside request id, timestamp, input checksum (e.g., SHA-256 of raw input or features) and sampled feature snapshot pointer.- For high-throughput systems: emit lightweight references (model_token + input_checksum + timestamp) to a durable event stream (Kafka) rather than full payload; enrich asynchronously during audit reconstruction by joining ledger, feature store snapshots, and trace logs.- Ensure trace fields are indexed (Elasticsearch/OpenSearch) for quick lookups.Storage & retention policies:- Audit Ledger: append-only, stored in WORM-capable object store or ledger DB (e.g., AWS QLDB) with cryptographic signing; retention: immutable for compliance window (e.g., 7 years) then archive to cold storage with retention metadata.- Telemetry logs/traces: short-term hot storage (30–90 days) for operational debugging; export references to long-term audit store when trace contains model_token for reproducibility—store minimal pointers and checksums to reconstruct inputs/features on demand.- Dataset snapshots & feature store: store full snapshots for records referenced by audits; apply tiered retention: full snapshots retained for investigation window (e.g., 1–2 years), hashed manifests kept longer for provenance.- Access controls: RBAC, audit logging of access; cryptographic verification of ledger entries on read.Reproducibility workflow:- Given a trace id: fetch model_token → ledger entry → model artifact + environment + dataset snapshot + preprocessing code; fetch input checksum → retrieve stored input/feature snapshot or re-materialize using feature store and seed; re-run in a sealed, versioned runtime (container digest) to reproduce inference and compare outputs.Trade-offs:- Full-input storage ensures perfect reproducibility but is expensive; use checksums + feature re-materialization as compromise.- Ledger immutability via WORM or blockchain-like systems increases complexity but ensures non-repudiation.- Sampling telemetry reduces storage but may miss some investigations — use adaptive sampling with guaranteed retention for flagged events.This design balances immutability, low inference overhead, and practical storage/retention for auditable, reproducible investigations.
Unlock Full Question Bank
Get access to hundreds of Observability and Monitoring Architecture interview questions and detailed answers.
