Metrics, Logs, and Traces Strategy Questions
Strategy for instrumentation and observability, covering what metrics to collect, how to structure and collect logs, distributed tracing, data retention and aggregation, alerting, and how telemetry informs reliability engineering, incident response, and performance tuning.
MediumTechnical
85 practiced
Explain how histograms are aggregated across services in Prometheus; why must bucket boundaries be identical across services to merge histograms accurately; and describe a strategy for choosing bucket boundaries for multi-service latency aggregation.
Sample Answer
Prometheus histograms are exposed as a set of cumulative counters: for each bucket boundary b you get a metric count of observations <= b (plus sum and total count). When Prometheus (or remote storage) aggregates histograms across services/instances it simply sums those per-boundary counters. That merging only produces correct aggregated distributions if every source used the exact same bucket boundaries.Why identical boundaries are required- A bucket counter represents "≤ b". If service A uses boundary 100ms and service B uses 120ms, their counts are not measuring the same cutoff; summing them mixes different ranges and produces incorrect percentiles and distribution estimates.- Mismatched boundaries make it impossible to reconstruct correct combined cumulative distribution or approximate quantiles without additional per-observation data.Strategy for choosing bucket boundaries (multi-service latency)1. Standardize centrally — ship a shared bucket config (client library or service template) so every service uses the same boundaries.2. Cover SLOs with high resolution — place denser buckets around SLO/warning thresholds (e.g., 50–200ms) so percentile estimates near the SLO are accurate.3. Use exponential/log-scale for wide ranges — e.g., start at 1ms and multiply by ~2 until you reach a high tail (1ms,2ms,4ms,...,~10s). This balances resolution and bucket count.4. Hybrid approach — combine linear buckets near the SLO and exponential elsewhere.5. Mind cost trade-offs — more buckets increase metric cardinality and memory on clients and Prometheus; keep bucket count reasonable (e.g., 15–40).6. Validate with real traces — use A/B experiments or sampled traces to tune boundaries to actual latency distributions.7. Alternatives — if precise per-service quantiles are needed without shared buckets, consider distributed tracing or using summaries where appropriate (but summaries don’t merge across instances).Example: for typical web request latencies with an SLO at 200ms, you might use linear buckets every 10ms from 0–200ms, then exponential 200ms, 400ms, 800ms, 1.6s, 3.2s, 6.4s. Deploy this as the organization standard so aggregated histograms yield correct percentiles.
MediumTechnical
69 practiced
A service is generating high metric cardinality because teams started adding 'user_id' and 'experiment_id' as labels. As the SRE on-call, propose immediate mitigation steps, medium-term architecture changes, and long-term guardrails to prevent reintroduction of high-cardinality patterns.
Sample Answer
Immediate mitigation (what I’d do on-call, within minutes–hours)- Silence noisy alerts and open an incident ticket describing cardinality spike and impacted dashboards.- Apply metric relabeling/drop at the ingestion edge (Prometheus relabel_configs or pushgateway/proxy) to immediately drop user_id and experiment_id labels for the offending metric: - Example: relabel_configs: - source_labels: [__name__] regex: my_metric action: labeldrop: user_id,experiment_id- Throttle or temporarily disable client-side emission of the high-cardinality metric (feature-flag or deploy a hotfix) if relabeling isn’t possible.- Add a short-term alert: cardinality-growth-rate > threshold to catch regressions.Medium-term architecture changes (days–weeks)- Move high-cardinality dimensions to logs/events or a metric-backed derived store (e.g., export counts per experiment to a sampled/event pipeline like Kafka + ClickHouse) instead of metrics.- Implement an aggregation layer: have services emit only aggregated buckets (e.g., per-service, per-region) and push raw identifiers to an analytics pipeline.- Enforce ingestion-side validation: metric schema registry that rejects metrics with blacklisted labels or over-size label sets.- Add metric-cardinality dashboards for teams and automations to pinpoint sources.Long-term guardrails (weeks–quarters)- Define metric naming and labeling standards in an observable engineering policy; require PRs to add metrics and run linting.- Gate new label additions behind code review + metric-ownership approval; maintain a registry (with cardinality expectations).- Enforce quotas/limits in the metrics backend (per-metric and per-tenant series limits) and automated alerts when close to quotas.- Provide libraries and helper functions (SDKs) that make safe patterns easy: explicit allowed label list, label cardinality checks, sampling helpers.- Educate teams with runbooks, examples of acceptable patterns (histograms/timers, aggregated counters), and postmortems for incidents.Why this approach- Immediate drop prevents backend overload; medium-term keeps analytics needs while removing metric pressure; long-term prevents recurrence via policy, automation, and developer ergonomics.
MediumSystem Design
67 practiced
You ingest 100k series/sec into Prometheus and must retain 365 days of historical data for business analytics at lower resolution. Design a strategy using recording rules, remote_write, and downsampling to reduce load and cost while keeping recent high-resolution data for alerts. Mention trade-offs.
Sample Answer
Requirements & constraints- Ingest 100k series/sec, keep high-resolution recent data for alerts, keep 365d history at lower resolution for analytics, minimize cost and query latency for alerts.High-level strategy1. Short-term hot Prometheus: retain raw 1s–15s scrape data for a short window (e.g., 7–15 days) to support alerts and ad-hoc troubleshooting.2. Recording rules (nearline aggregation): in the hot Prometheus cluster define recording rules that compute rollups at a coarser resolution (e.g., 1m and 5m aggregates: avg(), sum(), max(), p95 approximations) and store those series alongside raw series.3. remote_write to long-term store: stream both raw (if retaining short window only) and the precomputed rollups to a scalable remote storage backend (Thanos/Cortex/Mimir, ClickHouse, or an object-store backed timeseries DB).4. Downsampling in long-term store: use the backend’s compaction/downsampling (Thanos Compact/Downsample or Mimir retention policies) to keep: - High-res rollups (1m) for recent weeks (e.g., 30 days) - Coarser rollups (5m/1h) for 365d5. Query routing: keep query layer that routes alerting queries to hot Prometheus (high res) and analytics queries to long-term store using downsampled data.Concrete choices & params (example)- Hot Prometheus retention: 14 days, full resolution.- Recording rules: produce metrics like metric_name:avg_1m, metric_name:sum_5m, metric_name:max_1h.- remote_write: send only rollup series and optionally a sampled subset of raw series; use batching, compression, and TLS.- Long-term store: Thanos/Cortex with object storage; configure compaction to produce 1m/5m/1h downsampled blocks and retention policies.- Alerting: Alertmanager uses hot Prometheus targets; for historical alert evaluation, use remote store if necessary but prefer hot.Data flow- Scrapers → Hot Prometheus (raw TSDB)- Recording rules evaluate every scrape interval and create rollup series- remote_write forwards rollups (and optional raw subset) → long-term store- Long-term store compacts and downsampled blocks for 365d retention- Queries/alerts: alerts → hot Prometheus; analytics → long-term storeScalability & operational points- Shard scraping across multiple Prometheus instances to manage 100k series/sec; use service discovery.- Use relabeling to drop low-value labels and reduce series cardinality before ingestion.- Tune recording-rule evaluation concurrency and CPU to avoid rule evaluation overload.- Monitor remote_write throughput, retry/backoff, and network egress costs.- Use tenant/label-based partitioning in backend to scale horizontally.Trade-offs- Accuracy vs cost: downsampling and rollups lose per-sample fidelity (percentiles become estimates). Choose aggregation functions aligned with analytics needs.- Complexity: adds deployment (rule management, long-term store, compactor) and operational overhead.- Latency of analytics: aggregated/downsampled stores introduce compaction delay (minutes to hours) — acceptable for analytics, not for alerts.- Storage vs compute: precomputing rollups increases write volume but reduces long-term query cost; alternatively rely on backend downsampling (more compute during compaction).- Alert correctness: keep raw short-term data to avoid missing transient spikes that downsampling could hide.Why this is safe for SRE goals- Keeps low-latency, high-fidelity data where it matters (alerts).- Pushes historical/aggregate storage to cheap, scalable backends.- Recording rules and downsampling reduce long-term storage costs and query times while preserving business-necessary trends.
HardSystem Design
74 practiced
Design an alerting pipeline that supports dynamic suppression based on SLO burn rate and leverages ML-based noise reduction to reduce false positives. Describe system components, feedback loops, model inputs/labels, human-in-the-loop corrections, and how to evaluate model performance for safety-sensitive alerts.
Sample Answer
Requirements & constraints:- Functional: real-time alerts, dynamic suppression based on SLO burn rate, ML noise reduction, human-in-loop corrections.- Non‑functional: <30s latency to alert decision, explainability, auditable decisions, safety-first (no automatic suppression of critical safety alerts without human confirmation).High-level architecture:- Telemetry → Feature Extraction & Enrichment → Rule Engine (baseline) → ML Noise-Reduction Model → Dynamic Suppression Policy → Alert Router / Incident Platform → Human Operator UI → Feedback Store → Model Training Pipeline / CIComponents and responsibilities:- Ingest & Enrichment: collects metrics, logs, traces; computes rolling burn rate, error budget remaining, service topology, owner info.- Feature Extractor: per-alert features (metric history, derivatives, seasonality, associated traces, change points), SLO-derived features (current burn rate, burn-rate trend, time-to-breach) and contextual features (deploys, maintenance windows).- Baseline Rule Engine: deterministic thresholds & safety blacklist (never suppressed alerts like safety-critical processes). Ensures minimal safety guarantees.- ML Noise-Reduction Model: real-time scorer that outputs probability that an alert is actionable/true positive. Lightweight explainable model (calibrated tree ensemble or monotonic GBM with SHAP) for latency and interpretability.- Dynamic Suppression Policy: combines SLO rules and model score: if burn rate high (near breach) lower suppression threshold; if burn rate low, allow higher suppression. Policies expressed as priority matrix.- Alert Router: takes final decision: fire, suppress, aggregate, or escalate for human confirmation.- Human UI & Feedback: shows model score, top contributing features, recommended action; allow operators to override and label.- Feedback Store & Training Pipeline: stores operator labels, incidents, resolution times, and replays for offline/online training with active learning.- Monitoring & Safety: drift detection, model A/B canarying, kill-switch (fallback to baseline rules), audit logs.Model inputs & labels:- Inputs: time-series windows, derived statistics (mean, variance, slope), change-point indicators, recent deploys, correlated signals, SLO burn-rate, historical alert frequency per entity, past suppression decisions.- Labels: actionable vs noise (binary) derived from incident postmortems, operator-confirmed alerts, pager escalations, and remediation-runbook triggers. Use delay windows: e.g., label true if within 15m there was a remediation or incident ticket created.Human-in-the-loop & active learning:- Present top-N uncertain alerts to operators for labeling (uncertainty sampling).- Provide explainability (feature attributions) so operators trust corrections.- Use operator overrides to create high-quality labels; weight recent labels higher.- Periodic review meetings for edge cases; incorporate rule updates when model errors are systematic.Safety & evaluation for safety-sensitive alerts:- Maintain whitelist: safety-critical alert types always bypass suppression.- Evaluate with conservative metrics: precision at high recall thresholds (e.g., precision@recall≥0.99), false negative rate bounded (<X per month).- Use confusion matrix over recent window, but prioritize recall for safety classes. Monitor time-to-detect/MTTD, mean time to acknowledge (MTTA) and false positive rate (FPR).- Backtest on historical incidents, run shadow mode & gradual rollout (canary 1%, 10%, 100%) with automated rollback on safety metric regression.- Use adversarial tests (simulated incidents) to validate suppression behavior under burn-rate surges.Key trade-offs:- Aggressiveness of suppression vs risk of missed incidents — controlled via SLO burn-rate and whitelist.- Model complexity vs explainability/latency — prefer interpretable models for operator trust.- Label latency — operator-labeled data is high-quality but slower; combine with heuristics from incident tickets.Operational practices:- Track model drift, feature importance, calibration; retrain weekly or on drift.- Keep full audit trail of suppression decisions and human overrides.- Define SLAs for model performance; tie rollbacks to safety metrics.This design ensures dynamic, SLO-aware suppression while keeping humans and deterministic safeguards in the loop for safety.
MediumSystem Design
67 practiced
You own an HTTP API. Propose a concrete Prometheus metric and label design to capture request count, latency histogram, and error rate per endpoint and method while minimizing cardinality. Explain label choices and how they support SLOs.
Sample Answer
Requirements & constraints:- Capture per-endpoint+method request count, latency distribution, and error rate.- Minimize cardinality (no raw URLs, no user/request identifiers).- Support SLOs: p99/p90 latency and error-rate SLOs per logical endpoint.Metric & label design (concrete):1) Request counter (total)http_requests_total{service="payments",env="prod",handler="/payments/{id}",method="POST",status_class="5xx"}- type: Counter- handler: templated route name (framework route or explicit handler id), not raw path- method: HTTP verb- status_class: "2xx","3xx","4xx","5xx" (reduces many status codes)2) Latency histogramhttp_request_duration_seconds_bucket{service="payments",env="prod",handler="/payments/{id}",method="POST",le="0.05"}http_request_duration_seconds_sum{...}http_request_duration_seconds_count{...}- type: Prometheus histogram- buckets: [0.005,0.01,0.025,0.05,0.1,0.25,0.5,1,2.5,5,10] (captures p50/p90/p99 without insane cardinality)- same labels as counter (no status_class on buckets by default to limit series; include if latency by error-class matters)3) Error counter (optional explicit)http_request_errors_total{service="payments",env="prod",handler="/payments/{id}",method="POST",error_type="timeout"}- type: Counter- error_type: coarse categories only (timeout, upstream, validation); avoid stack traces or unique idsLabel rationale & cardinality:- handler: single small set (bounded by routes) vs raw path (unbounded).- method + status_class: small finite sets; status_class collapses 100+ codes into 4 values reducing series explosion.- service/env: necessary for multi-service, low-cardinality.- Avoid high-cardinality labels: user_id, request_id, hostnames per pod — prefer instance only when needed for debugging; use instance in high-cardinality contexts sparingly.How this supports SLOs:- Error-rate SLO: derive errors / total via rate(http_request_errors_total[5m]) or use rate(http_requests_total{status_class="5xx"}[5m]) to compute error percentage per handler+method over window.- Latency SLOs: use histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m])) grouped by handler+method to get p99 latency per endpoint.- Define recording rules to compute per-handler SLO metrics (error_rate: 5m, latency_p99: 5m) and alert when error_rate > target or latency_p99 > target.- Aggregation: sum by (service,env,handler,method,le) for histograms; sum by (service,env,handler,method) for counters.Operational notes & trade-offs:- If you need exact status-code breakdown for a small set of handlers, add status (e.g., "404") only for those handlers via special instrumentation or label sanitization.- Consider using exemplars (trace IDs) with histograms for linking traces to slow/error samples.- Record rules reduce query cost and centralize SLO logic.This design balances observability for SLOs with controlled cardinality.
Unlock Full Question Bank
Get access to hundreds of Metrics, Logs, and Traces Strategy interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.