Covers hands on familiarity with modern monitoring and observability platforms and the practices for instrumenting and operating production systems. Candidates should be able to describe one or more tools such as Prometheus, Grafana, Datadog, CloudWatch, and explain how to write queries, design dashboards, and configure alerts. Include understanding of metrics collection, time series databases, log aggregation, distributed tracing, and common query languages used by these platforms. Also cover integrating monitoring with incident management systems such as PagerDuty and Opsgenie, defining service level indicators and objectives, setting alerting thresholds to reduce noise, and using dashboards and alerts to troubleshoot performance and availability issues.
EasyTechnical
79 practiced
Define SLI, SLO, and SLA in practical terms a business stakeholder can understand. Then propose a concrete SLI and a corresponding monthly SLO for a payment-authorize API, specifying measurement method, window, and acceptable error budget.
Sample Answer
SLI / SLO / SLA — plain terms- SLI (Service Level Indicator): a measurable signal — e.g., “percent of payment authorization requests that succeed.” It’s the metric we watch to know if the service is healthy.- SLO (Service Level Objective): the target for that metric over a time window — e.g., “99.9% successful authorizations this month.” It’s an internal reliability goal that defines acceptable performance and an error budget for change/risk decisions.- SLA (Service Level Agreement): a contractual commitment to customers, often with penalties if missed — e.g., “we guarantee 99.5% success or we provide credits.” SLAs are legal/financial; SLOs are operational.Concrete proposal for a payment-authorize API- SLI: Authorization success rate. - Definition: (Number of authorization responses with status = APPROVED or AUTHORIZED) / (Total authorization attempts that reached the authorizer, excluding obvious client errors like malformed requests). - Measurement method: Server-side logging at the API gateway; count request/response pairs; filter out client error codes (400–499) and retries counted once using request-id deduplication. - Granularity & window: 1-minute aggregation, stored as time series; monthly roll-up for SLO calculation.- Monthly SLO: - Objective: 99.95% authorization success rate over a 30-day calendar month. - Error budget: 0.05% failure budget ≈ 2,160 failed requests per 4.32M attempts (example scale: 144k attempts/day). Error budget can be spent on planned changes, canary rollouts, or tolerated transient failures. - Measurement details: Calculate rolling 30-day success percentage from per-minute aggregates; alerts if rolling success drops below 99.97% (early warning) and incident if below 99.95%. - Exclusions: Downtime due to agreed maintenance windows and major upstream provider outages (with documented RCA) to avoid unfair SLA breaches.Why this is useful- Business-friendly metric (success rate) ties directly to revenue and user experience.- 99.95% is a high-availability target appropriate for payments while leaving a small error budget for deployments and mitigations.- Clear measurement rules and exclusions prevent disputes when reporting performance to customers.
MediumSystem Design
144 practiced
Design an observability architecture for a microservices ecommerce platform on AWS: 200 services, 10k RPS, multi-AZ, 6-month metrics retention, 3-month logs retention, cross-service distributed tracing, and low-latency Grafana dashboards. Describe components (ingestion, TSDB, log store, tracing, dashboarding), data flow, HA, scaling strategy, and a high-level operational/cost plan.
Sample Answer
Requirements clarified:- 200 microservices, ~10k RPS, multi-AZ, 6-month metrics retention, 3-month logs, distributed tracing, low-latency Grafana.High-level architecture (components)- Metrics ingestion: Prometheus sidecars per service (or Prometheus exporters + Prometheus Agent in each K8s node) with remote_write to a horizontally scalable long-term store (Cortex or Thanos Receive).- TSDB / long-term metrics: Cortex (for multi-tenant, HA, write-scaling) or Thanos (if leveraging Prometheus + object storage). Backing store: S3 for object blocks + DynamoDB (Thanos) or Cassandra/Bigtable alternatives for index if needed.- Logs ingestion: Vector/Fluentd -> Kafka (buffering) -> Elasticsearch/OpenSearch for indexed log queries + S3 (cold storage). Or Loki for label-based, cost-efficient logs with S3 chunks.- Tracing: Tempo (Grafana Tempo) or Jaeger using OTLP; traces stored in object storage (S3) and indexed minimally for trace-id lookup. Ingest via collector (OTel Collector) with Kafka buffering for smoothing spikes.- Dashboarding: Grafana querying Cortex/Thanos (metrics), OpenSearch/Loki (logs), Tempo (traces). Use Grafana Agent in front for dashboards caching and Prometheus-style queries.- Instrumentation: OpenTelemetry for metrics/traces; structured JSON logs with consistent labels (trace_id, span_id, service, env).Data flow summaryServices -> exporters/OTel Agent -> (1) Prometheus agent remote_write -> Cortex/Thanos -> S3; (2) OTel Collector -> Tempo & Kafka -> S3; (3) Fluentd/Vector -> Kafka -> OpenSearch + S3. Grafana queries TSDB, logs, traces; dashboards use cached query layer.HA and scaling strategy- Multi-AZ deployment for all components; stateless components (receivers, collectors, agents) autoscaled by K8s HPA; stateful: Cortex/Thanos store-gateway + compactor replicated; use S3 (multi-AZ durable) for long-term storage.- Ingest scale: shard Cortex/Thanos by tenant or use consistent hashing; use write-ingestion ring with multiple receivers behind ALB/NLB; Kafka sized for retention buffer to absorb spikes.- Read scaling: query frontends and caching (Cortex query-frontend, Thanos querier) with autoscaling; Grafana caching (proxy layer) and TTLs to satisfy low-latency dashboards.Operational and cost plan (high-level)- Tier metrics: high-cardinality ephemeral metrics (shorter retention in hot TSDB, e.g., 7–30d) vs aggregated metrics (6-month retention in long-term store). This reduces storage cost and query latency.- Logs: index only necessary fields; keep full logs in S3 for 3 months and tier to Glacier after that if needed. Consider Loki for lower-cost logs if full-text search not required.- Traces: sample at edge (dynamic sampling), keep full traces for errors/alerts, aggregated spans for 3 months. Use adaptive sampling to control cost.- Monitoring & alerts: self-observability stack; alerting on ingestion lag, storage growth, query latency, error rates.- Backup & DR: S3 lifecycle and cross-region replication for critical indices; periodic snapshots of OpenSearch.- Cost estimate levers: S3 storage (cheap), index storage in OpenSearch (expensive) — reduce indexed fields and retention; adjust trace sampling; downsample metrics before long term retention.- Runbook & SLOs: define metrics ingestion SLO, query latency SLO (<500ms for dashboard panels), alerting thresholds and on-call rotation.Trade-offs- Cortex gives stricter multi-tenant and ingestion scale; Thanos is simpler if you prefer Prometheus UX. Loki vs OpenSearch depends on log-search needs vs cost.- Use Kafka as buffer to decouple spikes at ingestion — adds operational overhead but improves reliability.This design meets durability, scale, multi-AZ HA, cross-service tracing, and low-latency dashboards while controlling cost via tiering, sampling, and indexing choices.
HardTechnical
103 practiced
A client wants to detect novel performance regressions automatically (unknown patterns). Propose an approach using open-source tools: discuss feature selection from metrics, statistical vs ML models (e.g., ARIMA, isolation forest, LSTM), training/labeling strategy, alert integration, and how you'd limit false positives operationally.
Sample Answer
High-level approach- Build an ingestion + feature store from existing observability metrics (Prometheus, StatsD, OpenTelemetry) into a time-series store (Prometheus remote write → Thanos/InfluxDB) and a streaming layer (Kafka) for real-time processing. Use Grafana/OpenSearch for visualization and historical queries.Feature selection from metrics- Raw features: rate, count, latency pXX (p50/p95/p99), error rates, saturation (CPU/RAM), QPS, queue lengths.- Derived features: rolling-window deltas, seasonally-adjusted residuals, normalized ratios (errors/QPS), z-scores, trend slopes, autocorrelations, and frequency-domain features (FFT) for periodic signals.- Use automatic selection: mutual information / correlation filtering + anomaly-specific heuristics (e.g., increase in tail latency relative to median) to reduce dimensionality. Keep both aggregated (service-level) and granular (instance/container) features.Model choices: statistical vs ML- Statistical baseline: ETS/ARIMA/Prophet or seasonal decomposition (STL) for univariate baselines — good for explainability and low-data regimes; yields residuals for anomaly scoring.- Unsupervised ML: Isolation Forest / One-Class SVM / LOF for multivariate outlier detection — fast, requires little labeling.- Time-series ML: LSTM/Seq2Seq or Temporal Convolutional Networks for reconstruction/prediction anomalies (detect when real != predicted). Use autoencoders (LSTM/TCN) for novelty detection.- Hybrid: Use statistical models to remove seasonality and trend, then feed residuals into multivariate detectors (isolation forest/autoencoder). Ensembles reduce model-specific blind spots.Training / labeling strategy- Start unsupervised/semi-supervised: train on long windows of "healthy" historical data using rolling windows. Create synthetic anomalies to stress models (spikes, drifts, sustained regressions) for validation.- When incident labels exist, store them in a label store (ELK/MLflow) and progressively create supervised classifiers or tune thresholds. Use active learning: surface borderline anomalies to SREs for labeling and feed back.- Adopt continuous training (daily/weekly) with concept-drift detection (monitor feature distribution drift via Population Stability Index) and model versioning via MLflow or Feast.Alert integration & operationalization- Score → normalize → enrich (service, owner, runbook link) → send to Alertmanager / ElastAlert / OpenSearch alerting → PagerDuty/Slack webhook with context: recent graphs, root-cause candidate features, model confidence.- Implement alert grouping (by service, host, or anomaly fingerprint), mute windows, and dynamic priority based on impact estimate (e.g., % user requests affected).- Provide a “triage UI” in Grafana with suggested RCA features and a one-click “false positive” feedback that funnels into labeling.Limiting false positives operationally- Ensemble scoring: combine multiple detectors (statistical residual + isolation forest + autoencoder) and require consensus or weighted score to trigger.- Use adaptive thresholds: percentile-based thresholds over baseline windows or maintain precision@k targets; tune per-metric thresholds rather than global.- Rate-limit and deduplicate alerts, require persistence (anomaly must persist for N windows or exceed cumulative severity).- Silence low-value alerts with business-context filters (e.g., deploy windows, low-traffic services).- Human-in-the-loop: initial detection goes to a low-noise “review” channel for SRE validation before auto-escalation; use feedback to reduce future false positives.- Monitor detector health (alert volume, precision/recall on labeled incidents) and implement rollback if false-positive rate spikes.Operational considerations & trade-offs- Start simple: univariate seasonal baselines + isolation forest ensemble for coverage, then iterate to LSTM/autoencoder where data and ROI justify complexity.- Cost vs latency: heavy models require GPUs/TF serving; prefer streaming, lightweight models for real-time critical services.- Compliance & explainability: keep statistical baselines and feature importances available for audits and on-call trust.Metrics for success- Reduce time-to-detect regressions, track precision/recall, mean time to acknowledge (MTTA), and false positive rate per week. Use these to iterate model and threshold tuning.This architecture provides a pragmatic, incremental path from simple statistical detection to production-grade ML-driven novelty detection with operational controls to minimize false positives and integrate into existing SRE workflows.
MediumTechnical
75 practiced
Compare hosted observability (Datadog/New Relic) with an open-source stack (Prometheus + Grafana + Loki) for a mid-size SaaS startup. Discuss operational overhead, feature parity (APM, logs, metrics), TCO, vendor lock-in risks, scaling concerns, and which choice you'd recommend for a company with a small SRE team.
Sample Answer
Situation overview: For a mid-size SaaS startup with a small SRE team, choosing between hosted observability (Datadog/New Relic) and an OSS stack (Prometheus + Grafana + Loki) requires balancing speed, features, cost, and operational load.Operational overhead- Hosted: Minimal ops — vendor manages ingestion, storage, upgrades, scaling, alerting UI. Team focuses on instrumentation and runbooks.- OSS: Significant ops — deploy, scale, upgrade, handle retention/compaction, HA, backups, and storage (especially logs). Requires CI/CD, k8s/statefulset ops, and observability for the observability stack.Feature parity (APM, logs, metrics)- Metrics: Prometheus excels for metrics with powerful querying; Grafana provides rich dashboards. Hosted matches and often adds integrations, long-term storage, and UIs.- Logs: Loki is cost-efficient but immature vs. full-featured log platforms; hosted providers offer richer indexing, faster search, and log parsing.- APM: Datadog/New Relic provide turnkey distributed tracing, service maps, automatic instrumentation, flame graphs, and correlation between traces/metrics/logs. OSS has Jaeger/Zipkin + OpenTelemetry but needs integration effort and lacks some polished UIs.TCO- Hosted: Higher recurring subscription cost that scales with ingestion/hosts. Predictable but can grow quickly with logs/traces.- OSS: Lower licensing cost but higher people + infra cost. For small team, hidden ops cost can exceed hosted fees as scale and SLAs increase.Vendor lock-in risks- Hosted: Proprietary formats and queries create migration cost; mitigated by exporting metrics/traces via OpenTelemetry and setting retention exports.- OSS: Lower vendor lock-in; standards-based tooling eases migration.Scaling concerns- Hosted: Scales transparently; watch cost and ingestion spikes.- OSS: Scaling logs is hardest — storage, indexing, and query performance require careful architecture (object storage, sharding). Metrics scale with federation/remote_write patterns.Recommendation- For a mid-size SaaS with a small SRE team and need to move fast: choose hosted observability (Datadog/New Relic) for turnkey APM, integrated UX, and low operational overhead. Start with retention/export policies and OpenTelemetry to reduce lock-in risk.- Consider a hybrid: use hosted for traces/logs/APM initially, and run Prometheus+Grafana for low-latency, high-cardinality metrics you control. Re-evaluate after 12–18 months as scale and team grow.
EasyTechnical
74 practiced
Explain metric cardinality, why it is harmful for Prometheus and other TSDBs, and provide three concrete strategies you would recommend to a client to control cardinality (with examples such as label whitelisting, relabeling, or bucketing). For each strategy state the trade-off.
Sample Answer
Metric cardinality is the total number of unique time series produced by a metric name combined with its label set (each unique combination = one series). High cardinality (e.g., user_id, request_id or timestamps as labels) multiplies series count, driving memory, disk usage, slow queries, longer startup/rebuild times, and risk of OOMs in Prometheus and other TSDBs.Why it's harmful:- Memory pressure: TSDBs keep index/state in memory per series.- Disk growth and compaction overhead.- Query latency: scanning many series is expensive.- Operational risk: scraped targets or bursts can spike series count.Three concrete strategies to control cardinality (with trade-offs):1) Label whitelisting (scrape-time or ingestion policy)- What: Only accept a predefined set of labels per metric (drop unknown labels).- Example: Allow {instance, job, env} but drop user_id and request_id.- Trade-off: Simple and safe; may lose useful context if overly strict. Requires governance to update whitelist when new useful labels are needed.2) Relabeling / dropping at scrape-time- What: Use Prometheus relabel_configs to modify or drop labels before storing.- Example: relabel to remove query parameters: replace 'path'=/api/item/123 -> /api/item/:id, or drop session_id.- Trade-off: Flexible and powerful; needs careful rules to avoid accidental data loss. Relabel logic complexity increases maintenance cost.3) Bucketing / aggregation (reduce cardinality by grouping values)- What: Convert high-cardinality label values into buckets or rollups before ingestion.- Example: Replace latency_ms label with latency_bucket=[0-10,10-50,50-200,200+]; or hash and mod to limited buckets for rare identifiers.- Trade-off: Preserves trends and enables aggregation but loses exact values and per-entity granularity; impacts some drill-down analyses.Recommendation: combine strategies—whitelist baseline labels, relabel to normalize/drop noisy labels, and bucket high-cardinality dimensions. Monitor series_count, memory, and label value histograms; automate alerts when new label explosions occur.
Unlock Full Question Bank
Get access to hundreds of Monitoring Tools and Observability interview questions and detailed answers.
