Metrics Analysis and Monitoring Fundamentals Questions

Fundamental concepts for metrics, basic monitoring, and interpreting telemetry. Includes types of metrics to track (system, application, business), metric collection and aggregation basics, common analysis frameworks and methods such as RED and USE, metric cardinality and retention tradeoffs, anomaly detection approaches, and how to read dashboards and alerts to triage issues. Emphasis is on the practical skills to analyze signals and correlate metrics with logs and traces.

HardSystem Design

91 practiced

Design an automated incident playbook that uses metric signals to trigger a safe rollback of a Kubernetes deployment when reliability degrades. Specify which metrics and thresholds should be considered (e.g., p99 latency, 5xx rate, pod restarts), how to avoid false positives during rolling deploys, safety checks before rollback (canary checks, cooldowns), testing strategy, integration with on-call workflows, and how rollback actions affect SLO/error-budget calculations.

Sample Answer

Requirements / goals:- Automatically detect rollout-induced reliability regressions and perform a safe, auditable rollback while minimizing false positives and operator disruption.- Integrate with Kubernetes (Deployments/Argo/Flux), monitoring (Prometheus), alerting (PagerDuty/Slack), and CI/CD.

High-level design:- A controller (stateless microservice) subscribes to metrics, deployment events, and CI/CD webhooks. When configured thresholds are breached during a recent rollout, it initiates a rollback workflow (not always immediate destructive revert).

Metrics & thresholds (examples; tune per-service):- p99 latency: > SLO p99 * 1.5 sustained for 5m- p95 latency: > SLO p95 * 1.3 for 5m- 5xx rate: > 0.5% absolute increase or > 3x baseline for 3m- error rate (client/non-retryable): > SLO error budget burn rate (e.g., burn > 2x expected) for 5m- pod restarts: > 1% of pods restarting within 5m or > 3 restarts per pod in 10m- CPU/Memory OOMs: any OOMKilled event during rollout- Request throughput drop: > 30% drop vs 95th percentile baseline

Avoiding false positives during rolling deploys:- Correlate metric anomalies with active rollout windows (deployment timestamp, image tag). Only consider triggers if they occur within rollout window plus short grace (e.g., 10m).- Require at least two independent signals (latency + 5xx or restarts + 5xx) before escalating to rollback.- Require sustained breach windows (exponential backoff) rather than single spikes.- Use canary-aware grouping: if release uses canary (10–20% traffic), only roll back if metric breaches both in canary and in control comparison or if canary sample size indicates significance (statistical test, e.g., 95% CI).

Safety checks before rollback:- Canary checks: verify the canary group shows regression with statistical significance and control group is healthy.- Cooldown: enforce a minimum cooldown after previous rollback/deploy (e.g., 15–30m) to avoid thrashing.- Manual approval gate for high-risk services: auto-notify and wait for one-click approve within 2 minutes; otherwise proceed with automated rollback if configured as automatic.- Dependency check: ensure rollback image exists and was healthy historically (lookup in deployment history / image provenance).- Readiness probe verification: confirm rollback will produce Ready pods (simulate rollout dry-run via kubectl rollout status check against expected replica counts).- Circuit-breaker: limit rollback frequency per hour/day.

Rollback workflow:- Step 1: Open incident in incident system with context (deployment ID, metrics, dashboards, last good revision).- Step 2: If automatic rollback allowed and safety checks pass, initiate kubectl rollout undo or promote last-stable ReplicaSet via controller.- Step 3: Monitor targeted metrics for improvement for a validation period (e.g., 10m).- Step 4: If rollback fails or metrics don't improve, escalate to on-call with runbook steps.

Testing strategy:- Unit & integration tests for controller logic, metric ingestion, and policy evaluation.- End-to-end test environment that runs simulated rollouts with fault injection (latency, 5xx, pod OOMs) and verifies controller triggers and safe rollback behavior.- Chaos testing in staging/limited production (e.g., kube-monkey) to validate rollback and rollback-to-stable image selection.- Synthetic traffic to validate SLO/metrics detection sensitivity.- Canary A/B statistical testing utility included in tests.

Integration with on-call workflows:- PagerDuty alerting with rich context: deployment ID, implicated version, diffs, top error traces, runbook link, one-click rollback or pause buttons.- Slack/Teams channel updates with actions and rollback status.- Automatic incident creation with correlated alerts and runbook links; include “who” (on-call), “what” (rollback initiated or pending), and “how to revert” steps.- Post-rollback blameless postmortem auto-assignment and link to CI/CD commit.

SLO/error-budget effects:- Rollbacks should be recorded as mitigation actions and included in incident timeline.- If rollback prevents further customer impact, count observed errors only; do not retroactively charge error budget for automated rollback events themselves.- Define policy: if rollback is corrective to a bad release, errors during the faulty release count toward error budget; the rollback action or brief downtime caused by rollback (if user-facing) should be accounted transparently—e.g., classify as “mitigation-induced downtime” and include in SLO reporting with tags.- Use burn-rate logic: if error budget burn exceeds threshold and automated rollback occurred, prefer automated rollback to stop further burn; tag incident so SRE/Product can decide whether to tighten deploy controls.

Operational best practices:- Keep last-known-good images and metadata immutable and accessible.- Run rollbacks as Kubernetes-native (replicaSet promote) to preserve declarative state.- Provide dry-run and manual override paths.- Maintain telemetry dashboards and a “deployment health” page for quick triage.

This design balances automation speed with safety checks, statistical rigor to avoid noise-triggered rollbacks, and clear on-call integration and SLO accounting.

MediumTechnical

70 practiced

Your service reports an increase in 5xx rate but logs show few error messages. Describe a plan combining metrics, increased structured logging, distributed tracing, and exemplars to narrow root cause while keeping metric cardinality under control. Which additional metrics or exemplar placements would you add and why?

Sample Answer

Plan (high level): treat this as a targeted triage problem — use metrics to localize (service / endpoint / deployment / client), increase structured logging and tracing selectively to get context, attach exemplars to metrics to link traces to metric spikes, and control cardinality via sampling, whitelists, and bounded tag values.

Steps

1. Rapid localization with metrics- Query 5xx rate by host, pod, deployment, container image, endpoint, client/caller service, and region over the incident window.- Correlate with related signals: request latency p50/p95/p99 histograms, request rate (RPS), time-series of deploys, CPU, memory, connection/retry counts, DB latency, and upstream/downstream error rates.- If one dimension (e.g., a single revision or node pool) accounts for the spike, scope follow-up there.

2. Add low-risk, structured logging- Increase structured logs only for the suspect code paths or for a small percentage of requests (e.g., 1–5% sampling). Emit JSON with fixed keys: timestamp, trace_id, span_id, req_id, route, status, upstream_id, backend_latencies, error_code, error_message.- Use dynamic, on-the-fly log level toggles (feature flag or admin API) so you can turn sampling up for affected instances and down after.

3. Distributed tracing with exemplars- Ensure every request has a propagated trace_id/req_id. Use traces to capture span-level failures (HTTP client errors, DB errors, retries, timeouts).- Use tail-based sampling for traces in the error window (sample all traces with 5xx or >X ms latency).- Add exemplars: attach trace IDs as exemplars to the histogram bucket for 5xx counts and to the high-latency histogram buckets. This creates direct links from metric spikes to representative traces.

4. Cardinality control strategies- Limit tag cardinality: only tag metrics with low-cardinality dimensions (service, environment, route-group — not full URL or user id). For route, map routes to a small set of route IDs.- For logs/traces, allow high-cardinality context (user id, session) only in sampled traces/logs — never as metric tags.- Use sampling policies: fixed-rate for background, dynamic ramp-up for affected shards, and whitelist specific endpoints for full capture.

5. Additional metrics & exemplar placements (what I'd add and why)- Per-backend/upstream 5xx and latency counters — to see whether downstream services are failing.- Per-replica instance 5xx rate and thread/worker pool saturation metrics — to detect localized overload.- Connection/retry/backoff counters (HTTP retries, circuit-breaker tripped) — reveals cascading failures.- DB pool wait times and queue lengths — to detect exhausted DB connections causing 5xx.- Error-class histogram (by error type bucketed) with exemplars on errors that map to internal exceptions or status codes.- Place exemplars on: - 5xx counter metric (so an exemplar trace shows what caused the error) - latency histogram p99 bucket (to see why long requests are occurring) - per-upstream error-rate metric (to link to failing dependency traces)

Trade-offs and rationale- Sampling + exemplars gives “needle-in-haystack” trace linkage without exploding metric tag cardinality.- Tail-based sampling ensures we capture the problematic traces (errors/long tails) while keeping trace volume manageable.- Dynamic logging prevents permanent log bloat and lets you get richer context only where needed.

End condition and next steps- Once traces/logs identify root cause (bad deploy, misconfigured retry, DB saturation, malformed client requests), roll back or patch, validate by watching 5xx and exemplar-linked traces, then reduce sampling/log verbosity.- Post-incident: add focused synthetic checks, SLO adjustments, and instrumentation (low-cardinality metrics or derived alerts) to detect recurrence without high cardinality.

MediumTechnical

92 practiced

Write a Prometheus recording rule YAML snippet that computes a per-service 1-minute request rate and a 5-minute moving average of that rate. Explain where to place the file in Prometheus config, an appropriate evaluation interval, and how recording rules help for high-cardinality dashboards.

Sample Answer

Here's a concise example of two Prometheus recording rules: one that computes a per-service 1-minute request rate, and a second that computes a 5‑minute moving average of that recorded rate.

yaml

groups:
- name: service_request_rates
  rules:
  - record: service:request_rate:1m
    expr: |
      sum by (service) (
        rate(http_requests_total{job="frontend"}[1m])
      )
  - record: service:request_rate_avg:5m
    expr: |
      avg_over_time(service:request_rate:1m[5m])

Where to place the file:- Put this YAML in a file referenced by the "rule_files" section of prometheus.yml (common path: /etc/prometheus/rules/service_request_rates.yml).- Example in prometheus.yml: rule_files: - "/etc/prometheus/rules/*.yml"

Evaluation interval:- Set Prometheus global `evaluation_interval` to something that balances freshness and load; 15s is typical for 1m rate windows (you can use 15s or 30s). Ensure evaluation_interval << the smallest window in your rules so rate() samples are meaningful. Example: evaluation_interval: 15s.

Why recording rules help for high-cardinality dashboards:- Precomputing heavy queries (rate, aggregations) reduces CPU and query latency because dashboards read compact, pre-aggregated metrics instead of recomputing expensive functions on every dashboard load.- Aggregating by service reduces cardinality exposed to downstream queries; keep labels limited (e.g., service only) to avoid exploding series.- Recorded metrics are cached in TSDB, enabling faster joins, alerting, and long-range queries with predictable resource usage.

Notes / gotchas:- Ensure your scrape_interval and evaluation_interval frequencies align with your chosen rate windows.- If you have many dimensions, aggregate early (sum by service) to control cardinality. If using histograms, use rate on _sum/_count or histogram_quantile as needed.

EasyTechnical

84 practiced

Explain what Prometheus recording rules are and why they improve performance for dashboards. Provide a concise example (in one or two lines) of a recording rule that computes a per-service 1-minute request rate, describe where you store rule files and how often to evaluate them in Prometheus configuration, and explain scenarios where recording rules reduce query cost.

Sample Answer

Prometheus recording rules are precomputed time-series: you define a query once, Prometheus evaluates it on a schedule and stores the result as a new metric. This shifts expensive query work from dashboard-time to evaluation-time, making dashboards faster, more stable, and less resource-intensive.

Example (1-minute per-service request rate):

yaml

groups:
- name: service_rules
  rules:
  - record: service:requests:rate1m
    expr: sum by (service) (rate(http_requests_total[1m]))

Store rule files on disk (e.g., /etc/prometheus/rules/*.yaml) and reference them in prometheus.yml under rule_files. Set global scrape_interval/evaluation_interval (common: evaluate_rules every 15s or 30s) via:

yaml

global:
  evaluation_interval: 30s
rule_files:
  - "rules/*.yaml"

When recording rules reduce query cost:- Dashboards that aggregate high-cardinality labels (sum by service) instead of running costly instant queries per panel.- Complex joins/aggregations (e.g., expensive subqueries, rate/irate over long windows) precomputed once.- Alerting rules that reuse the same heavy query—record once and reference the stored metric in many alerts/panels.

EasyTechnical

77 practiced

Explain the RED (Rate, Errors, Duration) and USE (Utilization, Saturation, Errors) monitoring frameworks. Provide concrete examples of metrics you would inspect under each framework and, given a service experiencing increased latency, describe a short checklist of RED and USE metrics to inspect first to narrow down root causes.

Unlock Full Question Bank

Get access to hundreds of Metrics Analysis and Monitoring Fundamentals interview questions and detailed answers.

Join thousands of developers preparing for their dream job.