Monitoring, Logging, and Operational Visibility Questions

Understand that running systems need constant visibility. Know basic monitoring concepts: metrics (numerical measurements like CPU, memory, request count), logs (detailed event records), and alerts (notifications when issues occur). Know the monitoring tools: CloudWatch (AWS), Azure Monitor (Azure), Cloud Operations/Stackdriver (GCP). Understand what should be monitored: application health (uptime, error rates), infrastructure health (CPU, memory, disk), and security events (access logs, permission denials). Know that proper monitoring enables quick issue detection and troubleshooting. Be familiar with dashboard creation (visualizing metrics) and alert configuration (notifying on problems). Understand log aggregation—collecting logs from multiple sources for centralized analysis.

HardTechnical

50 practiced

Compare head-based, tail-based and adaptive sampling strategies for distributed tracing in production. For each method describe implementation complexity, bias introduced, storage cost implications, and how you would preserve traces that matter for tail-latency debugging.

Sample Answer

High-level summary: head-based sampling decides at trace start (client/ingress), tail-based decides after seeing most or all spans (collector/ingest), adaptive adjusts sampling rates dynamically based on traffic/conditions. Each has trade-offs for implementation, bias, storage, and tail-latency debugging.

Head-based sampling- Implementation complexity: Low. Implement at SDK/ingress with deterministic or probabilistic sampler (OpenTelemetry supports this).- Bias introduced: Strong bias toward frequent/short requests; rare slow/errorful traces are likely dropped because decision is made before observing outcome.- Storage cost: Predictable and low; fixed sampling fraction reduces downstream storage/ingest load.- Preserving tail traces: Add rule-based overrides — e.g., always sample on error status codes, high response size, or specific endpoints. Use trace-context propagation and sticky sampling keys (user-id, request-id) to prefer sampling important users.

Tail-based sampling- Implementation complexity: High. Requires buffering spans at ingest, correlation across services, and a decision pipeline that can inspect full trace (or most spans) before committing to storage. Needs scaling and backpressure handling.- Bias introduced: Much lower for latency/error signals because decision uses observed behavior; better at preserving rare but important traces.- Storage cost: Higher transient cost (store/forward buffers) and higher retention if more interesting traces are kept; allows lower long-term storage by only retaining important traces.- Preserving tail traces: Natural fit — decision rules can keep traces with high p95/p99 latency, errors, or SLA violations. For very high throughput, implement fast heuristics (e.g., sample if any span > latency threshold) to reduce buffering.

Adaptive sampling- Implementation complexity: Medium–High. Requires telemetry-driven control-plane: metrics on traffic, error/latency rates, and dynamic adjustment logic (rate controllers, PID controllers, ML-based).- Bias introduced: Reduced if tuned; may still bias toward classes seen frequently unless controllers consider cardinality.- Storage cost: Variable and more efficient than naive tail-based; adapts sampling to keep costs within budget while prioritizing important traces.- Preserving tail traces: Configure policies that temporarily raise sampling for endpoints or user segments showing increased tail latency/errors. Combine with reservoir sampling per-key to ensure low-volume but high-impact keys aren’t starved.

Practical patterns to preserve traces for tail-latency debugging (SRE checklist)- Hybrid approach: head-based baseline + tail-based overrides. Probabilistic sampling at client + short-lived collector buffer that promotes traces if they contain errors/slow spans.- Minimal metadata flow: Always send a small “trace preview” (trace ID, latency, status, key spans or aggregated stats) synchronously to collector; only full spans for promoted traces to reduce buffer pressure.- Deterministic sticky sampling: seed sampler with high-cardinality keys (user, tenant) to ensure consistent capture across retries and correlated traces.- Reservoirs & per-service quotas: keep N traces per minute per service/endpoint to avoid domination by a noisy endpoint.- Backfill linking: persist logs/metrics with trace IDs so analysts can reconstruct or request full traces when alerts fire.- Observability SLAs: build sampling budgets tied to SLOs (e.g., always capture 99% of error traces affecting SLO).

Trade-offs summary:- Head-based: simplest and cheapest, but risks missing the very signals you need for tail-latency analysis.- Tail-based: best signal fidelity for debugging but operationally complex and transiently expensive.- Adaptive: best practical middle ground—automates cost-control while preserving high-value traces when behavior changes.

For production SREs: start with head-based for cost control, add lightweight trace previews and rule-based overrides, then evolve to adaptive or partial tail-based promotion as traffic and need justify the operational complexity.

EasyTechnical

63 practiced

Explain the difference between 'monitoring' and 'observability' from an SRE perspective. Include concrete examples of metrics, logs, and traces, describe when observability is needed over basic monitoring, and explain operational implications for incident response and debugging.

Sample Answer

Monitoring is about known-knowns: collecting predefined signals (metrics, logs, basic traces) and alerting on thresholds or SLOs. Observability is the property of a system that lets you ask new questions about unknown-unknowns — it provides rich, correlated telemetry so you can explore system behavior without predefining every alert.

Examples:- Metrics (monitoring): CPU usage, request rate (RPS), error rate, 95th/99th latency percentiles. Alerts: error-rate > 1% for 5m, SLO burn > threshold.- Logs (both): structured application logs with request IDs, stack traces, and contextual fields (userID, feature flag). Monitoring uses counts of error logs; observability uses logs to investigate flow and root cause.- Traces (observability-heavy): distributed traces that show span timings across services, where a single request spends time. Useful for pinpointing latency in a specific downstream call.

When observability is needed:- Incidents with no prior alert pattern (partial degradations, cascading failures, weird resource spikes).- Performance regressions whose root cause isn't apparent from metrics alone.- Debugging race conditions, rare errors, or interactions across many services.

Operational implications:- Incident response: Monitoring gives fast detection and routing to runbooks; observability enables rapid hypothesis testing and root-cause discovery (correlate traces + logs + metrics to find offending service/version).- Debugging: With observability you can pivot on a trace id, examine related logs, inspect downstream metrics, and form minimal reproductions. This shortens MTTD/MTTR.- Investment trade-offs: Monitoring is cheaper and essential for SLOs; observability requires instrumentation (distributed tracing, structured logs, high-cardinality metrics) and storage/cost planning.- Team practices: Define alerting for actionable signals, instrument spans and request IDs, keep runbooks updated, and run postmortems that expand instrumentation gaps found during debugging.

MediumTechnical

62 practiced

Write a CloudWatch Logs Insights query that finds the top 5 request paths producing HTTP 500 responses in the last 1 hour given JSON log lines with fields 'timestamp', 'path', 'statusCode', 'requestId'. Provide the query and explain aggregation/window choices.

Sample Answer

Query:

fields @timestamp, path, statusCode, requestId
| filter @timestamp >= ago(1h)                      # limit to last 1 hour
| filter statusCode >= 500 and statusCode < 600     # only 5xx errors (or use statusCode = 500 if strictly 500)
| stats count(*) as errorCount, count_distinct(requestId) as uniqueRequests by path
| sort errorCount desc
| limit 5

Explanation:- I filter to the last 1 hour with @timestamp >= ago(1h) so the query scope matches the requirement and remains performant by reducing scanned data.- I filter on statusCode in the 5xx range; if you need strictly HTTP 500, change that line to filter statusCode = 500.- Aggregation: stats count(*) by path gives total 5xx occurrences per request path. I also include count_distinct(requestId) to show unique request occurrences (useful if logs can contain repeated lines for the same request).- Sorting and limit: sort by errorCount desc and limit 5 returns the top 5 offending paths.- Window choice: a 1-hour window is appropriate for recent incident detection; for trend analysis use sliding windows (e.g., bin(1m) with stats sum(...) over time) or run the same query for different windows (15m, 24h) to spot persistent vs transient hotspots.

HardTechnical

58 practiced

Design an online anomaly detection algorithm for SRE time-series that handles seasonality and trends, minimizes false positives, and runs with O(1) amortized update per sample. Describe data structures, math (e.g., EWMA, rolling z-score, Holt-Winters), threshold selection and validation approach.

Sample Answer

Situation: We need a streaming anomaly detector for SRE metrics (latency, error rate) that handles trends + seasonality, keeps false positives low, and costs O(1) amortized per new sample.

Design summary (online Holt-Winters + robust residual stats):- Model: use multiplicative Holt-Winters (level, trend, seasonal) in an online form with fixed season length s. Update formulas per sample t: - level Lt = α * (Xt / St−s) + (1−α) * (Lt−1 + Tt−1) - trend Tt = β * (Lt − Lt−1) + (1−β) * Tt−1 - seasonality St = γ * (Xt / (Lt−1 + Tt−1)) + (1−γ) * St−s - forecast Ft = (Lt−1 + Tt−1) * St−s Use multiplicative to handle scale changes; additive variant if metrics can be negative.- Residual r_t = Xt / Ft (or Xt − Ft for additive). Maintain robust online statistics of residuals: - EWMA of log-residuals: m_t = λ*m_{t−1} + (1−λ)*log r_t - EWMA variance: v_t = λ*v_{t−1} + (1−λ)*(log r_t − m_t)^2 - Convert to z-score: z_t = (log r_t − m_t)/sqrt(max(v_t, eps))- Data structures (O(1) per sample): - Circular array of length s storing past seasonal factors St−s (constant memory ~O(s)) - Scalars for Lt, Tt, m_t, v_t - Optionally a small fixed-size reservoir for bootstrapping initial seasonality - All updates are constant-time.

Threshold selection and minimizing false positives:- Use z-score thresholds (e.g., |z| > 3) but adapt thresholds per-channel using historical false-positive budget. Calibrate α,β,γ,λ by maximizing precision subject to recall constraints on labeled historical incidents.- Use asymmetric thresholds (higher for alerts on metric increases for latency/errors) and rate-based suppression (aggregate N anomalies in window before page).- Use robust statistics (log transform + EWMA variance) to reduce sensitivity to spikes; clip influence of extreme points.

Validation and tuning:- Backtest on historical data: compute ROC/PR, calibrate threshold for target false positive rate (e.g., alerts/week).- Inject synthetic anomalies covering amplitude, duration, and timing to measure detection delay and true positive rate.- Run shadow alerts in prod for a period: route to slack but not on-call; log operator feedback to refine thresholds.- Continuous evaluation: track precision, MTTA, alert fatigue metrics; tune season length s (daily/weekly) and smoothing params via grid/BO.

Complexity:- Time: O(1) per sample (constant arithmetic + circular buffer access)- Space: O(s) per time series

Trade-offs:- Holt-Winters handles seasonality and trend with small state; if seasonality drifts nonlinearly, consider online STL or stateful ARIMA (more CPU/state).- For many metrics, shard detector per-metric and use adaptive parameter presets per metric class to save ops.

This design provides a conservative, robust online detector with low computational cost, explicit seasonality/trend handling, and a validation pathway to minimize false positives.

MediumTechnical

59 practiced

You receive many duplicate and low-importance alerts each week. Propose concrete strategies to reduce alert fatigue: include alert grouping and deduplication, symptom-first alerting, suppression during deploys, dynamic thresholds, and on-call ergonomics with examples and trade-offs.

Sample Answer

Situation: As an SRE, frequent duplicate and low‑importance alerts cause on‑call burnout and mask real incidents. Below are concrete strategies, examples, and trade‑offs to reduce alert fatigue.

1) Alert grouping & deduplication- Strategy: Group related signals into a single incident object (by trace id, host, service/component) and de‑duplicate identical alerts within a sliding window.- Example: Combine CPU spike alerts across N hosts into “cluster CPU pressure” with aggregated top hosts.- Trade‑offs: Risk of over-grouping (hiding an individual host issue). Mitigation: allow drill‑downs and keep per‑entity symptom metrics.

2) Symptom‑first alerting- Strategy: Alert on customer‑visible symptoms (error rate, latency) before infrastructure manifestations.- Example: Page when 5xx rate > 1% for 5m, but create non‑paging infra alerts (disk, DB connections) to a dashboard.- Trade‑offs: Might delay catching root cause early; mitigate with escalations: symptom alert → on‑call looks at linked infra signals.

3) Suppression during deploys- Strategy: Auto‑suppress or route to a “deploy noise” channel while CI/CD flags an active deployment; short lived, transparent suppression.- Example: If deployment flag active and alert baseline is within expected transient window, suppress paging and log to a deploy ticket.- Trade‑offs: Could suppress real regressions. Mitigate by tighter windows, higher‑severity alerts still paging, and manual override.

4) Dynamic thresholds & anomaly detection- Strategy: Use baseline/seasonal models or adaptive thresholds (percentile, rate of change) rather than static thresholds.- Example: Alert when latency > baseline + 3σ or 95th percentile jumps 2x.- Trade‑offs: Complexity and false positives from model drift. Mitigate with model retraining, guardrails, and human review of new alerts.

5) On‑call ergonomics & processes- Strategy: Improve runbooks, reduce cognitive load, provide one‑click remediation and contextual context in alerts (link to dashboards, runbooks, deploy IDs).- Example: Alert payload includes top 5 logs, related traces, and “action buttons” (restart service, rollback).- Trade‑offs: Automation risks accidental actions; require safeguards (confirmation, auth).

Operational recommendations:- Implement phases: dedupe+grouping → symptom alerts → deploy suppression → dynamic thresholds → automation.- Measure impact: alert volume/week, mean time to acknowledge, noisy alert ratio; iterate with SLOs.- Governance: regular alert reviews with owners, retire low‑value alerts, and keep a “noisy alerts” backlog.

These measures reduce noise, prioritize customer impact, and preserve on‑call capacity while balancing risk with careful mitigations and metrics.

Unlock Full Question Bank

Get access to hundreds of Monitoring, Logging, and Operational Visibility interview questions and detailed answers.

Join thousands of developers preparing for their dream job.