Observability Fundamentals and Alerting Questions

Core principles and practical techniques for observability including the three pillars of metrics logs and traces and how they complement each other for debugging and monitoring. Topics include instrumentation best practices structured logging and log aggregation, trace propagation and correlation identifiers, trace sampling and sampling strategies, metric types and cardinality tradeoffs, telemetry pipelines for collection storage and querying, time series databases and retention strategies, designing meaningful alerts and tuning alert signals to avoid alert fatigue, dashboard and visualization design for different audiences, integration of alerts with runbooks and escalation procedures, and common tools and standards such as OpenTelemetry and Jaeger. Interviewers assess the ability to choose what to instrument, design actionable alerting and escalation policies, define service level indicators and service level objectives, and use observability data for root cause analysis and reliability improvement.

HardTechnical

84 practiced

Prepare a 30–minute postmortem outline for a deployment that unexpectedly consumed a large portion of the error budget for a service. The outline should use observability data to explain what happened, quantify impact, describe remediation, and list preventive actions. Specify which dashboards and queries you'd include and who should be on the review.

Sample Answer

Goal: Run a focused 30-minute postmortem that uses observability to explain why a deployment consumed a large portion of the error budget, quantify impact, record remediation, and create follow-ups to prevent recurrence.

Agenda (30 minutes)- 0:00–0:02 — Opening: purpose, blamelessness, scope (which deployment, timeframe, SLOs affected)- 0:02–0:10 — Observability recap: present dashboards/queries and timeline of events- 0:10–0:16 — Root-cause analysis: correlate signals, narrow failure modes- 0:16–0:22 — Impact quantification: users affected, error budget consumed, business metrics- 0:22–0:27 — Remediation & immediate changes (what we did/can do now)- 0:27–0:30 — Preventive actions, owners, deadlines, and follow-up meeting

Dashboards & queries to include (show live and snapshots)- SLO dashboard: current SLI (success rate/latency), error budget burn rate chart (last 72h)- Deployment timeline: CI/CD events overlaid with error rate spike- Errors by type: top error codes/exceptions (e.g., 5xx by endpoint) Query: sum(rate(http_requests_total{status=~"5..",service="svc"}[1m])) by (endpoint, error)- Latency distribution: p50/p90/p99 before/during/after deployment- Traffic/throughput: RPS and client retries- Resource metrics: CPU/memory, GC, threadpool saturation on hosts/pods- Dependency health: downstream service latency/error rates and DB latency/connection counts- Traces: representative distributed traces around failing requests (show root span and high-error spans)- Logs: sample structured logs filtered by error and request-id correlated with traces

Root-cause analysis approach- Correlate deployment timestamp with first error spike.- Use traces to find failing span/component.- Check host/pod rollout (canary percentage, failed pod restarts).- Verify config/feature flags and schema migrations.- Determine whether cause is code regression, config, infra, or downstream.

Impact quantification- Error budget consumed: e.g., SLO dropped from 99.95% to 99.70 over 6 hours → X% of monthly budget- Users affected: % of requests failing, number of unique users or sessions- Business KPIs: estimated lost transactions or revenue (if applicable)- Latency/throughput degradation across regions

Immediate remediation (what happened or recommended)- Roll back deployment or disable faulty feature flag (who, when)- Increase capacity or revert config if resource exhaustion- Apply hotfix and run smoke tests against canary- Notify downstream teams and customers per communication plan

Preventive actions & owners- Add pre-deploy automated SLI smoke tests and canary gating (owner: CI/CD eng; due: 2 weeks)- Tighten canary criteria and shorten rollout window (owner: release engineer; due: 1 week)- Add targeted alerting: composite alert that triggers on simultaneous rise in error rate + latency + deployment event (owner: SRE; due: 3 days)- Increase observability: add dashboards for endpoints introduced/changed by this PR and trace sampling for high-risk paths (owner: service owner; due: 1 week)- Post-deploy checklist: dependency contract checks, schema migration dry-run (owner: PM/eng; due: next sprint)- Run a game-day to rehearse rollback and mitigations (owner: SRE lead; schedule within month)

Attendees (30-minute core)- Service owner / primary engineer (presenter)- Release engineer / CI-CD owner- SRE or on-call engineer- Backend engineers who authored change- Product manager (impact context)- Downstream service owner(s) if implicated

Follow-up- Assign action items in tracker, 1-week checkpoint to verify completion.- Optional full write-up for wider audience with timeline, RCA, and lessons learned.

This outline keeps the meeting evidence-driven, timeboxed, and actionable while producing concrete observability artifacts and owners for prevention.

HardTechnical

91 practiced

You need to design alert thresholds for a business-critical batched payment job that runs every 5 minutes. The job occasionally has transient failures that recover on retry. Define alerting rules with severity levels and explain thresholds that detect real regressions while avoiding alerts for acceptable transient failures.

Sample Answer

Situation & goals: The 5-minute batched payment job is business‑critical but experiences occasional transient failures that succeed on retry. Alerts must catch real regressions (code/config/db/network) while suppressing noise from acceptable transient errors.

Proposed metrics to track:- run_success_rate: percent of runs that ultimately succeed (including automatic retries) per rolling window- first_attempt_failure_rate: percent of runs failing first attempt- average_end_to_end_latency: time from start to final success- retry_count_per_run: average retries used

Alerting rules & thresholds (rolling windows chosen to smooth transient spikes):1) Info / Operational (no paging)- Condition: first_attempt_failure_rate > 10% over 1 hour AND run_success_rate >= 98%- Action: post to ops channel, annotate run logs for investigation- Rationale: transient upstream instability — monitor only

2) Warning (on-call notify, no immediate paging)- Condition: first_attempt_failure_rate > 20% OR average_retry_count > 1.5 over 30 minutes AND run_success_rate >= 95%- Action: notify on-call, include recent failure samples and graph- Rationale: elevated retries may indicate degrading dependency; requires human attention

3) Critical / Pager (page immediately)- Condition A: run_success_rate < 95% over 15 minutes OR- Condition B: run_success_rate < 99% over 1 hour with error volume > 5 failed runs in last 15 minutes AND any downstream payment failures (DLQ, reconciliation mismatch)- Action: page SRE/eng, run rollback/playbook, open incident- Rationale: impacting payments or sustained regressions

4) SLA breach (business escalation)- Condition: run_success_rate < 99.5% over 24 hours OR cumulative failed payment value > $X- Action: exec alert, prioritize fix

Implementation details & mitigations:- Use rolling windows (15m/30m/1h/24h) to avoid alerting on single-run flukes (12 runs/hr).- Suppress alerts during planned maintenance/deploys (deploy annotations).- Attach sample failing run IDs, logs, and automated root-cause hints (timeout vs auth failure).- Auto-remediation hooks for known transient fixes (retry/backoff tuning, circuit breaker toggles) but ensure safe‑guards.- Review thresholds quarterly and after major infra changes.

This balances sensitivity to real regressions with tolerance for transient failures by combining short-window critical checks with longer-window trend detection and business-value–aware SLA alerts.

EasyTechnical

75 practiced

Explain the three pillars of observability — metrics, logs, and traces. For each pillar describe primary use cases (for example: alerting, debugging, capacity planning), strengths and blind spots when diagnosing production issues, and give a concrete example of a problem best solved by that pillar.

EasyBehavioral

90 practiced

Tell me about a time you reduced alert noise or improved observability on a project. Describe the situation, the concrete actions you took (instrumentation, alert rule changes, dashboards), how you measured improvement, and the outcome for the team or customers.

MediumTechnical

90 practiced

An alert currently pages the on-call team for every 5xx response on a public API. During deployments this generates a flood of redundant pages. Redesign the alert to be actionable while still catching real regressions — describe rules, aggregation windows, suppression strategies, and how to validate the new rule.

Sample Answer

Situation: Our on-call team was paged for every 5xx on the public API, causing a flood of redundant alerts during deployments and alert fatigue. The goal was to make alerts actionable—catch real regressions quickly without noisy pages.

Design / Rules:- Severity tiers: - P1 (page): sustained spike in user-facing 5xx rate indicating possible production regression. - P2 (ticket/Slack): transient or localized increases to investigate next business day.- Primary metric: 5xx rate per-minute normalized by traffic (5xx_count / total_requests) and affected endpoints/users.

Aggregation windows & thresholds:- Short window for detection: compute rolling 1-minute rate and compare to a 15-minute baseline (median of last 6 x 15-min windows or rolling 3-hour baseline).- Page when: current 1-min 5xx rate > baseline * 5 AND absolute 5xx rate > 0.5% AND at least N unique clients or >X requests failed (e.g., >100 failures/min).- Create endpoint-level and region-level rules: page only if multiple endpoints or regions exceed threshold, otherwise lower severity.

Suppression & noise controls:- Deployment suppression: integrate with CI/CD — automatically suppress pages for services flagged as "in-deploy" for a short window (e.g., 10–30 minutes) unless the increase exceeds an override threshold (e.g., baseline *10).- Maintenance windows: scheduled suppressions for planned work.- Rate-limit pages: deduplicate alerts using an aggregation window (group identical alerts for 10 minutes into one page).- Escalation policy: if a suppressed alert persists beyond deploy window or increases, escalate immediately.

Validation & rollout:- Run in “alert-only” (no page) mode for 2–4 weeks; analyze hits: false positives, missed incidents.- Create synthetic fault tests: inject controlled 5xx bursts and simulate deploy traffic to verify suppression/override behavior.- Monitor key metrics: time-to-detect, pages/day, missed incidents count. Adjust thresholds (multipliers, absolute mins) iteratively.- After tuning, roll out staged paging: subset of services/team, then full rollout.

Why this works:- Normalizes for traffic, avoids paging on expected low-volume blips.- Uses baseline comparison to detect regressions, not absolute counts alone.- Integrates with deployment metadata to avoid known noise but preserves safety via override thresholds.- Validation through shadow mode and fault injection ensures it catches real regressions without creating new blind spots.

Unlock Full Question Bank

Get access to hundreds of Observability Fundamentals and Alerting interview questions and detailed answers.

Join thousands of developers preparing for their dream job.