Observability Fundamentals and Alerting Questions

Core principles and practical techniques for observability including the three pillars of metrics logs and traces and how they complement each other for debugging and monitoring. Topics include instrumentation best practices structured logging and log aggregation, trace propagation and correlation identifiers, trace sampling and sampling strategies, metric types and cardinality tradeoffs, telemetry pipelines for collection storage and querying, time series databases and retention strategies, designing meaningful alerts and tuning alert signals to avoid alert fatigue, dashboard and visualization design for different audiences, integration of alerts with runbooks and escalation procedures, and common tools and standards such as OpenTelemetry and Jaeger. Interviewers assess the ability to choose what to instrument, design actionable alerting and escalation policies, define service level indicators and service level objectives, and use observability data for root cause analysis and reliability improvement.

HardTechnical

136 practiced

Design blackbox monitoring checks to verify DNS resolution, TCP handshake, and TLS certificate validity for a global fleet of endpoints. Explain how you would schedule checks from multiple regions, reduce false positives due to transient network issues, and alert on meaningful degradations that affect customers.

Sample Answer

Requirements & constraints:- Blackbox checks must verify: DNS resolution correctness and freshness, TCP 3‑way handshake success/latency, and TLS chain validity/expiry/OCSP stapling.- Global coverage with diversity (cloud providers, ISPs, continents).- Minimize false positives from transient network issues; alert only on customer-impacting degradations.

High-level design:- Distributed probe fleet: small probe agents (~lightweight containers) deployed in ≥12 regions (multi‑AZ, multi‑cloud, third‑party probes) to maximize path diversity.- Central controller schedules probes, aggregates results into metrics service (Prometheus + remote write) and event store (Kafka).- Dashboard + alerting (Grafana/Alertmanager, optionally PagerDuty).

Checks (per probe) — run sequentially for one target per probe execution:1) DNS check- Query authoritative and recursive resolvers and compare: expected A/AAAA/CNAME, TTL, RRSIG if DNSSEC enabled.- Validate consistency across probes and against authoritative NS.- Measure DNS latency and SERVFAIL/NXDOMAIN rates.2) TCP handshake- Open TCP socket to target port; measure SYN→SYN/ACK RTT and full connection establishment time. Optionally send small application hello (for protocol-aware).- Validate that the connection can be established repeatedly.3) TLS validation- Perform full TLS handshake, validate chain against system roots, check expiration (alert window: cert expiring within 30d), check OCSP stapling and revocation status, validate SNI correctness, and measure handshake time and TLS version/cipher suites.

Scheduling & execution- Staggered randomized schedule per region to avoid synchronized bursts: e.g., each probe runs a target every 60s ± jitter(0–15s).- Multi-region cadence tiers: high‑priority endpoints (every 30s), normal (60s), low (5m).- Distributed retries with backoff: on first failure, immediate retry from same probe (1s), then a second retry after short delay; also try same check from 2 other regions within a 30s window.

Reduce false positives- Multi-dimensional deduplication and voting: - Require N-of-M region failures: e.g., alert only if ≥3 regions fail the same check within 2 minutes OR ≥20% of probes fail (choose whichever indicates customer impact). - Use exponential backoff grouping: transient single-probe failures suppressed; persistent failures escalate.- Correlate with network signals: - If failures are isolated to one ASN/region, mark as network-blip (informational) unless errors spread. - Compare synthetic failures to real-user metrics (RUM, CDN logs, server logs). If synthetic is failing and RUM shows user impact > threshold, escalate immediately.- Adaptive noise reduction: - Auto-blacklist misbehaving probe endpoints and schedule health checks for probes themselves; re-enable after health confirmation. - Use historical baseline and anomaly detection (rolling windows) to suppress alerts for known maintenance windows or expected variance.

Alerting & meaningful degradations- Define SLO-mapped alerts: - Severity P1 (customer-impacting outage): DNS resolution failures or TCP/TLS failure across ≥50% of regional probes for >3min OR synthetic failure + RUM error rate increase >X% of sessions. - Severity P2 (degradation): latency/handshake RTT/p95 TLS handshake increase causing >10% user error budget burn in 30m. - Cert warnings: notify cert owner at 30d/14d/7d before expiry; P1 if expired.- Alert enrichment: include probe list, regions impacted, ASN distribution, recent traceroutes, DNS responses, TLS chain details, and timestamps.- Escalation policy ties to SLO error budget: automatic paging only if alert pushes error budget thresholds.

Operational considerations- Probe observability: metrics for success rate, latencies, probe health, noise rates. Store raw events for debugging (pcaps optional for security).- Security: probes validate certificates, do not retain private data, use ephemeral keys; outbound IPs documented for allowlisting.- Testing & runbooks: maintain runbooks for common failures—DNS misconfiguration, CA expiry, load balancer misrouting, BGP issues.- Cost/scale: use mix of self-hosted probes + SaaS synthetic monitoring to cover edge cases; sample fewer low-priority targets.

Example alert rule (conceptual):- Fire if: (count_failed_probes(target) >= 3 AND failed_regions_fraction >= 0.2) for 3 minutes AND real_user_error_rate(target) > baseline+3σ OR synthetic_failure_persistent > 3m.

Why this works- Multi-region diversity reduces false positives from local network flakiness.- Voting + correlation with real-user telemetry ensures alerts reflect customer impact, preserving on‑call focus and protecting the error budget.- Automated probe health & adaptive suppression keep noise low while ensuring meaningful, actionable alerts.

HardTechnical

89 practiced

Design a telemetry pipeline resilient to intermittent network connectivity between edge data centers and a central collector. Describe transport choices (gRPC vs HTTP), buffering and durable local storage, ordered delivery needs, retry strategies, and schema evolution handling so new telemetry fields do not break older collectors.

Sample Answer

Requirements clarification: central collector must receive telemetry reliably from edge DCs despite intermittent networks, tolerate bursts, preserve order where required, and allow schema evolution without breaking older collectors.

Transport choices:- gRPC (HTTP/2) for persistent bi-directional streams, lower latency, built-in flow control and multiplexing — good for high-throughput, ordered streams and efficient retries.- HTTP/1.1/REST (or HTTP/2 unary) for simpler fire-and-forget or management calls. Use HTTP when intermediaries/proxies block long-lived streams.- Recommendation: primary: gRPC streaming with HTTP fallback.

Buffering & durable local storage:- Use an on-disk write-ahead queue (WAL) per telemetry type (append-only files or RocksDB/leveldb) sized by retention policy. Writes are synchronous to disk (fsync) to avoid data loss.- Memory ring buffer for low-latency, flushed to WAL periodically or when thresholds hit. Background compaction of WAL to avoid unbounded growth.- Enforce quota and backpressure: when local storage full, apply configurable policies (drop oldest, drop lowest-priority, or stop ingest with backpressure).

Ordered delivery:- Preserve order by keeping sequence numbers per logical stream (e.g., per host/source). gRPC streaming + single WAL per stream preserves order.- For high parallelism, shard by source ID so ordering guarantees remain per-shard.- Support at-least-once delivery; provide idempotency keys or sequence numbers for deduplication at collector to achieve effectively-once processing if needed.

Retry strategies:- Exponential backoff with jitter for transient errors.- Fast retry for short disconnects using in-memory buffer; longer outages rely on WAL replay.- Circuit-breaker to avoid thrashing collector under load; health-checks & dynamic backoff based on collector ACKs/credits.- ACK model: collector sends ack per batch/offset; clients advance WAL pointers only after ack to guarantee durability.

Schema evolution:- Use a self-describing, forward/backward-compatible serialization (Protocol Buffers with explicit field numbers and optional fields, or Avro with schemas).- Rules: never reuse field numbers; add new fields as optional or with default values; avoid changing semantics of existing fields.- Versioning: include schema version in payload metadata. Collector should ignore unknown fields and use defaults for missing fields.- Migration plan: support dual-readers during rollout; collectors negotiate supported schema versions and log dropped/unknown fields for auditing.- Provide validation hooks and a compatibility test suite in CI that verifies new telemetry schemas are compatible with prior collectors.

Operational considerations:- Metrics: local buffer usage, WAL size, retry counts, dropped telemetry, ack lag.- Alerts on sustained WAL growth or high drop rates.- Provide tooling to re-play WALs to collector after outage and to inspect stored telemetry for debugging.

This design balances durability, ordered delivery per source, efficient transport using gRPC, robust retries with WAL-backed persistence, and safe schema evolution using protobufs/versioning.

EasyTechnical

94 practiced

What is OpenTelemetry and what are its main components (SDKs, Collector, instrumentations)? Describe how you would enable auto-instrumentation for a Python web service and what configuration choices you must consider for exporting metrics, logs, and traces to a backend.

Sample Answer

OpenTelemetry is an open standard and set of tools for collecting traces, metrics, and logs from applications and services so you can send them to observability backends. For an SRE it's useful because it standardizes telemetry, reduces vendor lock-in, and lets you centralize processing/forwarding.

Main components- SDKs: Language-specific libraries (opentelemetry-api + opentelemetry-sdk) that generate telemetry, provide context propagation, sampling, and batching.- Instrumentations: Libraries (automatic or manual) that hook into frameworks, DB drivers, HTTP clients, etc., to produce traces/metrics with minimal code changes.- Collector: A standalone agent/service (receivers → processors → exporters) that can receive OTLP (or other) telemetry, enrich/filter/aggregate it, and forward to one or more backends.

Enabling auto-instrumentation for a Python web service1. Install packages:

bash

pip install opentelemetry-api opentelemetry-sdk \
     opentelemetry-instrumentation opentelemetry-exporter-otlp

2. Run your app with the auto-instrument runner:

bash

opentelemetry-instrument --service-name myservice python app.py

This auto-enables supported instrumentations (Flask/Django/requests/DBs). For explicit control, install specific instrumentation packages (e.g., opentelemetry-instrumentation-flask) or initialize instrumentation in code.3. Optionally provide a small bootstrap config (environment variables or SDK code) for resources, sampling, and exporter settings.

Configuration choices for exporting metrics, logs, traces- Transport & endpoint: OTLP over gRPC/HTTP — set OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_HEADERS. Decide whether agents send directly to backend or via a Collector.- Use a Collector when you need local buffering, protocol translations, batching, sampling, enrichment, or to forward to multiple backends.- Sampling: configure sampler (trace percentage or parent-based) to limit high-cardinality trace volume.- Batching and retries: tune batch size, timeout and retry/backoff to avoid blocking app threads and to handle backend outages.- Security & auth: TLS, mTLS, API keys or headers — configure in exporter/collector.- Metrics specifics: choose aggregation temporality (cumulative vs delta), scrape vs push model, and histogram buckets if supported.- Resource attributes & semantic conventions: set service.name, environment, version for correct filtering/alerting.- Correlation: ensure trace IDs are attached to logs (configure log instrumentation or use Collector to correlate).- Cardinality control & attribute filtering: drop high-cardinality attributes before export (in SDK or Collector) to control cost.

In practice: run opentelemetry-instrument locally pointing exporters at your Collector (OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317), tune sampling and batching in Collector config, and enable attribute filtering in the Collector to protect backend cost and performance.

HardTechnical

96 practiced

A production service reports a 10% increase in p95 latency after enabling distributed tracing; profiling shows the tracer adds CPU and alloc overhead. Propose a structured plan to evaluate and reduce tracing overhead to acceptable levels without losing critical observability. Consider sampling, async export, batching, and instrumentation granularity.

Sample Answer

Goal clarification & success criteria- Target: reduce additional p95 latency from +10% to <= +1–2% or meet SLO. - Metrics: p95 latency, CPU %, heap allocs, trace throughput, trace export latency, dropped spans. Establish baseline and acceptable observability loss (e.g., retain 95% of error/latency-significant traces).

Step 1 — Measure & isolate- Re-run profiler with tracing enabled/disabled and collect: CPU flame graphs, allocation hotspots, lock/contention, and syscall latency. Correlate increased p95 requests to tracing activity (sampling decision, span creation, serialization, export).- Identify cost contributors: span allocation, context propagation, sync export, heavy serializers (JSON), filters.

Step 2 — Short experiments (canary)Implement one change at a time in a small subset (5–10%) to measure impact.

Step 3 — Reduce instrumentation cost- Reduce granularity: convert low-value spans (fast DB calls, internal helpers) into metric events or remove them. Keep high-value spans (external RPCs, slow DB queries).- Replace heavy objects with lightweight span attributes; avoid large baggage.- Use lazy/span-on-demand APIs: only create spans when sampling decision indicates capture.

Step 4 — Sampling strategy- Head-based sampling with adjustable rate for high throughput endpoints (e.g., 1–5% default).- Tail-based (adaptive) sampling for latency/error detection: accept all traces short-term, sample more for traces with high latency/error. Implement trace aggregator/collector to make tail decisions.- Dynamic sampling: increase sample rate on anomalous periods or for specific routes/users tied to incidents.- Ensure deterministic sampling key (trace-id hashing) to preserve trace continuity.

Step 5 — Async export & batching- Switch to fully async non-blocking exporter; keep small bounded queues to avoid unbounded memory growth.- Use batching with larger batch size and linger timeout (e.g., batch=100, linger=50–200ms) to amortize serialization cost.- Compress and use efficient binary protocols (protobuf/grpc) instead of JSON over HTTP where possible.- Backpressure: when exporter queue full, drop low-priority spans or mark as sampled=false to avoid impacting request latency.

Step 6 — Runtime controls & safeguards- Expose runtime flags to adjust sample rates and batch sizes without deploys.- Circuit-breaker: if tracer CPU alloc > threshold or queue latency grows, automatically reduce sampling to safe minimum.- Observe: track dropped spans, exporter errors, and trace completeness.

Step 7 — Validate & iterate- Run canary, measure p95, CPU, allocs, error detection fidelity (how many errors/latency spikes still captured). Use A/B comparison and statistical tests.- If still high, consider offloading sampling/export to sidecar or eBPF-based lightweight capture, or push decision into ingress/load balancer to avoid in-process cost.

Trade-offs & rationale- Less sampling/granularity reduces observability but preserves latency. Tail-based + adaptive sampling retains critical traces (errors/slow) while cutting cost. Async batching shifts CPU from critical path. Runtime controls enable fast rollback.

Example quick config suggestions- head-sample: 0.02 for high-QPS endpoints- batch: max_spans=200, flush_interval=100ms- queue_size: 10000, drop_policy: drop_oldest_for_low_priority

This structured, iterative approach ensures you measure impact, prioritize critical traces, and progressively reduce overhead while preserving actionable observability.

MediumTechnical

103 practiced

Explain metric cardinality and why allowing high-cardinality labels such as user_id or request_id on frequently-scraped metrics is dangerous. Propose three alternative approaches to get per-customer insights without creating cardinality explosion in Prometheus-style systems.

Unlock Full Question Bank

Get access to hundreds of Observability Fundamentals and Alerting interview questions and detailed answers.

Join thousands of developers preparing for their dream job.