Monitoring and Alerting Questions

Designing monitoring, observability, and alerting for systems with real-time or near real-time requirements. Candidates should demonstrate how to select and instrument key metrics (latency end to end and per-stage, throughput, error rates, processing lag, queue lengths, resource usage), logging and distributed tracing strategies, and business and data quality metrics. Cover alerting approaches including threshold based, baseline and trend based, and anomaly detection; designing alert thresholds to balance sensitivity and false positives; severity classification and escalation policies; incident response integration and runbook design; dashboards for different audiences and real time BI considerations; SLOs and SLAs, error budgets, and cost trade offs when collecting telemetry. For streaming systems include strategies for detecting consumer lag, event loss, and late data, and approaches to enable rapid debugging and root cause analysis while avoiding alert fatigue.

EasyTechnical

69 practiced

Explain synthetic monitoring and canary checks. For an e-commerce site, describe a synthetic test plan for login and checkout flows: which checks to run, recommended frequency, how to vary geographic locations, and how to set alerting thresholds differently during high-traffic sale events versus normal traffic.

Sample Answer

Synthetic monitoring: proactively running scripted, automated checks from outside your system to simulate user journeys and measure availability, performance, and correctness. Canary checks: small, frequent tests (or lightweight deploys) used to detect regressions early—often run before or immediately after releases.

Synthetic test plan for an e-commerce login + checkout flow:

Checks to run- Login smoke: load login page, measure load time, submit valid credentials, confirm redirect and presence of user-specific element.- Login negative: invalid creds → show error message.- Add-to-cart: search/product page load, add item, validate cart count and price.- Checkout happy path: address selection, shipping method, payment tokenization (mocked), order confirmation page and order ID.- Checkout edge cases: out-of-stock, payment decline, coupon application.- Dependency checks: API latency for auth, cart, payments; CDN asset fetch.- Data validation: confirm totals, taxes, and order persisted.

Recommended frequency- Critical happy paths (login, checkout happy path): every 1–5 minutes.- Negative/edge-case flows: every 15–30 minutes.- Dependency/cset endpoints: 1–5 minutes for auth/payment APIs.- Heavy synthetic load tests only during controlled windows.

Geographic variation- Run checks from major user regions (US-East, US-West, EU, APAC, LATAM) with synthetic nodes close to typical user latencies.- Stagger schedules so checks run at different offsets to detect regional transient issues.- Increase sampling in regions with high user concentration or recent incidents.

Alerting thresholds (normal vs sale events)- Normal traffic: - Availability: alert if failure rate >1% over 5 minutes. - Latency: warn if 95th percentile > 2s, critical if >5s.- High-traffic sale events: - Relax transient sensitivity but protect user impact: alert if failure rate >3% sustained for 3 minutes or if critical checkout failures (payment declines) >0.5%. - Raise latency thresholds (e.g., warn >3s, critical >8s) but prioritize alerts for increased error rates or rate-limited responses from upstream services.- Use multi-channel alerts (pager for critical checkout outages, paging only for region-wide or persistent failures) and tie to runbooks. Include escalation rules and automated rollback/canary-disable on deployment-linked regressions.

Rationale: frequent happy-path checks detect user-facing regressions quickly; negative/edge checks catch business logic regressions; geo variation reveals CDN/network/regional infra issues; adaptive thresholds during sales reduce noisy alerts while ensuring high-impact failures trigger immediate response.

EasyTechnical

58 practiced

What is metric cardinality and why can high cardinality be harmful for your metrics backend? Describe three strategies to manage cardinality and give examples of when to prefer histograms, exemplars, or labels.

EasyTechnical

66 practiced

Describe how to instrument a Python Flask endpoint to record: request count, request latency histogram (broken down per-stage: auth, handler, database), and error rate. Specify which metric types to use (counter, histogram, gauge), recommended labels (route, status_code, region), and include a concise pseudo-code snippet showing where metrics are recorded in the request lifecycle.

Sample Answer

Approach: use Prometheus metric types—Counter for total requests and errors, Histogram for latency (allows buckets) and stage breakdown, Gauge for in-progress/concurrent requests. Attach labels to slice by route, method, status_code, region (and optionally error_type).

Recommended metrics and labels:- counter: http_requests_total{route, method, status_code, region}- counter: http_request_errors_total{route, method, error_type, region}- histogram: http_request_latency_seconds{route, method, region} (overall)- histogram: http_request_stage_latency_seconds{route, method, stage, region} (stage ∈ {auth, handler, db})- gauge: http_inprogress_requests{route, method, region}

Pseudo-code (Flask + prometheus_client):

python

from prometheus_client import Counter, Histogram, Gauge
from time import perf_counter
from flask import Flask, request, g

app = Flask(__name__)

REQ_TOTAL = Counter('http_requests_total', 'Total requests', ['route','method','status_code','region'])
REQ_ERRORS = Counter('http_request_errors_total', 'Request errors', ['route','method','error_type','region'])
LATENCY = Histogram('http_request_latency_seconds', 'Request latency', ['route','method','region'])
STAGE_LATENCY = Histogram('http_request_stage_latency_seconds', 'Stage latency', ['route','method','stage','region'])
INPROGRESS = Gauge('http_inprogress_requests', 'In-progress requests', ['route','method','region'])

REGION = 'us-west-2'  # example

@app.before_request
def before():
    g.start = perf_counter()
    INPROGRESS.labels(route=request.path, method=request.method, region=REGION).inc()
    # auth timing
    g.auth_start = perf_counter()
    # perform auth...
    STAGE_LATENCY.labels(route=request.path, method=request.method, stage='auth', region=REGION).observe(perf_counter() - g.auth_start)

@app.route('/items')
def handler():
    try:
        handler_start = perf_counter()
        # DB call
        db_start = perf_counter()
        # perform db...
        STAGE_LATENCY.labels(route=request.path, method=request.method, stage='db', region=REGION).observe(perf_counter() - db_start)

        STAGE_LATENCY.labels(route=request.path, method=request.method, stage='handler', region=REGION).observe(perf_counter() - handler_start)
        return "ok", 200
    except Exception as e:
        REQ_ERRORS.labels(route=request.path, method=request.method, error_type=type(e).__name__, region=REGION).inc()
        raise
    finally:
        duration = perf_counter() - g.start
        LATENCY.labels(route=request.path, method=request.method, region=REGION).observe(duration)
        INPROGRESS.labels(route=request.path, method=request.method, region=REGION).dec()

@app.after_request
def after(response):
    REQ_TOTAL.labels(route=request.path, method=request.method, status_code=str(response.status_code), region=REGION).inc()
    return response

Key points:- Use Histogram for latencies to get quantiles/buckets; add per-stage label to break down auth/handler/db.- Counter for totals and errors (monotonic).- Gauge for current concurrency.- Labels: keep cardinality low (avoid user IDs); use route pattern instead of raw path.- Instrument exceptions and ensure finally blocks decrement gauge and record latencies.

MediumTechnical

63 practiced

Design an alerting policy for a critical API whose SLO is p95 latency < 200ms. Describe how you would derive alert thresholds for SEV1, SEV2 and SEV3, define violation duration that must persist before alerting, and specify an escalation policy mapped to on-call rotations. Explain why p95 is preferred over average for user experience tracking.

Sample Answer

Requirements recap: SLO is p95 latency < 200 ms. Alerts should detect meaningful SLO violations (not blips), map to severity, include minimum persistence to avoid noise, and have a clear escalation path tied to on-call rotations.

Derived thresholds and durations (suggested):- SEV1 (Service-impacting — immediate action) - Trigger: p95 > 350 ms for 5 consecutive minutes OR p95 > 500 ms for 1 minute OR simultaneous increase in user-facing errors + high latency (e.g., error rate >1% and p95 > 250 ms). - Rationale: Indicates broad, severe degradation affecting most users.- SEV2 (Degradation — important, timely fix) - Trigger: p95 between 250–350 ms sustained for 15 minutes. - Rationale: Noticeable slowdown for many users; not catastrophic but requires remediation.- SEV3 (Minor or early-warning) - Trigger: p95 between 200–250 ms sustained for 60 minutes OR a rising trend (e.g., 10% p95 increase over 1 hour). - Rationale: Early signal that SLO is at risk; useful for investigating root causes before user impact grows.

Additional rules:- Use rolling-window evaluation (e.g., 5m, 15m, 1h windows) and require consistent exceedance across the window to reduce flapping.- Suppress alerts during known maintenance windows or automated deploys (with guardrails).

Escalation policy mapped to on-call:- SEV1: - 0–5 min: page primary on-call (urgent mobile + SMS) - 5–15 min: if unacknowledged, escalate to secondary on-call - 15–30 min: notify engineering manager and product lead - Post-30 min: invoke incident response playbook, create incident bridge- SEV2: - 0–10 min: page primary on-call (non-blocking ringtone) - 10–30 min: escalate to secondary if unacknowledged - If persists >2 hours or user impact grows → escalate to SEV1 flow- SEV3: - Create a ticket in tracking board and notify on-call via low-priority channel (Slack/email) - If unresolved and persists >24 hours or increases in severity, escalate to SEV2 flow

Why p95 (not average):- p95 captures tail latency experienced by most users (95th percentile) and surfaces worst-case user experience; average can hide spikes because fast requests lower the mean while a subset of users suffer high latency. p95 aligns alerts to user-facing impact and prevents chasing irrelevant fluctuations in average metrics.

Actions per severity examples:- SEV1: roll back recent deploys, throttle non-critical traffic, enable circuit-breakers, open incident bridge.- SEV2: investigate backend queues, DB slow queries, autoscaling, hot keys.- SEV3: collect traces, logs, schedule deeper performance analysis.

Notes / best practices:- Correlate latency alerts with error rates, CPU, GC, queue lengths, and downstream service latencies.- Tune thresholds with historical telemetry to balance sensitivity vs noise; iterate after a few incidents.- Add runbooks for each severity so on-call can act quickly and consistently.

MediumTechnical

55 practiced

Describe how you'd measure and alert on backpressure and queue lengths in an event-driven system (for example Kinesis, Kafka, RabbitMQ). Include specific metrics to expose at producers, brokers and consumers, define what constitutes critical thresholds, and outline automatic mitigations such as rate limiting producers or autoscaling consumers.

Sample Answer

Situation: In an event-driven system I treat backpressure and queue length as first-class SLOs to keep client latency and data durability healthy.

Metrics to expose (concrete names for Prometheus-style telemetry):- Producer: - publish_rate_msgs/s, publish_rate_bytes/s - publish_errors_total, publish_retries_total - producer_queue_length (local buffer), producer_send_latency_ms_p50/p95- Broker (Kafka/Rabbit/Kinesis): - partition_end_offset, log_end_offset, partition_oldest_offset - partition_retention_bytes, broker_disk_utilization_percent - incoming_bytes/s, outgoing_bytes/s - per-partition consumer_lag (consumer_committed_offset vs log_end_offset) - per-queue message_count, oldest_message_age_seconds- Consumer: - consume_rate_msgs/s, process_rate_msgs/s - process_latency_ms_p50/p95/p99 - commit_rate, consumer_lag (per-partition) - retry_count, dlq_count

Alerting & critical thresholds (examples; tune to workload):- Critical: consumer_lag > 1M msgs OR oldest_message_age > 30 minutes OR queue message_count growth > 20%/5min → Pager.- Warning: consumer_lag > 100k OR oldest_message_age > 5 minutes → Slack/SEV-3.- Resource: broker_disk_utilization > 80% or broker_cpu > 85% for 5m → Ops alert.

Automatic mitigations:- Producer-side: - Token-bucket rate limiter and exponential backoff on publish errors; expose circuit-breaker metric and stop-at-queue-full policy. - Dynamic throttling: reduce producer publish_rate by factor when broker backpressure detected (publish_errors or high broker response times).- Consumer-side autoscaling: - Horizontal scale-out when sustained consumer_lag growth rate > X msgs/min for Y minutes OR process_latency_p95 exceeds threshold. - Use Kubernetes HPA with custom metrics: per-partition lag or queue_length mapped to desired replicas. Example policy: add 1 replica per 200k backlog, cap at N. - Graceful rebalance and warm-up: prefetch limits to avoid overload, increase parallelism per consumer instance.- Broker-side: - Auto-replication and rebalancing when disk or partition hotspots detected; increase provisioned throughput (Kinesis shards, Kafka brokers) when sustained incoming_bytes/s > provisioned capacity. - Move older messages to cold storage or increase retention only as emergency.

Operational patterns:- Track derivative metrics (lag growth rate) not only absolute.- Use multi-tier alerts: slow alerts for early-warning (growth rate), urgent alerts for absolute breaches.- Post-incident: root-cause (producer flood, consumer regressions, broker throttling) and add circuit breakers, quotas, or capacity increases.

This combination of explicit metrics, growth-rate alerts, and automated rate-limiting + autoscaling keeps the pipeline stable while minimizing data loss and latency.

Unlock Full Question Bank

Get access to hundreds of Monitoring and Alerting interview questions and detailed answers.

Join thousands of developers preparing for their dream job.