Monitoring and Alerting Questions

Designing monitoring, observability, and alerting for systems with real-time or near real-time requirements. Candidates should demonstrate how to select and instrument key metrics (latency end to end and per-stage, throughput, error rates, processing lag, queue lengths, resource usage), logging and distributed tracing strategies, and business and data quality metrics. Cover alerting approaches including threshold based, baseline and trend based, and anomaly detection; designing alert thresholds to balance sensitivity and false positives; severity classification and escalation policies; incident response integration and runbook design; dashboards for different audiences and real time BI considerations; SLOs and SLAs, error budgets, and cost trade offs when collecting telemetry. For streaming systems include strategies for detecting consumer lag, event loss, and late data, and approaches to enable rapid debugging and root cause analysis while avoiding alert fatigue.

EasyTechnical

63 practiced

Explain how to detect and quantify consumer lag in a Kafka-like system using standard broker and consumer metadata. Define the metric for partition lag, which offsets to compare (consumer committed offset vs log end offset), and how to detect a stuck consumer versus a healthy backlog that will drain.

MediumSystem Design

48 practiced

Design a monitoring and alerting plan for an ETL pipeline that handles 100k events/sec with near-real-time requirements (dashboard latency <= 5 minutes). Include required signal types, key metrics, suggested alerting approaches (threshold, baseline, anomaly detection), dashboards for different audiences, and how you'd control telemetry cost at scale.

Sample Answer

Requirements & constraints:- 100k events/sec, near-real-time dashboards with ≤5 min max end-to-end latency.- BI-focused: ensure data correctness, freshness, completeness, and performance for downstream reporting.

Signals to collect (high level):- Ingest: events/sec, bytes/sec, partition lag, input error rate, schema-change events- Processing: throughput per stage, processing latency per stage, success/failure counts, retry counts, backpressure indicators- Delivery: write throughput to warehouse, commit latency, sink error rate, rows per batch- Data quality: row-count deltas vs expected, cardinality changes, null-rate by field, validation failures- Infrastructure: CPU, memory, disk IO, network, queue lengths- User-facing: dashboard refresh time, query latency, missing/partial data markers

Key metrics (examples & targets):- End-to-end latency (arrival → visible in BI): target ≤5 min, alert at >4m (warn), >5m (critical)- Events/sec vs expected baseline (+/- %)- Pipeline lag (e.g., Kafka consumer lag) > threshold- Failed records rate >0.01% (threshold)- Row-count mismatch vs previous day/rolling baseline >X%- Schema-change detection (immediate alert)

Alerting approaches:- Threshold alerts for hard SLAs (end-to-end latency, lag, failure rate, sink errors). Use multi-level: warning/critical.- Baseline/seasonal thresholds for traffic-sensitive metrics (events/sec, row counts) using rolling-window percentiles (compare to same hour/day last week).- Anomaly detection for data quality signals and unusual cardinality or null-rate spikes using simple statistical models (z-score, EWMA) or lightweight ML (Prophet, STL) if available.- Alert routing: critical infra & SLA alerts -> on-call SRE/ETL owner (SMS/phone); data-quality alerts -> BI data steward and producer team (email/Slack); informational -> dashboard/watchlists.

Dashboards (audience + views):- Executive/Stakeholder: end-to-end freshness SLA, % dashboards up-to-date, recent incidents, business impact summary (rows missing, affected reports).- BI Team / Data Stewards: E2E latency heatmap, failed records table, schema-change history, data-quality KPIs, recent ingest volumes vs baseline, top impacted datasets.- Engineering / SRE: detailed per-component throughput/latency, consumer lag, worker utilization, error traces, retry/backpressure charts.- Alert console: current active alerts with severity, runbook links, last occurrence, ownership.

Practical practices:- Tag metrics by dataset, environment, team, and pipeline stage for filtering and alert scoping.- Include runbooks and owner metadata in alerts (auto-attach playbook link and rollback steps).

Controlling telemetry cost:- Tier metrics: high-cardinality low-frequency traces sampled; keep full-resolution only for critical signals (SLA, errors). Aggregate/rollup older metrics (e.g., 1s→1m→1h).- Use event sampling and tail-sampling for traces; store raw error events but sample normal events.- Push compute-side aggregations (emit per-minute aggregates rather than per-event counters).- Retention policy: short high-resolution retention (e.g., 7d at 1s/1m), longer low-res retention for trend analysis (90d at 1h).- Use open-source collectors (Prometheus + Cortex, Grafana, OpenTelemetry) and filter/transform before ingestion to SaaS monitoring to reduce ingest costs.- Monitor telemetry cost as a metric and enforce quotas per pipeline/team.

Success criteria:- Alerts detect incidents before business dashboards exceed SLA.- False positive rate low via tuned baselines.- Telemetry cost stays within budget while preserving actionable observability.

EasyTechnical

94 practiced

Define alert fatigue and describe five concrete strategies you would implement in a BI environment to reduce alert noise without missing important incidents. Include examples such as grouping alerts, deduplication, dynamic thresholds, enrichment, and playbook automation.

MediumTechnical

95 practiced

A dashboard shows a spike in P99 latency for a core query. Walk through a concrete triage process (logs, metrics, traces, SQL) to identify whether the cause is (a) upstream data skew, (b) compute resource saturation, or (c) a query plan regression. Include queries or commands you would run and what signals you'd expect to see.

Sample Answer

Start by clarifying scope: which dashboard/query spike (query id, time window). Then follow a step-by-step triage using logs, metrics, traces, and SQL.

1) Establish timeline & baseline- Query the BI tool / query history for the offending query id and timestamps.

-- example (postgres): recent runs and durations
SELECT query, state, start_time, end_time, total_time
FROM pg_stat_activity_history
WHERE query LIKE '%<core_query_name>%'
AND start_time > now() - interval '2 hours'
ORDER BY start_time DESC;

Signal: sudden jump in P99 at a specific timestamp vs gradual drift.

2) Check system metrics (compute saturation)- Look at CPU, memory, IO, and network around spike (Prometheus/Cloud metrics).

# Prometheus example
rate(node_cpu_seconds_total[5m]) by (instance)
node_memory_MemAvailable_bytes
rate(node_disk_io_time_seconds_total[5m]) by (instance)

Signal for (b): high CPU% across DB workers, memory pressure (swap), high disk queue length or high read latency concurrent with query latency spike. If saturation coincides with many queries, suspect resource contention.

3) Inspect query traces / EXPLAIN plans (query plan regression)- Pull query plan before and during spike.

-- capture recent plan (Postgres)
EXPLAIN (ANALYZE, BUFFERS, FORMAT JSON) <core_query>;

Signal for (c): plan shape changed (nested loop vs hash join), missing index usage, dramatically higher estimated vs actual rows, unexpected sequential scans. If a new plan appears only after stats update or parameter sniffing, that's a regression.

4) Look for upstream data skew / cardinality changes- Check distribution and volumes of key partition/join columns over time.

SELECT key_col, count(*) AS cnt
FROM source_table
WHERE ingestion_time >= now() - interval '1 day'
GROUP BY key_col
ORDER BY cnt DESC
LIMIT 50;

Signal for (a): one key suddenly dominates (hot partition), extreme growth in table size or skewed join leading to very large intermediate rows; if plan is same but actual rows processed rose hugely.

5) Correlate traces and logs- Use APM/tracing (Jaeger/NewRelic) to view span breakdown: which stage spent most time (scan, join, network).Signal mapping:- If spans show database CPU/IO waits and system metrics high → compute saturation (b).- If spans show a particular operator (join/aggregate) exploded in time with plan showing bad operator → plan regression (c).- If spans show disproportionate data transfer from upstream service or a single partition producing huge payloads → data skew (a).

6) Quick mitigation experiments- For suspected (c): force replan or add index and compare EXPLAIN.

SET enable_nestloop = off; EXPLAIN ANALYZE <core_query>;

- For (a): filter on top keys to confirm latency change; rebalance partitions.- For (b): spin up read-replicas or increase resources, or run the query during low load and compare.

7) Produce actionable conclusion with evidence- Attach metrics (CPU/time series), trace spans, before/after EXPLAIN JSON, and sample SQL proving skew. Recommend fix (index, stats vacuum, repartition, scale resources, or query rewrite).

Expected signals summary:- (a) skew: same plan, huge actual row counts for specific keys, spike in single partition ingestion/transfer.- (b) saturation: system-wide resource metrics spike, many queries slowed, increased IO wait.- (c) plan regression: changed plan operator or estimates → single-query latency spike without system-wide metric change.

HardTechnical

54 practiced

Propose an end-to-end plan to build an ML-based anomaly detection system for monitoring business metrics (e.g., revenue, purchase-rate). Cover label creation, feature engineering, model selection, evaluation metrics, productionization, and strategies to detect and mitigate model drift.

Sample Answer

Situation & goal: Build an end-to-end ML-based anomaly detection system to monitor business metrics (revenue, purchase-rate) that reliably surfaces actionable anomalies, minimizes false alerts for stakeholders, and is maintainable by BI/analytics teams.

1) Label creation- Start with weak / proxy labels: define business-rule anomalies (e.g., day-over-day revenue drop >30%, unexpected nulls, or sudden conversion drop vs. same weekday last 4 weeks). Use these for initial supervised training.- Augment with synthetic anomalies: inject spike/dip/noise patterns into historical series to increase coverage of anomaly types.- Add human-in-the-loop verification: present model candidates in dashboard and capture analyst feedback to build gold labels over time.- Keep label metadata (type, severity, confirmed) for supervised refinement.

2) Feature engineering- Time-series features: raw value, diff(1), pct_change(1,7,28), rolling mean/std (window 7/28), seasonal indicators (day-of-week, holiday, month), time since last peak.- Contextual features: product/category, region, campaign flags, user segment sizes, price changes, inventory events.- External signals: marketing spend, site traffic, payment gateway errors, deployments.- Aggregation levels: metric at multiple granularities (global, region, product) to detect local vs global anomalies.- Normalize features (z-score by rolling window) and store in a feature registry/feature store.

3) Model selection- Baseline rule/statistical: control charts (EWMA), STL decomposition + residual thresholding, Prophet/seasonal forecasting with residual anomaly detection.- Unsupervised ML: Isolation Forest, One-Class SVM, Local Outlier Factor for point anomalies.- Representation learning: autoencoders (dense or LSTM) for multivariate/time-window reconstruction error.- Supervised / semi-supervised: gradient-boosted trees or light models using engineered features when labeled data exists.- Practical approach: ensemble hybrid — forecasting model for expected value + anomaly scorer (reconstruction/error + isolation forest) to capture multiple anomaly types.

4) Evaluation metrics- Detection metrics: precision, recall, F1 on confirmed labels.- Operational metrics: false alarm rate (alerts per week), mean time to detection, lead/lag vs business impact, precision@k (top-k alerts).- Business impact: percent reduction in undetected revenue loss, % of high-severity incidents correctly flagged.- Use time-split cross-validation, preserve temporal ordering; evaluate by severity-weighted metrics.

5) Productionization- Data pipeline: scheduled ingestion from sources (ETL): raw metric -> feature computation -> feature store (versioned).- Model serving: batch scoring (hourly/daily) and streaming for near-real-time (if needed). Containerized model with API endpoints or use orchestration platform (Airflow + model job).- Alerting: integrate with Slack/Email/PagerDuty; deliver contextual dashboard link, comparison to baseline, suggested root-cause signals.- Dashboards: anomaly explorer in BI tool (Looker/Tableau) showing time-series, contributing features, and feedback buttons to label/annotate.- Observability: monitor model inputs, outputs, latency, and alert volumes; logging for investigations and audits.- Governance: runbook with triage steps, ownership, SLAs for response and model refresh.

6) Drift detection & mitigation- Data drift: monitor distributional changes on features (Kolmogorov–Smirnov, population stability index) and unseen categorical levels.- Concept drift: track degradation in precision/recall, increase in false positives, or prediction confidence shifts.- Strategies: - Shadow/champion-challenger: run candidate retrained models in parallel and compare. - Continuous labeling loop: auto-sample anomalies and request analyst confirmation to refresh training set. - Retraining policy: scheduled retrain (e.g., weekly/monthly) + triggered retrain on drift signals; keep rolling-window training to adapt seasonality. - Model versioning and A/B testing: controlled rollout and rollback capability. - Threshold calibration: re-calibrate alert thresholds based on recent baseline volatility (use percentile-based thresholds rather than fixed deltas). - Fallback: if drift exceeds threshold, fall back to simpler statistical detectors and notify ML/BizOps.

Key operational notes- Start simple: deploy forecasting + residual threshold and an isolation forest for quick wins; iterate with analyst feedback.- Prioritize explainability: return top contributing features and comparisons (expected vs actual) so BI consumers can act.- Measure business ROI (reduced undetected losses, reduced time to detect) and optimize model to maximize actionable alerts, not raw detection recall.

Unlock Full Question Bank

Get access to hundreds of Monitoring and Alerting interview questions and detailed answers.

Join thousands of developers preparing for their dream job.