Performance Monitoring & Observability Questions

Instrumenting systems so performance is continuously measured and regressions are visible. Covers performance metrics and SLIs, dashboards and time-series signals, tracing, alerting on latency and saturation, and using telemetry to guide tuning. Focuses on the ongoing measurement loop rather than one-off profiling.

MediumSystem Design

45 practiced

Design an observability strategy (metrics, structured logs, distributed tracing, and alerts) for a latency-sensitive service. Specify which metrics to collect (histograms, error rates, queue depths), sampling rates for traces, tag dimensions that matter, how to limit high-cardinality dimensions, and alerting thresholds to avoid alert fatigue.

Sample Answer

Start with requirements: latency-sensitive service with SLOs (e.g., 99.9% requests < 200ms, availability 99.95%). Design around SLOs to drive metrics and alerts.

Metrics- Latency: histogram buckets and retained percentiles (p50,p90,p95,p99,p999). Use HdrHistogram or Prometheus histograms with sufficient buckets for ms-level precision.- Error rates: counters by outcome (5xx, client errors, timeouts). Emit per-endpoint and aggregated service-level.- Throughput: requests/sec, bytes in/out.- Resource / backpressure: queue depth (gauge), consumer lag, threadpool utilization, connection pool usage.- Saturation: CPU, memory, GC pause, ephemeral sockets.

Tag dimensions (low-cardinality):- service, environment, region, deployment-version, endpoint/route, error_type- customer_tier (paid/free), host_poolLimit high-cardinality by disallowing raw user IDs, request IDs, or full URIs as tags. If needed, include hashed user_id or sample-specific attributes in logs/traces only.

Structured logs- JSON lines with keys: timestamp, level, service, trace_id, span_id, endpoint, error_type, short_message, context (small map)- Avoid PII; hash or omit user identifiers. Use log sampling and rate-limits at producer.

Tracing & sampling- Global sample rate: 1–5% for successful requests. Always-sample: - errors and 5xx responses - traces where latency > p95/p99 (tail-sampling) - traces flagged by feature flags/debug sessions- Implement adaptive/tail sampling to keep useful tail data while controlling volume.

High-cardinality controls- Enforce allowed tag list in SDKs.- Normalize tags (strip query params from paths, map similar URIs to templates).- Cap cardinality per metric; drop or shrink excessive label values.- Use aggregation pipelines to roll up low-volume labels into "other" bucket.

Alerts (avoid fatigue)- Drive alerts from SLO burn-rate / error budget policies rather than raw metrics.- Primary alerts (pager): - Burn rate alert: error-rate or latency burn > 4x over 1h (or consume >50% of error budget in short window) - Service-level p99 latency > 2x SLO for 10m AND increasing request queue depth / saturation metric - Total errors > threshold and rising (e.g., error-rate > 0.5% for 5m) with correlated service resource saturation- Secondary alerts (notification, not pager): - p95 latency approaching SLO (e.g., >75% of SLO) - queue depth steady growth (20% increase over 15m) - noncritical infra metrics (GC count, minor memory spikes)- Use multi-window checks (1m/5m/1h), require sustained violation and corroborating signals (CPU, queue depth) before paging.- Use aggregation (team-level alerts) and suppression during deploy windows; include runbook links and diagnostic queries.

Instrumentation practices- Emit trace_id with logs to link traces, logs, metrics.- Standardize metrics names and units.- Automate dashboards per SLO: latency percentiles, error budget, throughput, queue depth, top endpoints by latency/errors.- Run periodic audits to detect high-cardinality metrics and prune.

This strategy balances visibility into tail latency and errors while controlling cost and alert noise by SLO-driven alerts, sampling, and strict cardinality controls.

EasyTechnical

55 practiced

When asked to 'improve page load latency by 30% for the homepage', how would you define success criteria and key performance indicators (KPIs)? List at least 4 measurable criteria (including user-facing and technical metrics), explain target thresholds, and describe how you'd instrument them to collect reliable data.

Sample Answer

Success criteria: measurable improvement vs baseline (last 30 days). "Improve page load latency by 30%" means hitting a 30% reduction in key user-facing metrics for target percentiles (median and tail) without regressing errors or bandwidth.

Four measurable KPIs (with targets and instrumentation):

1) Largest Contentful Paint (LCP) — user-facing- Target: median LCP ≤ 0.7 × baseline and 75th/90th percentile ≤ 0.8 × baseline- Instrument: Real User Monitoring (RUM) using browser PerformancePaintTiming (Paint API) + send LCP events via PerformanceObserver to backend telemetry (sampled) tagged by URL, device, network.

2) Time to Interactive (TTI) / Total Blocking Time (TBT) — interactivity- Target: median TBT reduced by 30%; 95th percentile TTI improved proportionally- Instrument: RUM + Performance API + Lighthouse for synthetic checks. Collect TBT via PerformanceObserver and aggregate by percentile.

3) Time to First Byte (TTFB) / server latency — technical backend- Target: reduce median TTFB by 30% (or to ≤ 200ms depending on baseline)- Instrument: server-side request tracing (APM — e.g., OpenTelemetry): record TTFB, backend processing time, DB query times; correlate with RUM by trace-id headers.

4) Page weight and request count (network efficiency)- Target: total bytes transferred and number of requests reduced by ≥30% (or to thresholds like <500KB/30 req)- Instrument: Synthetic monitoring (headless browsers) and RUM Resource Timing API to capture actual bytes, caching status, gzip/brotli, and number of requests per session.

Additional guardrails:- Conversion / bounce rate (business metric): ensure no >2% negative impact; instrument via analytics events.- Error rate: front-end JS exceptions and HTTP 5xx must not increase; instrument via Sentry/APM.

Data reliability & rollout:- Collect both RUM (real users, geo/device/network breakdowns) and synthetic (controlled baselines) daily.- Tag data by version/deployment, region, device, connection type. Use percentile aggregation (p50/p75/p90/p95).- Establish dashboards and alerts (e.g., p90 LCP > target → page-performance alert).- Run A/B rollout, compare control vs. treatment with statistical significance before full release.

Unlock Full Question Bank

Get access to hundreds of Performance Monitoring & Observability interview questions and detailed answers.

Join thousands of developers preparing for their dream job.