Cloud Monitoring and Troubleshooting Questions

Monitoring, logging, and troubleshooting in cloud environments. Covers cloud provider monitoring services and patterns such as CloudWatch, Azure Monitor, and Google Cloud Monitoring; setting up metrics and alarms; centralized logging and log analysis in cloud contexts; health checks and auto recovery; diagnosing common cloud issues like performance degradation, networking and permissions problems, and resource exhaustion; and best practices for instrumenting cloud native services.

HardSystem Design

87 practiced

Design a scalable telemetry platform for a global SaaS vendor that must ingest 500M metrics and 10B log lines per day from multiple cloud regions and on-prem customers. Requirements: query latency under 2s for recent 24 hours, retention tiers (hot 30d, warm 365d), multi-tenant isolation, encryption at rest/in transit, and exports to external SIEMs. Describe architecture components, data flows, storage choices, scalability and operational concerns.

Sample Answer

Requirements & constraints (clarify): ingest 500M metrics/day (~6k/sec) and 10B log lines/day (~115k/sec), <2s query latency for recent 24h, Hot 30d + Warm 365d tiers, multi-tenant isolation, encryption in transit/at rest, SIEM exports.

High-level architecture:- Ingestion edge: lightweight agents (cloud & on‑prem) push via HTTPS/TLS to regional Ingest Gateways (API Gateway + load balancer). Gateways perform auth (mTLS / API keys / OAuth), tenant validation, rate‑limit, schema validation, and lightweight enrichment (metadata, tenant id, sampling hints).- Streaming layer: regional Kafka / Pulsar topics partitioned by tenant hash to scale. Use separate topics for metrics vs logs; compacted topics for metrics.- Processing layer: - Real-time pipeline (Flink/Beam) for metric rollups, indexing, anomaly detection, immediate hot-store writes. - Log pipeline for parsing, enrichment, redact PII, and writing to hot store and object storage.- Hot storage (0–30d): time-series DB for metrics (Cortex/Thanos or managed TSDB like M3DB) and an inverted-index log store (Elasticsearch/OpenSearch or OLAP like ClickHouse with log schema). Use per-tenant logical isolation (labels, index prefixes) and RBAC.- Warm storage (30–365d): columnar object storage on S3-compatible with partitioning by tenant/day. Metrics: pre-aggregated time-buckets stored in Parquet/ORC. Logs: compressed Parquet with indices for selective retrieval. Use query engine (Presto/Trino) + indices for warm queries.- Indexing & Query layer: API/Query Frontend routes queries to hot store first (meet <2s). If not present, query engine over warm layer with result caching. Use distributed query coordinators and caching (Redis/Velox) for frequent dashboards.- Exports: Push-based connectors or SIEM-forwarders reading from Kafka or export API; support syslog, CEF, JSON over HTTPS. Provide per-tenant export configs and throttling.- Security: TLS everywhere, envelope encryption for objects, KMS for keys, tenant key separation optional (customer-managed keys), E2E encryption for on‑prem when required. Audit logs stored in append-only store.- Multi-tenant isolation: logical isolation + quotas; for high-security customers offer dedicated tenants (single-tenant topics/namespaces, dedicated hot nodes).- Scalability: partition by tenant+time; autoscale ingestion gateways, Kafka brokers, stream processors, TSDB clusters. Use compaction/retention policies to bound storage. Use tiered downsampling for older data to reduce cost.- Operational concerns: - Backpressure & burst handling: client-side buffering, sampling, tiered ingestion queues; dead-letter queues for malformed data. - SLOs/observability: metrics for ingestion rate, lag, query latency, storage growth. Distributed tracing for pipelines. - Cost control: downsampling, TTL, archival lifecycle policies. - Disaster recovery: cross-region replication of Kafka and object storage; warm failover for query frontends. - Compliance: encryption, tenant data residency (regional ingestion & storage), retention deletion workflows.

Trade-offs:- Elasticsearch provides fast log queries but costs more at scale; ClickHouse + Parquet on S3 lowers cost but increases query complexity for low-latency queries—hence hybrid hot/warm approach.- Logical multi-tenancy maximizes utilization; dedicated deployments increase isolation at higher cost.

This design meets <2s recent-query SLA by directing queries to optimized hot stores and caches, scales horizontally for multi‑region ingestion, provides tiered retention and secure multi‑tenant isolation, and supports robust exports to external SIEMs.

MediumSystem Design

48 practiced

You are designing monitoring for Kubernetes (EKS) workloads that include stateless front-ends and stateful databases. Propose a monitoring stack (metrics, logging, tracing), where to run collectors (DaemonSet vs sidecar), and strategies to minimize performance overhead on critical nodes while preserving signal fidelity.

Sample Answer

Requirements:- Low-latency alerts for front-end errors, high-fidelity DB metrics for SLOs, centralized logs and traces, minimalCPU/memory impact on critical DB nodes, multi-tenant security on EKS.

Monitoring stack (metrics, logging, tracing):- Metrics: Prometheus for cluster + app metrics (kube-state-metrics, node-exporter, cAdvisor) + Prometheus exporters for DB (postgres_exporter, mysqld_exporter) and app-level business metrics exposed via /metrics.- Logging: Fluent Bit as lightweight log forwarder to a central log store (ELK/Opensearch, Datadog, or Sumo) with structured JSON parsing.- Tracing: OpenTelemetry (OTel) for distributed traces, export to Jaeger/Tempo or SaaS tracing backend.- Storage/Long-term: Thanos or Cortex for long-term Prometheus metrics.

Where to run collectors:- Node/system metrics: DaemonSet (Prometheus node-exporter, cAdvisor, Fluent Bit) to cover all nodes without per-pod overhead.- App-level metrics/tracing: Prefer sidecar or in-process SDKs for high-fidelity app metrics and tracing (OTel SDK in-process yields best context propagation). Use a lightweight OTel collector as sidecar only where trace sampling or enrichment is needed; otherwise use a centralized OTel collector as Deployment receiving data via OTLP.- DB exporters: Run as a Deployment with a single replica per DB pod where possible (for statefulsets use sidecar exporter only if local socket access required), or run as a Pod on the same node using a headless service to ensure locality.

Minimize performance overhead while preserving fidelity:- Sampling & adaptive rates: Use trace sampling (e.g., 1% by default, adaptive on error rates or PID spikes) and dynamic metric scrape intervals (shorter for front-end critical metrics, longer for low-churn DB metrics).- Pull vs push: Use Prometheus pull for app metrics; for very high-cardinality workloads use pushgateway or metrics aggregation to avoid scrape storms.- Resource limits & QoS: Pin collectors to lower QoS classes; reserve CPU/Memory on critical DB nodes and taint nodes for DBs. Tolerations only on essential collectors. Use node taints/labels so heavy collectors avoid critical nodes.- Edge filtering & parsing: Do parsing/filtering at Fluent Bit to drop verbose debug logs before network egress. Apply structured logging and log levels to reduce noise.- Local buffering and batching: Configure Fluent Bit and OTel Collector with bounded buffers and exponential backoff to avoid impacting apps during network issues.- Rate limits & cardinality control: Enforce label cardinality limits in Prometheus/remote write pipeline; normalize high-cardinality labels at collector/ingest.- Isolation: Run critical DBs on dedicated node pools (EKS node groups) and avoid running non-essential collectors on them. Use DaemonSets with nodeSelector so Fluent Bit/node-exporter run on non-critical pools if needed.

Trade-offs:- Sidecar/SDK gives best trace correlation but increases pod resources and operational complexity. DaemonSet reduces per-pod cost but may lose per-process context.- Sampling reduces cost but can miss rare events; use adaptive sampling for spikes.

Example policies:- Front-end: scrape 15s, traces sampled 10% baseline, 100% on error.- DB: scrape 60s, detailed internal metrics only during incidents (toggle via feature flags).- Logs: Info+ persisted, Debug forwarded only when a tracing alert triggers.

This design balances fidelity and cost by combining DaemonSets for low-overhead node-level collection, in-process/sidecar for critical context where needed, and operational controls (sampling, rate limits, node isolation) to protect critical nodes.

MediumTechnical

47 practiced

Write a PromQL (or CloudWatch metric math) expression to detect a sudden 3x increase in a service's error rate compared to the trailing 1-hour moving average, ensuring at least 100 requests in the comparison window. Explain assumptions and how you'd convert this into an alert rule.

Sample Answer

Approach: compute current error rate over a short window (e.g., 1m), compare to trailing 1‑hour moving average error rate, require at least 100 requests in the comparison window to avoid noisy signals, and fire when current >= 3x moving average.

PromQL (assumes metrics: http_requests_total{job="svc"} and http_requests_errors_total{job="svc"} as counters):

promql

# current error rate over last 1 minute (errors per second)
current_err_rate = rate(http_requests_errors_total{job="svc"}[1m])

# 1-hour trailing moving average error rate: average of per-minute rates across last 1h
hour_avg_err_rate = avg_over_time(rate(http_requests_errors_total{job="svc"}[1m])[1h:1m])

# total requests in last 1h (used as minimum-volume guard)
reqs_last_hour = increase(http_requests_total{job="svc"}[1h])

# final expression: alert when current >= 3 * hour_avg AND requests >= 100
current_err_rate > 3 * hour_avg_err_rate and on() (reqs_last_hour >= 100)

Notes/assumptions:- Using rate(...[1m]) reduces counter reset impact and gives per-second rate; avg_over_time(...[1h:1m]) computes moving average of those 1m rates across the hour.- Minimum 100 requests in last hour prevents alerts on low-traffic services.- Choose 1m/1h windows to balance sensitivity vs. noise; adjust to your SLOs.

Converting to an alert rule:- Alert name: HighErrorRateSpike- Expression: the boolean PromQL above (Prometheus requires combining into a single expression via vector math; implement reqs_last_hour >= 100 as a label filter or multiply by bool).- For Prometheus alertmanager: set for: 5m to avoid flapping; severity label (e.g., critical); add runbook with troubleshooting steps.- Add silencing/auto-resolve and integrate with paging channels.

CloudWatch Metric Math alternative (outline):- Use m1 = SUM(Errors) over 1m (rate via metric math), m2 = AVG(m1) over 1h, m3 = SUM(Requests) over 1h; alarm when m1 >= 3*m2 AND m3 >= 100. Configure evaluation periods and datapoints to alarm similarly.

MediumTechnical

82 practiced

During incidents, customers report intermittent connectivity failures between application instances and a managed database in AWS. Outline telemetry and monitoring checks you'd add to quickly identify network-related issues (packet loss, latency, DNS resolution, routing), and name specific metrics, logs or tools (VPC Flow Logs, CloudWatch metrics, tcpdump, etc.) you would use.

Sample Answer

Situation: Customers report intermittent connectivity failures between app instances (EC2/ECS/Lambda) and a managed AWS DB (RDS/Aurora). Goal: rapidly detect whether failures are network-related (packet loss, latency, DNS, routing) and locate the fault domain.

Telemetry & monitoring checks:

- VPC / ENI level - VPC Flow Logs (filter ACCEPT/REJECT, bytes, pkt counts, src/dst IPs, ports, TCP flags). Look for spikes in REJECT or DROP and unusual SYN/ACK patterns. - CloudWatch Metrics: NetworkIn/NetworkOut and ENI error counters. Correlate per-subnet/instance.

- Host / instance - OS metrics: interface rx/tx errors, drops (collect via CloudWatch Agent or Prometheus node_exporter). - tcpdump / packet capture on app host and a bastion to capture packet loss, retransmits, TCP resets:

bash

sudo tcpdump -i eth0 host <db-ip> and port 3306 -w capture.pcap

- Use tcptraceroute / mtr for path and per-hop loss.

- DNS - Route 53 Resolver Query Logs (for private zones) or enable VPC DNS query logging -> track resolution failures, elevated latencies, NXDOMAIN. - Instrument application to emit DNS resolution latency and errors.

- DB endpoint & service - RDS Enhanced Monitoring & CloudWatch: DatabaseConnections, CPU, FreeableMemory, NetworkReceiveThroughput/NetworkTransmitThroughput. Check RDS failovers or maintenance events. - RDS logs for connection resets.

- Routing / NACL / Security Groups - CloudWatch Events / Config to detect recent SG/NACL changes. - VPC Flow Logs + Route Table audits to catch blackholes or overlapping routes.

- Latency & packet loss synthetic checks - Create synthetic pings/tcp checks from multiple AZs (SSM Run Command or Lambda) to DB endpoint; emit metrics: tcp_connect_ms, p50/p95 latency, packet_loss_rate. - Use CloudWatch Synthetics canary for TCP/HTTP checks where appropriate.

- Correlation & alerting - Centralize logs in CloudWatch Logs / Elasticsearch. Create dashboards correlating: flow log rejects, host interface errors, DNS errors, tcp_connect latency, RDS network throughput. - Alerts: sustained packet_loss_rate > 1% or tcp_connect_ms > threshold or DNS error rate spike.

Triage steps when alarm fires:1. Confirm DNS resolution and TTLs from app host.2. Run mtr/tcptraceroute and tcpdump to confirm packet loss or retransmits.3. Check VPC Flow Logs for drops at AZ/subnet level and recent SG/NACL/route changes.4. Check RDS metrics for network/saturation/failover.5. If transient and AZ-specific, open a support case with AWS with packet capture and flow logs.

This blend of VPC Flow Logs, CloudWatch (RDS + ENI), host packet captures (tcpdump), DNS query logs, synthetic checks, and correlation dashboards gives fast identification of packet loss, latency, DNS resolution, and routing issues.

MediumTechnical

63 practiced

Define a practical alerting strategy for a SaaS product to reduce alert fatigue while ensuring SLO compliance. Include alert categories (page, ticket, info), thresholds tied to SLOs, aggregation windows, and an escalation policy. Outline how to implement these concepts using common cloud provider alarms or alerting platforms.

Sample Answer

Situation: We need an alerting strategy that prevents noise but guarantees SLO compliance for a SaaS product.

Approach (high-level):- Classify alerts into Page (immediate action), Ticket (investigate during business hours), Info (dashboard/weekly review).- Tie thresholds to SLO error budget and customer-impacting metrics (latency, error-rate, availability).

Alert categories & examples:- Page: “Service error rate > SLO threshold and error budget burn rate high.” E.g., 5xx rate > 0.5% sustained for 3m when SLO = 99.9%.- Ticket: “Elevated latency trending toward SLO breach.” E.g., p95 latency > 80% of SLO for 30m.- Info: “Daily aggregated error spikes under threshold” or configuration/deployment completions.

Thresholds & aggregation:- Use two-tier thresholds: warning (ticket) at 50–75% of remaining error budget over 1h window; critical (page) when burn rate > 4x over 5–15m.- Aggregation windows: short (3–5m) for pages, medium (30–60m) for tickets, daily for info. Use rolling windows to avoid flapping.

Escalation policy:- Page: Notify primary on-call via phone/SMS + Slack; if no ack in 5m escalate to secondary; if unresolved in 30m escalate to engineering lead and runbook owner; auto-create incident in tool.- Ticket: Create issue in tracking system, assign to on-call or owner, SLA: acknowledge 4h, resolve per priority.- Info: Post to dashboard and weekly review; no paging.

Implementation (cloud/alerting platforms):- AWS CloudWatch + SNS + PagerDuty: - Create CloudWatch metric math for burn-rate and rolling-window evaluation. - Alarms: Critical -> SNS -> PagerDuty escalation policy; Warning -> SNS -> Slack + Jira ticket via Lambda. - Use Tags/Alarm actions to attach runbook links.- Azure Monitor + Action Groups + OpsGenie: - Azure Metric Alerts with dynamic thresholds tied to historical baselines; use Scheduled Query Rules for log-based error rates. - Action Groups: Critical -> OpsGenie high-priority; Warning -> Email/Teams + create Azure DevOps work item; Info -> Azure Monitor dashboard.- Use alert deduplication and grouping: set dimensions (service, region, cluster) and set "group by" to collapse noisy bursts.- Instrument runbooks, automated remediation for known fixes (restart, circuit-breaker), and integrate incident postmortems to adjust thresholds.

Why this works:- Thresholds map to SLOs and error budget so alerts focus on customer impact.- Aggregation and burn-rate logic reduce noisy transient alerts.- Clear escalation minimizes on-call churn while ensuring SLO violations are handled promptly.

Unlock Full Question Bank

Get access to hundreds of Cloud Monitoring and Troubleshooting interview questions and detailed answers.

Join thousands of developers preparing for their dream job.