Cloud Monitoring and Troubleshooting Questions

Monitoring, logging, and troubleshooting in cloud environments. Covers cloud provider monitoring services and patterns such as CloudWatch, Azure Monitor, and Google Cloud Monitoring; setting up metrics and alarms; centralized logging and log analysis in cloud contexts; health checks and auto recovery; diagnosing common cloud issues like performance degradation, networking and permissions problems, and resource exhaustion; and best practices for instrumenting cloud native services.

EasyBehavioral

54 practiced

Behavioral: Tell me about a time you diagnosed and resolved a cloud service health issue, for example a virtual machine failing health checks or an API returning high error rates. Explain the situation, the immediate steps you took to isolate and mitigate the problem, how you prioritized actions under pressure, how you communicated with stakeholders, and what long-term changes you implemented to prevent recurrence.

HardTechnical

52 practiced

You observe intermittent, seemingly random latency spikes in a distributed system affecting a subset of user requests. Devise an investigation plan that uses synthetic monitoring, statistical anomaly detection, detailed tracing, sampling and profiling, and logging correlation to isolate the root cause. Explain how each technique reduces MTTR and how you would validate a fix.

Sample Answer

**Investigation plan — high level**

1) Clarify scope & SLA- Identify affected endpoints, client regions, user cohorts, and time windows using current metrics (CloudWatch/Stackdriver). Set initial hypothesis (network, DB, GC, bursting).

**Synthetic monitoring (smoke + targeted)**- Deploy global synthetic probes (real browser and API checks from multiple regions) at 30–60s frequency for affected endpoints; add higher-frequency probes from suspect regions.- Purpose: reproduce spikes on-demand, create deterministic signals for correlation.- MTTR benefit: gives earliest reproducible indicator and timestamps to pivot other tools.

**Statistical anomaly detection**- Run time-series anomaly models (e.g., seasonal-ESD, Holt-Winters residuals, or AWS Lookout) on latency, error-rate, and percentiles per dimension (region, instance type, API key).- Purpose: surface which dimensions spike and whether spikes correlate with resource metrics (CPU, network).- MTTR: reduces blast radius by automatically narrowing to dimensions to investigate.

**Distributed tracing & detailed traces**- Increase trace sampling for affected services (temporarily up to 100% for suspect paths) and instrument critical spans (load balancer, app, DB, cache, downstream services). Use X-Ray/Jaeger/Datadog.- Purpose: see per-request span durations and identify dominant latency spans.- MTTR: directly points to component causing tail latency.

**Sampling, profiling, and continuous profiling**- Apply CPU/memory flamegraphs and profiles (pprof, eBPF, Cloud Profiler) for instances serving slow traces; use correlation id from traces to sample those exact requests.- Purpose: detect GC pauses, lock contention, syscalls, or syscall backpressure (e.g., DNS, TLS).- MTTR: isolates code/runtime-level causes versus infra.

**Logging correlation**- Ensure structured logs include trace-id, request-id, user-id, instance-id, and timestamps. Correlate slow synthetic timestamps to logs and traces; search for retries, timeouts, throttling, or error messages.- MTTR: fills gaps when traces are absent and helps reconstruct causality.

**Actionable workflow**- When a spike occurs: use synthetic probe timestamp → anomaly tool flags dimension → pull traces for that window → filter traces to slow spans → extract trace-id → pull logs and profiles from instances handling those trace-ids → form root-cause hypothesis → create mitigation (config change, scale up, fix query, circuit-breaker).- Apply temporary mitigations (e.g., auto-scale, deploy rate-limiter) to reduce customer impact while fixing.

**Validation**- Run A/B: deploy fix to canary pool; run synthetic checks and compare percentiles and tail behavior; verify anomaly detectors show baseline; re-run targeted traces and profiles; monitor SLOs for several full recurrence windows.- Post-mortem: document RCA, metrics before/after, and add permanent observability (higher sampling for critical paths, synthetic coverage, runbooks, and alerts).

This approach ties deterministic probes, statistical detection, tracing, profiling, and log correlation into a fast feedback loop that narrows scope quickly and validates fixes, minimizing MTTR.

EasyTechnical

56 practiced

List and explain the key metrics and logs you would monitor for a managed relational database (for example AWS RDS or Google Cloud SQL) to detect performance or correctness problems. Include at least eight metrics or log sources, typical thresholds or baseline behaviors, and how you would correlate them with application issues.

Sample Answer

**Approach overview**As a Cloud Engineer I monitor both database internals and service-level logs to detect performance or correctness issues, then correlate with application metrics (latency, error rates, traffic) to triage root cause.

**Key metrics & logs (≥8)**- CPU Utilization (RDS/Cloud SQL): baseline <60% under normal load; sustained >80% => slow queries or insufficient instance size. Correlate with app request latency.- Memory / Freeable Memory: baseline stable with healthy cache; drops => swapping, increased disk I/O. Correlate with increased DB read latency.- DB Connections: threshold near max_connections (e.g., >70%): connection exhaustion causes errors/timeouts. Correlate with app connection errors/login failures.- Read/Write IOPS and Throughput: sudden spikes indicate heavy queries or batch jobs. Correlate with app p99 latency spikes.- Disk Queue Depth / Disk Latency (Read/Write Latency): high queue depth or >10–20 ms reads/writes => storage bottleneck. Correlate with timeouts and slow transactions.- Replica Lag (for read replicas): >100–500 ms or growing => stale reads; correlate with inconsistent reads from app.- Transaction/sec, Locks, Deadlocks: high lock waits or deadlock rates => contention from long transactions. Correlate with specific API failures and slow transactions.- Cache Hit Ratio / Buffer Pool Usage (e.g., InnoDB buffer pool): low hit ratio => poor indexing or insufficient memory. Correlate with increased IOPS and CPU.- Error Logs / Slow Query Logs / General Logs: slow query entries (>threshold like >500 ms) point to problematic SQL. Correlate timestamps with app traces.- Audit and Connection Logs: detect authentication failures, suspicious queries, or schema changes.

**Typical baselines & actions**- Establish rolling baselines per environment; alert on deviations (e.g., 2–3x normal, or absolute thresholds noted above).- Use APM/tracing to link DB calls to service endpoints; on alerts, inspect slow query logs, EXPLAIN plans, and recent deploys.

**Correlation workflow**1. Detect metric anomaly (e.g., CPU + IOPS spike).2. Check slow query and error logs at same timestamps.3. Map to application traces/endpoints and recent releases or traffic changes.4. Remediate: index/query optimization, scale instance/IOPS, add replicas, connection pooling.

This set covers performance, correctness, and operational context for managed RDBMS monitoring.

HardTechnical

64 practiced

You find that recurring midnight 5xx spikes correlate with a batch job. Describe how you would prove causation using logs, metrics, and traces, propose short-term mitigations to protect production traffic, and design long-term architectural changes to prevent resource contention between batch and real-time workloads.

Sample Answer

**Clarify goal & hypothesis**I would start with the hypothesis: the midnight batch job is causing resource contention that produces 5xx errors for real-time traffic. My plan: prove causation, mitigate immediately, then redesign to avoid recurrence.

**Prove causation (logs, metrics, traces)**- Correlate timestamps: align batch job start/finish with 5xx spike using centralized logs (CloudWatch/Stackdriver/ELK).- Check metrics: CPU, memory, disk I/O, network, database connection pools, thread counts on affected services around spike via CloudWatch/GCP Monitoring; look for saturation.- Traces: use distributed tracing (X-Ray/Jaeger) to inspect increased latency/error rates in downstream services during the job window; identify slow spans (DB queries, throttled APIs).- Synthetic tests: replay a scaled-down batch run in staging while running smoke tests to reproduce 5xx behavior.- Root-cause logs: capture application logs showing timeouts (DB connection pool exhaustion, OOM, rate-limit exceeded).

**Short-term mitigations**- Shift batch to a lower priority (nice/cgroups) or run on isolated, cheaper nodes; use autoscaling safeguards (scale real-time pool higher before batch starts).- Throttle batch concurrency and rate-limit DB/API calls.- Add circuit-breakers and backpressure to real-time services.- If urgent, reschedule batch to off-peak or pause it until mitigations deploy.

**Long-term architecture changes**- Isolate workloads: move batch to dedicated instance groups, autoscaling groups, or spot fleets with separate node pools; use separate DB replicas for reporting.- Implement resource quotas/limits with Kubernetes (cpu/memory requests and limits, pod priority/preemption).- Adopt queue-driven, rate-limited batch processing (e.g., AWS Batch, Dataflow, or K8s Job + rate limiter).- Add capacity planning and SLO-driven scheduling: only run heavy batches when headroom exists; integrate scheduler with monitoring to gate runs.- Improve observability: add end-to-end tracing, service-level dashboards, and alerting on resource contention.

These steps prove causation, protect production immediately, and architecturally separate batch from real-time to prevent future contention.

HardTechnical

51 practiced

Scenario (P0): the primary database cluster in us-east-1 has become read-only and the application is degraded. Provide an incident response playbook: immediate steps to restore write availability (failover or recovery), non-destructive diagnostics to capture for postmortem, communication steps to stakeholders, short-term mitigations to reduce customer impact, and how to proceed with a safe root-cause analysis after service restoration.

Sample Answer

**Immediate steps — restore write availability (first 0–15 mins)**- Declare P0, page on-call DB and infra leads; assign incident commander (IC).- Confirm scope: which cluster (us-east-1), read-only for all writers or subset.- Check provider managed failover: if Aurora/RDS, attempt controlled failover (RDS Console / AWS CLI: `rds reboot-db-instance --force-failover` for Multi-AZ or promote a read replica). If using self-managed DB, promote a healthy replica to primary (ensure WAL/replication state good).- If failover not possible or slow, temporarily re-point app write traffic to a healthy standby/replica in another AZ/region (update connection string via feature-flag / service discovery) or enable a secondary write endpoint.- Avoid destructive fixes (no restart with unsafe flags) until diagnostics captured.

**Non‑destructive diagnostics to capture (immediately)**- Take DB snapshots / backups and export them (preserve state).- Collect cloud metrics: CloudWatch/Stackdriver for CPU, IOPS, latency, free storage, replica lag.- Export DB logs (error, slow query, binary logs), RDS Events.- Run safe DB queries: `SHOW GLOBAL VARIABLES LIKE 'read_only';`, `SHOW PROCESSLIST;`, replication status (`SHOW SLAVE STATUS\G`), disk usage, open file descriptors.- Capture app logs, load balancer errors, recent deployment events, and timestamps.- Note timestamps of all commands executed during incident.

**Communication**- Immediately update status page and Slack with: P0, impacted scope, IC, mitigation in progress, ETA for next update (every 15 mins).- Notify stakeholders (SRE manager, product, customer support, legal if needed) with clear customer impact and mitigation steps.- After write restored, send “post-restore” message with actions taken and next steps (RCA timeline).

**Short‑term mitigations to reduce customer impact**- Enable write queuing: switch app to queue writes (SQS/Kafka) with retry/DRainback to avoid data loss.- Turn on degraded mode: disable non-essential write-heavy features, return 429 for new writes if necessary.- Increase replicas to offload reads; scale read tier horizontally.- Rate-limit noisy clients, implement backoff, and prioritize critical transactions.

**Safe RCA after service restoration**- Preserve all collected artifacts (snapshots, logs, metrics) in immutable storage.- Do not run destructive debugging on production; use copies/restore to staging for experiments.- Build incident timeline with correlated logs/metrics, deployment/change history, configuration changes, and human actions.- Hypothesis-driven RCA: test each hypothesis in a sandbox, document test results.- Identify root cause, contributing factors, and corrective actions (e.g., failover automation, alert thresholds, runbook updates).- Schedule a blameless postmortem with stakeholders within 72 hours, publish findings, assign owners and deadlines for remediation, and add tests or automation to prevent recurrence.

Unlock Full Question Bank

Get access to hundreds of Cloud Monitoring and Troubleshooting interview questions and detailed answers.

Join thousands of developers preparing for their dream job.