System Monitoring and Performance Tuning Questions

Operational monitoring and continuous tuning of system and infrastructure resources to maintain performance and reliability. Topics include key system health and performance metrics such as central processing unit usage memory utilization disk input output and latency network bandwidth process counts system load latency and throughput and queries per second, establishing baselines and normal ranges, anomaly detection and root cause triage, instrumentation and metric collection for system health, reading monitoring dashboards and recognizing common failure patterns, interpreting system logs and using diagnostic commands and tools, setting alert thresholds and prioritization and escalation pathways, capacity planning and remediation steps, resource tuning to remove bottlenecks, and knowing when to escalate to deeper engineering investigation. Candidates should be able to connect observed symptoms to likely causes describe basic troubleshooting workflows and propose mitigation and prevention measures.

HardTechnical

65 practiced

Leadership/case study: You have a backlog of long-running performance engineering projects and a queue of production incidents demanding immediate attention. Propose a prioritization framework that balances short-term reliability fixes, long-term performance investments, and feature delivery. Include metrics you would track to demonstrate ROI of performance work and how you would get leadership buy-in.

MediumTechnical

55 practiced

For a Kubernetes cluster hosting web services, list the instrumentation and metrics you would collect to detect CPU throttling events, OOMKills, node pressure (CPU/memory/disk), and slow application startup times. Specify which metrics come from kubelet, cAdvisor, kube-state-metrics, or the application itself and how to alert on them.

Sample Answer

**Approach (brief)** I’d collect node/cgroup/container, kubelet and app-level metrics, and translate them into Prometheus alerts and dashboards to detect CPU throttling, OOMKills, node pressure, and slow startup.

**Metrics & Sources**- CPU throttling - metric: container_cpu_cfs_throttled_seconds_total, container_cpu_cfs_throttled_periods_total - source: cAdvisor / kubelet metrics exposed on kubelet/cadvisor endpoints - alert: high increase in throttled_seconds or throttled_periods per container over 5m- OOMKills - metric: kube_pod_container_status_terminated_reason{reason="OOMKilled"} (kube-state-metrics) and node OOM events from kubelet (`container_last_seen`/eviction counters) - alert: any new OOMKilled termination or >0 OOMs in 1m for a workload- Node pressure (CPU/memory/disk) - metrics: node_exporter / kubelet: node_memory_MemAvailable_bytes, node_filesystem_avail_bytes, node_load1, node_cpu_seconds_total (or containerized equivalents); kube_node_status_condition{condition="MemoryPressure"/"DiskPressure"/"PIDPressure"} (kubelet via node exporter/kube-state-metrics) - alert: MemoryPressure/DiskPressure true; available disk < 10% or load >> cores- Slow application startup - metrics: application readiness probe latency or custom startup_duration_seconds histogram (application) and kube_pod_start_time_seconds (kube-state-metrics) - alert: pod startup_duration > expected SLA (e.g., >60s) or readiness not true after X seconds

**Example Prometheus alerts**

yaml

- alert: ContainerCPUThrottlingHigh
  expr: rate(container_cpu_cfs_throttled_periods_total[5m]) > 0.1
  for: 5m
  labels: {severity: warning}
  annotations: {summary: "High CPU throttling for {{ $labels.container }}"}

- alert: PodOOMKilled
  expr: increase(kube_pod_container_status_terminated_reason{reason="OOMKilled"}[5m]) > 0
  for: 1m
  labels: {severity: critical}

**How to act / runbooks**- Throttling: increase cpu limits/requests, move to node with spare capacity, tune QoS or use burstable QoS- OOMKills: increase memory requests/limits, fix memory leaks, adjust JVM flags, add liveness probes- Node pressure: cordon and drain unhealthy nodes, add capacity, clean up disk usage- Slow startup: optimize init logic, lazy-load, warm caches, adjust readinessProbe/startupProbe

Collect logs (kubelet logs, container stderr) and correlate alerts with Grafana dashboards showing time series and top offenders.

MediumTechnical

45 practiced

Which kernel or sysctl parameters would you tune to improve server performance under very high numbers of short-lived TCP connections (thousands per second)? Explain at least five parameters, their effects, and potential risks of changing them on a live system.

Sample Answer

**Approach (short)** I’ll list key kernel/sysctl knobs I would tune for thousands/sec short-lived TCP connections, explain what each does, why it helps, and the risks on a live system.

1) net.ipv4.tcp_tw_reuse / net.ipv4.tcp_tw_recycle - Effect: allow reuse/recycling of TIME_WAIT sockets so ephemeral ports are freed faster. - Why: reduces TIME_WAIT buildup when many short connections from same client IPs. - Risk: tcp_tw_recycle breaks NAT/clients with changing timestamps; can cause hard-to-diagnose connectivity issues. tcp_tw_reuse is safer; change carefully and test.

2) net.ipv4.tcp_fin_timeout - Effect: how long sockets in FIN-WAIT/closing states linger. - Why: lower value frees kernel memory and ephemeral ports faster for servers dominating short connections. - Risk: too low can drop connections during slow close or retransmits, causing application-level errors.

3) net.ipv4.ip_local_port_range - Effect: range of ephemeral ports available for outbound connections. - Why: expanding the range (e.g., 1024–65535) increases available ports to support many concurrent short-lived client connections. - Risk: larger range slightly increases port-space for scans; ensure no port collisions with services.

4) net.core.somaxconn and net.ipv4.tcp_max_syn_backlog - Effect: backlog queue sizes for listen()/SYN queue respectively. - Why: raise to handle bursty accept() and SYN arrival rates without dropping new connection requests. - Risk: increasing uses more kernel memory and masks slow-accept bugs in app; excessive values won’t help if app can’t accept fast enough.

5) net.core.netdev_max_backlog and net.core.rmem_max/wmem_max - Effect: packet receive queue length and socket buffer maxima. - Why: larger backlog and buffers prevent packet drops under receive bursts and allow kernel to queue more data while application processes connections. - Risk: higher buffers consume RAM; if set too high on many servers, can exhaust memory and cause OOM.

6) net.ipv4.tcp_max_tw_buckets / net.ipv4.tcp_max_syn_backlog (extra) - Effect: caps for TIME_WAIT and SYN backlog; useful to raise if system hits limits. - Risk: raising limits masks root cause and increases memory usage.

Operational notes: test changes on staging, incrementally apply, monitor netstat/ss, /proc/net/sockstat, memory and accept rates. Use connection pooling or keepalives at application layer where feasible to reduce connection churn.

HardTechnical

85 practiced

Network troubleshooting: you see 1% packet loss and excessive TCP retransmits between datacenters. Describe a set of tests and captures you would perform (including tcpdump, iperf, mtr, and host-side counters), exactly what signatures you would look for in packet captures, and how you would determine if the problem is congestion, faulty hardware, or a misconfigured load balancer.

Sample Answer

Situation: I observe ~1% packet loss and high TCP retransmits between datacenters. I’d run a targeted, methodical set of tests to isolate where packets are dropped and why.

Initial checks (host-side)- Check interface & error counters: - ethtool -S eth0; ip -s link show eth0 — look for rx_errors, tx_errors, dropped, fifo, overruns.- Check CPU/interrupt saturation: top/iostat, /proc/interrupts, softirq counters.

Active throughput tests- Run iperf3 (TCP and UDP) between hosts to reproduce and measure:

bash

# server
iperf3 -s
# client
iperf3 -c server -P 8 -t 60
# UDP
iperf3 -c server -u -b 1G -t 30

- Look for throughput drops, jitter, and packet loss on UDP test.

Path and latency diagnostics- Run mtr (long run) to map loss per hop:

bash

mtr --report --report-cycles 100 <remote>

- Identify hop where loss spikes—if mid-path, network device likely; if only at destination, host/LB issue.

Packet captures- Capture on both endpoints and on suspected transit devices with tcpdump:

bash

tcpdump -i eth0 -s 0 -w hostA.pcap host hostB
tcpdump -i eth0 -s 0 -w lb.pcap host hostB

Signatures to inspect in pcap (use Wireshark/tshark):- Retransmit patterns: repeated identical seq ranges, fast retransmit (duplicate ACKs), or RTO retransmits (long gaps before retransmit).- Duplicate ACKs frequency: many dup ACKs before retransmit ⇒ likely loss on path (congestion or single-link loss).- SACK blocks: presence and SACK negotiation shows selective loss handling.- Out-of-order packets: many out-of-order frames before dup ACKs ⇒ switching/routing issues or asymmetric path.- ICMP Destination Unreachable or TTL exceeded messages.- TCP window sizes and zero-window events (receiver-side congestion).- Check for TCP reset bursts (misconfigured LB/firewall).

Correlate timing & counters- Compare host NIC dropped counters timestamped with pcap times. If host increments drops but pcap shows packets sent to NIC, likely NIC driver/hardware.- If pcap at sender shows packet leaves, but no arrival at receiver capture and intermediate hop shows loss in mtr—path device problem.- If mtr shows uniform small loss across many hops ⇒ possible upstream congestion.- If loss only when LB is in-path and captures show SYNs/ACKs dropped or resets near LB ⇒ misconfigured load balancer (e.g., health checks, ephemeral port exhaustion, flow table limits).

Determining cause — checklist- Congestion: UDP iperf shows loss/jitter; tail drops in router queues; ECN marks present (if enabled); loss correlated with high utilization on interfaces (ifInOctets).- Faulty hardware/driver: interface error counters (rx_errors, crc) increase; pcap shows corrupted frames or link-level errors; replacing NIC or moving endpoint resolves.- Misconfigured load balancer: losses only when traffic goes through LB; captures show asymmetric NAT, TCP termination issues, or connection tracking timeouts; LB metrics (conn table full, high CPU) match.

Follow-ups & mitigations- If congestion: enable QoS, increase buffers, tune TCP (BBR/CUBIC tuning, cong avoidance), or add capacity.- If hardware: replace cable/NIC, update firmware/driver, verify SFP transceivers.- If LB misconfig: adjust timeouts, scale backend pool, review SNAT/conn tracking, enable keepalives/health checks.

I’d document findings, include pcaps and counters, and recommend targeted remediation based on which signature and counters point to.

MediumTechnical

62 practiced

Explain the differences between SLA, SLO, and error budget. Given a target SLO of 99.9% latency under 250ms over a 30-day window, describe how you would implement monitoring and alerting to ensure the service does not exceed its error budget.

Sample Answer

**Explain differences (concise)**

- **SLA (Service Level Agreement):** Contractual commitment to customers — may include penalties. - **SLO (Service Level Objective):** Internal measurable target (e.g., 99.9% requests < 250ms over 30 days). - **Error budget:** Allowed amount of failures (1 - SLO) used for releases/risk decisions.

**Compute error budget**

text

Error budget = 1.0 - SLO = 1.0 - 0.999 = 0.001 = 0.1% of requests over 30 days

Plain English: up to 0.1% of requests can exceed 250ms in 30 days.

**Monitoring & alerting plan (implementation)**

- Instrumentation: - Collect per-request latency in app/service (prometheus histograms or distributed tracing). - Export metrics: total_requests, requests_over_250ms by service/region.- Aggregation: - Calculate rolling 30-day ratio: sum(requests_over_250ms, 30d) / sum(total_requests, 30d). - Store high-resolution short-term (1m) and aggregated long-term (1h/24h) metrics.- Alerts: - Immediate (P1): short-term spike — if requests_over_250ms / total_requests > 0.5% in 5m => page on-call. - Warning (P2): burn-rate alert — if error budget burn-rate > 2x sustained for 1h. Burn rate = (observed_bad_fraction / error_budget) / time_fraction. - Critical (policy): if remaining error budget < 25% => pause risky deployments and convene incident review.- Dashboards & runbooks: - Dashboard: 30-day SLO %, daily burn rate, top endpoints by latency, regions. - Runbook: steps to mitigate (scale up, rollback, throttle, increase capacity), who to notify.- Validation & testing: - Simulate load tests to verify metric accuracy and that alerts fire. - Weekly review of SLO compliance and post-mortems on breaches.

I would implement this using Prometheus + Alertmanager (or CloudWatch + CloudWatch Alarms), Grafana dashboards, and automated deployment gates that check remaining error budget before allowing canary/production rollouts.

Unlock Full Question Bank

Get access to hundreds of System Monitoring and Performance Tuning interview questions and detailed answers.

Join thousands of developers preparing for their dream job.