Performance Fundamentals and Troubleshooting Questions

Core skills for identifying, diagnosing, and resolving general performance problems across applications and systems. Topics include establishing baselines and metrics, using monitoring and profiling tools to determine whether issues are CPU bound, memory bound, input output bound, or network bound, and applying systematic troubleshooting workflows. Candidates should be able to prioritize fixes, recommend temporary mitigations and long term solutions, and explain when to escalate to specialists. This canonical topic covers general performance awareness, common diagnostic tools, and basic remediation approaches for slow systems and resource exhaustion.

MediumTechnical

60 practiced

A microservice shows increasing p95->p99 latency under load testing while average latency stays flat. List likely root causes tied to common backend architectures (e.g., GC pauses, lock contention, retries, cold caches). For each cause, describe a test or instrumentation you would run to confirm it.

EasyTechnical

109 practiced

Define CPU-bound, I/O-bound, and memory-bound conditions for a backend service. For each category, give: (a) one metric that best indicates the condition, (b) one tool to diagnose it in production, and (c) one short-term mitigation you might try immediately.

MediumTechnical

74 practiced

Explain how swap usage affects performance on Linux. Describe how you would distinguish swap-related performance problems from other memory issues (commands and metrics), and list three remediation strategies with pros and cons for each.

Sample Answer

**Brief effect on performance**Swap stores memory pages on disk; heavy swap activity increases latency (page-in/page-out) and can cause CPU stalls, higher I/O, and application slowdowns or thrashing.

**How to distinguish swap issues (commands & metrics)**- Check swap usage and activity:

bash

swapon --show
free -h
vmstat 1 5        # look at si (swap in) and so (swap out)
iostat -x 1 5     # device I/O utilization and latency
sar -B 1 5        # page in/out stats

- Inspect kernel reclaim and OOM behavior:

bash

dmesg | grep -i oom
cat /proc/vmstat    # pgpgin, pgpgout, pswpin, pswpout

Key metrics: high si/so, sustained high disk latency (await), low free memory + high cached/active, frequent page faults (vmstat's fault columns).

**Remediation strategies (3)**1) Add RAM- Pros: Eliminates swap pressure, best performance improvement.- Cons: Hardware cost, may require downtime or capacity planning.

2) Tune VM and swappiness (reduce swapping)- Pros: Quick, no hardware change; use `sysctl vm.swappiness=10`.- Cons: May increase OOM risk if memory pressure persists; requires careful monitoring.

3) Improve swap I/O (faster storage / dedicated swap)- Pros: Reduces swap latency (NVMe, zram, or swap on fast disk).- Cons: Adds cost/complexity; zram trades CPU for compressed RAM (may not help at high load).

Also consider cgroups limits, memory overcommit tuning, and profiling offending processes to fix memory leaks.

EasyTechnical

93 practiced

What is 'iowait' reported by Linux tools (e.g., top/vmstat)? Describe how sustained high iowait typically affects application performance and list three immediate actions you might take to reduce user-visible impact while you investigate the root cause.

EasyTechnical

56 practiced

A Linux server shows sustained high CPU utilization. Provide a step-by-step troubleshooting checklist to determine whether the load is user-space CPU, kernel CPU, or I/O wait. Include specific commands (top/htop, mpstat, iostat, vmstat, pidstat, perf) and what specific output patterns would indicate each cause.

Sample Answer

**Step-by-step checklist (Systems Engineer POV)**

**1) Quick live view**- Run top/htop to see overall CPU split and hot PIDs:

bash

top -o %CPU
htop

- Indicators: - High %us (user) → user-space CPU. - High %sy (system) → kernel CPU. - High %wa → I/O wait.

**2) Per-CPU and interrupt view**- Use mpstat to separate user/system/irq/softirq per CPU:

bash

mpstat -P ALL 1 3

- Indicators: - High all-cpus %usr → CPU-bound user processes. - High %sys, %irq, %soft → kernel/interrupts.

**3) I/O wait and device stats**- Use iostat to confirm I/O bottleneck:

bash

iostat -x 1 3

- Indicators: - High %util and high await / svctm on disks → storage latency (causing %wa). - Low disk util but high %wa is unusual—check kernel.

**4) Process-level historical CPU**- Use pidstat to see per-PID user vs system:

bash

pidstat -u -r 1 5

- Indicators: - PID shows high %usr → user-space. - PID shows high %system → kernel-mode heavy (syscalls, context switches).

**5) System-wide context and VM stats**- Use vmstat for context switches and blocked processes:

bash

vmstat 1 5

- Indicators: - High ‘bi’/’bo’ and many blocked (b) → I/O wait. - Very high cs (context switches) and low idle → kernel overhead.

**6) Kernel hotspots**- Use perf to profile kernel vs user stacks:

bash

perf top
perf record -a sleep 10; perf report

- Indicators: - Kernel symbols dominant → kernel CPU (drivers, syscalls). - User symbols dominant → application CPU.

**7) Follow-up checks**- Check dmesg for driver/interrupt errors.- Inspect application logs, GC pauses (for JVM), busy loops.- For I/O: check filesystem, multipath, network storage latency.

Use this sequence to triage: top → mpstat/iostat → pidstat/vmstat → perf + logs. Each command’s specific output patterns above point to user-space, kernel, or I/O-wait causes.

Unlock Full Question Bank

Get access to hundreds of Performance Fundamentals and Troubleshooting interview questions and detailed answers.

Join thousands of developers preparing for their dream job.