Software Troubleshooting and Support Questions
Covers diagnosing and resolving application and software issues across the stack, including installation and deployment problems, configuration errors, dependency conflicts, performance bottlenecks, and runtime failures. Candidates should be comfortable reading and interpreting application logs, performing root cause analysis, using debugging tools and techniques, reproducing issues, implementing hot fixes and long term fixes, and knowing when and how to escalate to development teams. Also includes best practices for incident response, rollback strategies, patching, compatibility verification, and communication with stakeholders during incident resolution.
MediumTechnical
63 practiced
Your p99 latency has increased significantly. Describe a step-by-step investigation plan including which metrics (CPU, I/O wait, GC), traces, flamegraphs, logs, and deployment metadata to examine. How do you distinguish between network, application, and infrastructure causes?
Sample Answer
1) Triage & scope- Confirm alert and time window (p99 spike start/end). Check SLO/error budget impact.- Query Prometheus for p50/p95/p99, request rate (RPS), error rate, and concurrency over that window to see if increased load explains latency.2) Correlate infrastructure metrics- CPU user/steal, load average, context switches- CPU saturation vs iowait: high iowait → disk/IO bottleneck- Memory/Swap usage and page faults- Disk IOPS, latency, queue length- Network: NIC tx/rx, errors, retransmits, tcp_retransmits, RTT- Node/container restarts, OOMs, cgroup throttling- Tools/queries: Prometheus dashboards/Grafana; example PromQL: rate(http_requests_total[5m]) and histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m]))3) Application-level signals- GC metrics: GC pause time, promotions, heap usage (for JVM/Go). Spikes in GC pause correlate to stop-the-world pauses increasing p99.- Thread pool/connection pool saturation, queue lengths, semaphore waits- Event loop blocking (Node) or syscalls blocking4) Tracing & flamegraphs- Inspect distributed traces (Jaeger/Zipkin): find slow traces, longest spans, which service/span contributes most of the latency.- Look for increased fan-out or retries.- Generate CPU flamegraph / pprof on the service during spike to find hot functions or syscalls (e.g., blocking I/O, serialization).- Compare flamegraphs pre/post spike to identify new hotspots.5) Logs & deployment metadata- Search logs (correlated by request id) for timeouts, retries, DB slow queries, 5xxs.- Check recent deployments, config changes, feature flags, JVM/GC settings, container images, and autoscaler events. Tie deploy timestamps to latency onset.- Inspect DB slow query logs, lock waits, and connection pool exhaustion.6) Isolation & repro- If traces show single downstream service, run targeted load/trace there.- Use canary/traffic split to see if only a version exhibits the problem.- Reproduce in staging with similar load to test fixes (GC tuning, query indexes, scaling, network path).7) Short-term mitigations- Roll back offending deploy if correlated.- Increase replicas/threads/connection pool or enable circuit breakers to reduce retries.- Apply rate limiting or route traffic to healthy instances.How to distinguish causes- Network: increased RTT, packet retransmits, errors on NIC, all services calling that network segment show latency; traces show span waiting on network/dns; p99 across multiple unrelated services sharing network path.- Application: high CPU in user space, flamegraph shows hot code path, GC pause spikes, thread/queue saturation, slow serialization; problem isolated to a specific service/version in traces.- Infrastructure: node-wide resource issues (high iowait, disk latency, host OOMs, hypervisor steal), many services on same host affected, kube node conditions, cloud host maintenance or noisy neighbor.Key principle: correlate across layers (metrics → traces → flamegraphs → logs → deploy metadata) to move from symptom to root cause, then apply targeted mitigation and post-incident fixes (monitoring, autoscaling, retries/backoffs, tuning, rollbacks).
EasyTechnical
60 practiced
During an incident, you open a crashed service's log. Describe the first five pieces of information you look for to triage the crash and why each is important. Assume you have access to logs, system metrics, and service metadata. Be concrete about specific log markers, timestamps, processes, and correlate to SLO impact.
Sample Answer
1) Exact timestamp and log line marker (e.g., “2025-12-06T14:23:12.345Z ERROR main”)- Why: anchors the incident window so you can correlate with metrics/alerts and other services. Precise times let you map to SLO violation windows and to request traces.2) Error type and stack trace or exception message (e.g., “java.lang.OutOfMemoryError: Java heap space” or “SIGSEGV at …”)- Why: gives immediate root-cause candidates (OOM, null pointer, segmentation fault). Stack trace points to code path and whether failure is application-level or native.3) Process/thread id and container/pod metadata (PID/THREAD/PodName/Node)- Why: isolates whether crash is single-process, single-container, or node-wide. Helps decide rollback vs. node drain, and to check kubelet/system logs for kube-eviction or OOM killer.4) Recent lifecycle events and restart count (e.g., “CrashLoopBackOff; restartCount=5”; system logs “Killed process 1234 (app) signal 9”)- Why: indicates instability pattern and blast radius. High restartCount during SLO window implies degraded availability and likely SLO breach.5) Correlated observability markers: request-id, HTTP status spikes, latency/error-rate graphs around the timestamp (e.g., 5xx rate from 0.2%→8% at same time)- Why: quantifies customer impact and maps to SLOs (error budget burn). Request-ids let you replay a failing request through traces and find upstream/downstream dependencies.Quick triage steps after gathering above: check node metrics (CPU, mem, disk, iowait), container logs for kube events, and alerting dashboards to decide mitigation (scale up, rollback, restart node).
HardTechnical
94 practiced
A large Postgres cluster exhibits silent data loss for some UPDATEs after failover. Design a plan to reproduce in staging, instrument the system to catch the bug (WAL checksums, archive verification, synchronous_commit settings), and propose a remediation and migration plan that minimizes downtime. Discuss verification and legal/regulatory implications if users' data were affected.
Sample Answer
Reproduce in staging (goal: trigger the same silent UPDATE loss)- Mirror production topology, Postgres version, config, OS, storage, WAL archiving, replication method (physical streaming + any logical replicas). Use anonymized but schema-identical data subset.- Create a test harness that performs high-rate UPDATEs with identifiable keys/timestamps and checksums (e.g., add a seq_id and compute hash after each update). Run workload while inducing failover scenarios: - Controlled primary crash (kill -9 postgres), network partition between primary and standby, and simultaneous WAL archive delays. - Timeline switches: promote standby while primary still writes to disks. - Inject fsync/fs corruption delays at storage layer (tc qdisc, drop writes) to emulate delayed WAL flushes.- Repeat until you reproduce lost/overwritten UPDATEs. Capture full logs, WAL files, and timeline history for each run.Instrument the system to catch the bug- Enable WAL checksums (pg_checksums / initdb --data-checksums) in staging to detect on-disk corruption early; note enabling on production requires full re-init or pg_upgrade/pg_basebackup + restore plan.- Verify archived WAL integrity: compute cryptographic hashes on WAL files as they are archived (e.g., sha256) and store checksums in an immutable store (S3 with versioning). Build an archive-verifier job that replays/validates WAL checksums before accepting.- Ensure wal_level = replica or higher and track wal_sender/receiver stats.- Audit synchronous_commit and synchronous_standby_names: - Detect any replicas marked synchronous but with synchronous_commit = local/remote off. Instrument via periodic checks: SELECT name, setting FROM pg_settings WHERE name IN ('synchronous_commit','synchronous_standby_names'); SELECT application_name, state, sync_state FROM pg_stat_replication; - Alert if sync_state != 'sync' while synchronous_standby_names expects sync.- Add application-level verification: after every write, persist a write-intent record to an append-only table and verify it exists on promoted node after failover.- Enable detailed logging: log_checkpoints, log_replication_commands, log_connections, log_disconnections, log_statement_sample_rate for suspicious statements.- Automated WAL replay dry-run: before promotion in staging, run pg_waldump / pg_verifychecksums and simulate recovery to identify missing WAL segments.Root-cause hypotheses to test- Asynchronous commit or misconfigured synchronous replication causing acknowledged writes not to be flushed to standby.- WAL archiving/transfer lag, missing WAL segments during promotion.- Timeline/branching causing some WAL records to be lost on promotion (primary continued accepting writes after network partition).- Storage-level write reordering or fsync not honored (O_DIRECT/O_SYNC misconfig).Remediation strategy (short-term + long-term)Short-term (stopgap, minimal config changes, fast to deploy)- Immediately set synchronous_commit = on (if safe) and ensure at least one genuinely synchronous standby (confirm sync_state = 'sync'). This ensures commit only returns after WAL is on both primary and standby.- Disable risky auto-promotions or adjust failover automation to require manual verification when WAL lag > threshold.- Configure wal_keep_size and replication slots to avoid WAL removal before standby receives them.- Start archiving WAL to redundant locations and enable WAL integrity verification pipeline.Long-term (robust, low-risk migration)- Build a migration to a verified-consistent replica using logical replication to minimize downtime: - Create a logical replication slot on a healthy primary/secondary; create publication for tables. - Create a new target server (new cluster) via base backup and apply WAL; set up subscriber on target to catch up with minimal lag. - Switch reads to target gradually; for cutover, briefly pause writes (application-level quiesce <60s), ensure subscriber caught up (pg_stat_subscription), then switch writes to target and finalize.- Alternative: Use physical replication with planned switchover (repmgr/Patroni) + pg_rewind if divergence small. For clusters requiring zero data divergence, logical replication is safer to transform and validate data.- When enabling WAL checksums in production, perform a coordinated rebuild: provision new servers, pg_basebackup from verified primary, enable checksums on new cluster, deploy, validate, then cutover.Verification after remediation- End-to-end data integrity checks: - Row counts, checksum hashes per partition, per-table diff tools (pg_diff or custom queries) between old and new cluster. - Application-level reconciliations: replay write-intent records, compare sequence gaps. - Spot-check transaction-level audits using pg_wal_lsn_diff and txid snapshots.- Test failover scenarios again in staging with instrumentation active; require that any promotion preserves all acknowledged commits.- Continuous monitoring: set alerts for WAL archive gaps, replication lag > threshold, sync_state drift, and failed WAL checksum verifications.Migration minimizing downtime- Use logical replication for near-zero downtime cutover: - Provision target, do base backup, start logical replication, let it catch up until lag < acceptable window. - Quiesce application writes for a short window (seconds to a few minutes), ensure final LSN applied, switch writes, promote subscriber to primary. - Validate post-cutover transaction integrity and rollback plan ready.- For stricter compliance where writes can’t be paused, coordinate multi-phase migration: route write traffic via a middleware that buffers new writes and replays if necessary.Verification queries / checks to run- Replication health: SELECT application_name, state, sync_state, write_lag, flush_lag, replay_lag FROM pg_stat_replication;- WAL archive gap detection: Compare latest WAL segment in pg_current_wal_lsn()/pg_walfile_name and latest archived filename stored with checksum.- Example data consistency check: SELECT count(*) FROM table WHERE md5(concat_ws('|', col1, col2, ...)) != expected_hash_grouped_by_partition;Legal/regulatory implications and actions- Treat silent data loss as potential data integrity breach. Assemble incident response: involve legal, privacy, compliance, and security teams immediately.- Determine scope: - Which users/records affected, timeframe, and whether data loss is irreversible. - Preserve all logs, WAL archives, backups, and system images for forensics (chain of custody).- Regulatory reporting: - GDPR: If loss constitutes personal data loss affecting data subject rights, assess whether data breach notification is required to supervisory authority within 72 hours. Document risk to rights/freedoms. - PCI/FISMA/other sector rules: check notification timelines and contractual obligations.- Communicate transparently but carefully: - Prepare internal timeline and technical cause analysis draft before external comms. - Notify impacted customers with remediation steps and offer remedies (data restore, compensation).- Post-incident: perform formal RCA, document mitigations, update runbooks, and adjust SLOs/SLAs; retain records for regulatory audits.Summary (SRE priorities)- Reproduce reliably in staging, instrument WAL/archive/replication and application-level verifies, and run controlled failovers.- Short-term: enforce synchronous commit and harden failover automation; long-term: migrate to a verified-consistent replica via logical replication with planned cutover.- Verify with checksums and reconciliation; treat any real user data loss as an incident with legal reporting, forensic preservation, customer notification, and remediation.
EasyTechnical
53 practiced
When opening a major incident ticket for executives and cross-functional stakeholders, what minimal but sufficient information do you include on the initial page to keep everyone informed without overloading them? Specify status fields, impact assessment, owner roles, next steps, and how frequently to update the ticket during the first hour.
Sample Answer
I use a one-page “minimal but sufficient” incident header so execs and cross-functional partners get clarity quickly without noise.Essential status fields (top-line)- Incident ID & title- Severity/Priority (e.g., Sev1)- Current status: Open / Investigating / Mitigating / Resolved- Start time (UTC) and last updated timestamp- ETA for next update (explicit time)Impact assessment (concise)- Impact statement: who is affected (customers, internal teams) and user-visible symptoms- Scope: % of traffic/users/systems affected, key regions, services, SLAs impacted- Business impact: revenue, compliance, or major customers at risk (if known)Owner roles (single-line contacts)- Incident Commander (IC) — name, pager/phone- Communications Lead — name, Slack/channel, exec update contact- Engineering Owner(s) per service — names + links to runbooks- Scribe — owns notes & action logImmediate next steps (actionable)- Current action being taken (1-2 bullets)- Short-term plan (next 15–30 minutes)- Blocking issues or needs (e.g., elevated access, rollback approval)Links / attachments- Monitoring dashboards, logs, runbooks, postmortem templateUpdate cadence (first hour)- First 5–10 minutes: acknowledge and publish initial header- Every 10 minutes for the first 30 minutes (or sooner on major changes)- Then every 15–30 minutes thereafter until stabilized- Send an explicit “exec update” when impact or ETA changes substantiallyTone: factual, timestamped, and action-oriented — no raw debug dumps.
HardTechnical
49 practiced
Disk I/O spikes on a database node are causing timeouts. Propose a detailed investigation plan: commands to collect (iostat, sar, blktrace, vmstat), what metrics to examine (queue depth, avg wait, latency percentiles), immediate mitigations (IO cgroup throttling, failover, read-only mode), and long-term fixes (storage tiering, sharding, hot/warm separation).
Sample Answer
Situation: Disk I/O spikes on a DB node causing timeouts. Investigation plan (fast triage → deep forensics → mitigation → long-term fixes).Immediate data collection (run live, send outputs to central storage):- iostat -x 1 60 — look at %util, await, svctm, avgqu-sz per device.- vmstat 1 60 — blocked processes (b), free memory, swap in/out, run queue.- sar -d 1 60 and sar -q 1 60 — historical device and queue stats.- blktrace -d /dev/sdX -o - | blkparse -i - — capture per-request timing and latencies.- ioping -c 100 /mountpoint — quick latency distribution.- pidstat -d 1 60 /proc or top/HOST output — find processes causing I/O.- echo 2 > /proc/sys/vm/dirty_background_ratio (temp) and collect dstat or atop.Key metrics to examine:- Device %util (sustained >70% indicates saturation)- avgqu-sz / queue depth (growing queue => bottleneck)- await / svctm / avg wait latency (ms)- I/O ops/sec (r/s, w/s) and throughput (MB/s)- Latency percentiles (p50/p95/p99) from blktrace/atop/Perf- Number of blocked processes and context switches- Co-tenant noisy neighbors (if virtualized) and RAID rebuilds/SMART errorsImmediate mitigations (low-risk → higher risk):- Throttle noisy processes: ionice -c2 -n7 or cgroups v2 io.max for containers to limit read/write bps/iops.- Put node in read-only mode (DB-specific) to avoid writes while failing over writes.- Trigger failover to healthy replica if within SLOs; remove node from LB/pool.- Offload heavy analytical queries to replica or run queries with lower priority.- Temporarily increase DB timeouts, circuit-breakers, and connection queue to avoid cascading failures.- If hardware failing (SMART errors), schedule replacement and failover immediately.Deep forensics & next steps:- Correlate with backups, snapshots, cron jobs, compactions (Cassandra), checkpointing or autovacuum (Postgres).- Analyze blktrace to get per-request latency percentiles and identify large sequential vs random IO.- Check storage layer: multipath, HBAs, SAN controller, network latency for iSCSI/NFS.Long-term fixes:- Storage tiering: move hot data to NVMe SSDs, cold to HDD; use caching (LVM cache, bcache, Redis).- Sharding / partitioning: reduce per-node working set to lower IOPS.- Hot/warm separation: dedicate nodes for write-heavy workloads vs analytic reads.- Tune DB parameters: write-back buffer sizes, checkpoint frequency, compaction strategies, WAL batching.- Capacity planning: increase IOPS headroom, adopt provisioned IOPS or faster disks, tune RAID stripe size.- Observability: add percentile-based I/O latency alerts, correlate with DB errors, per-tenant metrics.- Automation: auto-scale read replicas, automatic failover playbooks, and runbooks for noisy-neighbor throttling.Communicate: declare incident state, mitigation chosen, affected customers, ETA for recovery, and next steps. Post-incident: root-cause analysis, timeline, and permanent remediation plan.
Unlock Full Question Bank
Get access to hundreds of Software Troubleshooting and Support interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.