Production Incident Response and Diagnostics Questions

Covers structured practices, techniques, tooling, and decision making for detecting, triaging, mitigating, and learning from failures in live systems. Core skills include rapid incident triage, establishing normal baselines, gathering telemetry from logs, metrics, traces, and profilers, forming and testing hypotheses, reproducing or simulating failures, isolating root causes, and validating fixes. Candidates should know how to choose appropriate mitigations such as rolling back, applying patches, throttling traffic, or scaling resources and when to pursue each option. The topic also includes coordination and communication during incidents, including incident command, stakeholder updates, escalation, handoffs, and blameless postmortems. Emphasis is also placed on building institutional knowledge through runbooks, automated diagnostics, improved monitoring and alerting, capacity planning, and systemic fixes to prevent recurrence. Familiarity with common infrastructure failure modes and complex multi system interactions is expected, for example cascading failures, resource exhaustion, networking and deployment issues, and configuration drift. Tooling and methods include log analysis, distributed tracing, profiling and debugging tools, cross system correlation, and practices to reduce mean time to detection and mean time to resolution.

MediumTechnical

61 practiced

You suspect a network partition between application servers and the regional database is causing timeouts. Describe how you would use tcpdump, traceroute, and system metrics to confirm whether the issue is packet loss, routing, or firewall changes. Provide example tcpdump filters you might use and explain what packet patterns (retransmits, ICMP unreachable, RSTs) would indicate each problem type.

Sample Answer

**Approach (quick plan)**- Reproduce traffic while capturing packets on app server and, if possible, DB side.- Use tcpdump to inspect TCP behavior (retransmits, RSTs) and ICMP.- Use traceroute/MTR to find hop where loss/latency appears.- Correlate with system/network interface and firewall metrics (ifconfig/ip -s, ethtool, netstat/ss, iptables/logs).

**Key tcpdump captures (examples)**

Capture application→DB DB port (save for later analysis):

bash

sudo tcpdump -i eth0 host 10.0.5.10 and tcp port 5432 -w /tmp/db.pcap

Show verbose live (see retransmit notes):

bash

sudo tcpdump -i eth0 host 10.0.5.10 and tcp port 5432 -vv

Watch for ICMP unreachable messages:

bash

sudo tcpdump -i eth0 icmp and host 10.0.5.10 -vv

Filter TCP RSTs:

bash

sudo tcpdump -i eth0 'tcp[tcpflags] & (tcp-rst) != 0 and host 10.0.5.10' -vv

Detect obvious retransmits/duplicate acks (tcpdump prints “[TCP Retransmission]”/“[TCP Dup ACK]” when verbose).

**What to look for and what it indicates**- Frequent “[TCP Retransmission]”, growing RTTs, duplicate ACKs: - Indicates packet loss or congestion somewhere on the path (loss -> retransmits). Use traceroute/MTR to identify hop with >0% loss and rising latency.- ICMP Destination/Host/Net Unreachable messages: - Indicates routing blackhole, ACLs on routers, or route changes. Traceroute will often stop or show * at a particular hop where ICMP errors originate.- Many TCP RSTs from intermediate IPs or from DB: - If RSTs originate from firewall IPs or gateway, likely firewall/ACL actively rejecting connections (policy change). If RSTs originate from DB, service closing connections (app or port closed).- No packets arriving at DB (seen on app tcpdump) but traceroute shows path: - Could be outbound filtering or asymmetric routing; check capture on DB side or mirror, review iptables/nsgs and cloud security groups.

**Use traceroute/MTR**- Run:

bash

traceroute -n 10.0.5.10
mtr -c 100 -r 10.0.5.10

- Look for hop where loss or latency spikes begin. Consistent loss at a single hop -> network device/routing issue. Increasing loss toward end hops -> last-mile or host problem.

**System & firewall checks**- Interface drops / errors:

bash

ip -s link show eth0
ethtool -S eth0

Significant tx/rx errors -> NIC/cable issues.- Socket stats and queue drops:

bash

ss -s
netstat -s | grep retransmit

- Firewall/rules/logs:

bash

sudo iptables -L -n -v
sudo tcpdump -i eth0 'host <firewall-ip> and icmp'
sudo journalctl -u firewalld -e

**Correlation and next steps**- If tcpdump shows retransmits + MTR shows hop loss → escalate to network team for packet-loss/capacity checks (link, switch, provider).- If ICMP unreachables or traceroute stops at particular router → check route tables, ACLs, BGP changes, cloud route tables/security group updates.- If RSTs with firewall IP or iptables counters increased → revert recent firewall rule changes or adjust ACLs and validate.- Always capture on both ends (or use TAP/span) where possible to detect asymmetry.

This method gives deterministic evidence: tcpdump shows symptoms (retransmit/RST/ICMP), traceroute localizes the hop, and system metrics confirm host/interface/firewall causes.

MediumTechnical

72 practiced

Write a script in Bash or Python that scans /var/log and rotated logs in /var/log for entries containing 'ERROR' within the last hour, aggregates counts by process name or tag, and prints the top 5 sources with counts. Describe how your script handles log rotation, compressed logs, timezone differences, and production performance concerns.

Sample Answer

**Approach (brief)**- Walk /var/log for active and rotated logs (including .1, .gz).- Stream each file (use gzip for compressed) and parse syslog-like timestamps.- Keep entries from the last hour, extract process/program tag, aggregate counts, print top 5.- Handle rotation by scanning rotated filenames; handle timezone by interpreting timestamps as local time; use streaming to limit memory and scale.

**Sample Python script**

python

#!/usr/bin/env python3
import os, re, gzip, time
from collections import Counter
from datetime import datetime, timedelta

LOG_DIR = "/var/log"
SINCE = datetime.now() - timedelta(hours=1)
TS_RE = re.compile(r'^(?P<mon>\w{3})\s+(?P<day>\d{1,2})\s+(?P<time>\d{2}:\d{2}:\d{2})\s+(?P<host>\S+)\s+(?P<tag>[^:\[]+)')
MONTHS = {'Jan':1,'Feb':2,'Mar':3,'Apr':4,'May':5,'Jun':6,'Jul':7,'Aug':8,'Sep':9,'Oct':10,'Nov':11,'Dec':12}

def iter_log_files(path):
    for name in os.listdir(path):
        if name.startswith('.'): continue
        full = os.path.join(path, name)
        if os.path.isfile(full) and (name.endswith('.log') or name.endswith('.txt') or name.endswith('.1') or name.endswith('.gz') or name.endswith('.0') or name.endswith('.1.gz')):
            yield full

def open_file(path):
    if path.endswith('.gz'):
        return gzip.open(path, 'rt', errors='ignore')
    return open(path, 'r', errors='ignore')

def parse_ts(mon, day, timestr):
    now = datetime.now()
    year = now.year
    month = MONTHS.get(mon, now.month)
    dt = datetime(year, month, int(day), *map(int,timestr.split(':')))
    # adjust year if logs from Dec and now Jan
    if dt > now + timedelta(days=1):
        dt = dt.replace(year=year-1)
    return dt

counts = Counter()
for f in iter_log_files(LOG_DIR):
    try:
        with open_file(f) as fh:
            for line in fh:
                if 'ERROR' not in line: 
                    continue
                m = TS_RE.match(line)
                if not m: 
                    continue
                ts = parse_ts(m.group('mon'), m.group('day'), m.group('time'))
                if ts >= SINCE:
                    tag = m.group('tag').strip()
                    counts[tag] += 1
    except PermissionError:
        continue

for tag, c in counts.most_common(5):
    print(f"{tag}: {c}")

**How it handles key concerns**- Log rotation: scans rotated filenames (*.1, *.1.gz, *.0) alongside active logs so rotated content is included.- Compressed logs: opens .gz with gzip module and streams text; avoids full decompression to disk.- Timezones: assumes syslog timestamps are local system time; constructs datetimes in local timezone. If logs come from other TZs, normalise by parsing TZ info or using UTC timestamps when available.- Performance in production: streams files line-by-line (low memory), filters early with substring test 'ERROR' to reduce parsing cost, skip inaccessible files, and can be parallelized per-file or replaced by system tools (zgrep + awk) for very large volumes; schedule as a cron job and throttle IO to avoid contention.

MediumTechnical

65 practiced

A specific API endpoint shows sporadic CPU spikes. Describe how to safely reproduce and debug the issue using profiling tools (for example perf, pprof, async-profiler) in production with minimal overhead. Explain how to analyze resulting profile data (flamegraphs) to find hotspots and how to deploy a fix safely (canary, rollback plan, verification).

HardTechnical

84 practiced

A multi-tenant relational datastore has detected data corruption affecting a subset of tenant rows. Walk through a comprehensive incident response plan: immediate containment, forensic data collection, trade-offs between full point-in-time restore and selective repair, strategies for data reconciliation, customer communication priorities, and regulatory/compliance reporting that may be required.

Sample Answer

**Clarify scope & initial goal**- Confirm which tenants/rows are affected, time window, and whether corruption is logical (schema/data) or physical (storage-level).

**Immediate containment**- Isolate affected DB replicas/read replicas from promotion; disable writes to affected partitions/schemas; enable read-only mode for impacted tenants if possible.- Quarantine related application services and client connections; throttle jobs that touch suspect ranges.

**Forensic data collection**- Snapshot current storage (block-level) and database backups; preserve WAL/binlog segments, transaction IDs, and timestamps.- Collect DB logs, audit trails, OS metrics, recent deploys, and user activity. Record checksums/hashes of corrupted rows.- Work with storage team to avoid auto-repair actions (scrubbing) until copies taken.

**Trade-offs: full point-in-time restore vs selective repair**- Full PITR: faster correctness guarantee, simpler verification, but downtime for all tenants and potential data loss for post-restore writes.- Selective repair: less customer disruption, complex to identify consistent cut, risk of referential/integrity mismatches and longer validation.- Choose based on blast radius, SLAs, RTO/RPO, and regulatory constraints.

**Reconciliation strategies**- Use verified backups + WAL replay to reconstruct clean state for affected tenant rows.- If selective: export clean rows from PITR copy, apply via vetted merge scripts with transactions and FK checks; use checksums and replay logs to ensure sequence consistency.- Run automated integrity checks and application-level acceptance tests; run canary restores for sample tenants.

**Customer communication & priorities**- Triage customers by severity/SLA; notify impacted tenants immediately with facts, next steps, ETA, and remediation options (full restore vs selective repair).- Provide regular updates, post-incident report, and remediation timeline. Offer data-loss mitigation (compensations, support for manual reconciliation).

**Regulatory/compliance reporting**- Determine jurisdictional regs (GDPR, SOC2, PCI) — report breaches within required windows if personal/PII affected.- Preserve chain-of-custody for forensic artifacts; engage security/compliance and legal teams; document decisions, timelines, and remediation evidence.

**Post-incident**- Root cause analysis, strengthen backups, add checksums, improve monitoring/alerts, rehearse recovery runbooks, and update incident playbooks.

MediumTechnical

73 practiced

A production Java service shows steadily increasing heap usage and occasional OutOfMemoryError (OOME) entries in the logs. With minimal downtime, provide a step-by-step diagnosis and mitigation plan. What data would you collect (GC logs, heap dumps, thread dumps), which tools would you use (jmap, jstack, async-profiler, GC logs), and how would you safely capture and analyze this evidence in production?

Sample Answer

**Summary approach**Provide quick mitigation to reduce customer impact, then collect evidence for root cause with minimal downtime, analyze, remediate, and validate.

**Immediate mitigation (minutes)**- Temporarily reduce load: scale out instances / shift traffic via load balancer.- Reduce memory usage: lower JVM max-heap (-Xmx) not recommended mid-run; prefer restarting one instance at a time with increased logging and GC options.- Enable/rotate logs and alerting.

**Data to collect**- GC logs (for allocation/retention / Full GC frequency)- Heap dumps (for retained set analysis)- Thread dumps (to correlate stuck threads)- Process stats (RSS, CPU, open files)

**Tools & safe capture commands**- GC logs: ensure JVM started with

bash

-XX:+UseGCLogFileRotation -Xlog:gc*:file=/var/log/app/gc.log:time,uptime,level,tags

- Live GC stats: jcmd <pid> GC.class_histogram- Heap dump (safe, may pause): use jmap -dump:live,format=b,file=/tmp/heap.hprof <pid> - If jmap unsafe on your JVM, prefer jcmd <pid> GC.heap_dump /tmp/heap.hprof- Thread dump: jstack -l <pid> > /tmp/threaddump.txt- Low-overhead profiling: async-profiler (cpu/alloc) attached briefly:

bash

./profiler.sh -d 30 -e wall -o collapsed <pid> > profile.txt

- OS: top, ps, pmap, /proc/<pid>/status, lsof -p <pid>

**Capture precautions**- Take one instance out of rotation before heap dump to avoid user impact.- Schedule dumps during low traffic; warn stakeholders.- Heap dumps can be large—ensure disk space and compress after capture.- Prefer jcmd over jmap on newer JVMs; test commands on staging first.

**Analysis plan**- GC logs: look for increasing tenured occupancy, frequency of Full GC, promotion failure.- Heap dump: open in Eclipse MAT -> look for biggest retained set, leak suspects (singletons, caches).- Histograms: jcmd GC.class_histogram to find growing class/instance counts.- Thread dumps: correlate with long GC or blocking I/O.- async-profiler allocation flamegraphs to find high-allocation code paths.

**Remediation**- Fix code (unbounded caches, listeners), patch libraries, tune GC or heap, add monitoring and alerts.- Roll out fix on canary then full rollout; validate with retained-set and GC metrics.

This sequence minimizes downtime, collects safe evidence, and leads to an actionable fix.

Unlock Full Question Bank

Get access to hundreds of Production Incident Response and Diagnostics interview questions and detailed answers.

Join thousands of developers preparing for their dream job.