Security Incident Investigation and Remediation Questions

Focuses on systematic investigation methodology and the distinction between immediate mitigation and long term prevention. Topics include collecting and preserving evidence, establishing a reliable timeline, identifying affected systems, performing root cause analysis, containment versus remediation, and documenting findings. Covers basic digital forensics principles and chain of custody, techniques for reducing blast radius and restoring service as a short term response, and planning permanent fixes to prevent recurrence. Also addresses privacy incident investigation practices such as interviewing stakeholders, assessing regulatory and compliance implications, timeliness and documentation requirements, remediation planning, and using post incident analysis to improve processes and controls.

MediumTechnical

82 practiced

Explain how you would perform Root Cause Analysis (RCA) on a recurring outage using both the '5 Whys' technique and causal graphs/fault-tree analysis. Show how you would derive actionable, prioritized fixes from each method and how you'd verify that your fixes remediate the true root cause rather than symptoms.

Sample Answer

I’d run RCA in two complementary ways—quick hypothesis with 5 Whys to get immediate mitigations, and a causal graph / fault-tree to map complex interactions and derive durable fixes.

5 Whys (fast, human-driven)- Start with the outage: “Service X became unresponsive.”- Ask Why 1: “Why did it become unresponsive?” → CPU saturated on API nodes.- Why 2: “Why CPU saturated?” → Background job spiked CPU.- Why 3: “Why did job spike?” → Job retried massively after transient DB errors.- Why 4: “Why DB errors?” → Connection pool exhausted.- Why 5: “Why pool exhausted?” → Misconfigured pool + sudden traffic surge.- Actionable fixes from this: emergency rate-limiter on job queue, backoff fixes in job retrier, immediate pool config rollback. Prioritize by impact/effort: emergency mitigations (high impact, low effort) first.

Causal graph / Fault-tree (systematic, captures dependencies)- Build nodes: client traffic, LB, API, job queue, DB, connection pool, autoscaler, monitoring. Draw causal edges and annotate probabilities/frequency from logs/metrics.- Use fault-tree to show top event (service unresponsive) composed of AND/OR combinations (e.g., OR: CPU overload OR memory OOM; AND: pool exhaustion AND retry storm).- Derive fixes: strengthen capacity (autoscale tuning), fix retrier logic (circuit breaker + exponential backoff), pool-sizing policy, add backpressure at ingress, add health-check gating for background jobs.- Prioritize using risk matrix: likelihood * impact, implementation effort, and blast-radius. Rank: fix retrier/circuit breaker (high impact, med effort), add backpressure/ingress throttling (high impact, high effort), tune autoscaling (med impact, low effort).

Verification (prove root cause, not symptom)- Hypothesis-based tests: deploy fix in staging with synthetic workload reproducing the causal chain (simulate DB transient -> retry storm -> pool exhaustion).- Canary rollout: enable fix for small traffic slice; compare key metrics (CPU, error rate, retries/sec, connection utilization) against control.- Define success criteria: SLOs restored, retries drop to baseline, no pool exhaustion, no recurrence for a defined window.- Observability: add focused dashboards & traces, increase sampling of relevant spans, add alerts for leading indicators (retry rate, pool utilization).- Post-verification: run a blameless postmortem, lock in permanent changes (code fixes, config, runbooks), and add tests (integration test for retrier behavior) to prevent regressions.

This dual approach gives fast mitigation via 5 Whys and durable, prioritized engineering fixes via causal graphs, while verification ensures we addressed the true root cause.

EasyTechnical

62 practiced

List the data sources and concrete steps you would use to establish a reliable incident timeline across multiple services. Explain how you reconcile timestamps from different systems, handle missing or partially corrupted logs, and what artifacts you would preserve to support later forensic analysis and stakeholder reporting.

Sample Answer

Data sources:- Application logs (structured JSON), service stdout/stderr- Distributed tracing (Jaeger/Zipkin/OpenTelemetry)- Ingress/load‑balancer logs (NGINX/ALB)- System logs (syslog/journalctl), kernel messages- Metrics and alert timelines (Prometheus, Datadog)- Orchestration events (K8s events, scheduler logs)- Network flow captures (pcap, VPC flow logs)- Storage / DB audit logs, backup metadata- Deployment & CI/CD events, config management history- User reports, tickets, and screenshots

Concrete steps to build a reliable timeline:1. Ingest all sources into a central store (ELK/Opensearch + tracing backend + object store for raw files).2. Normalize records to a canonical schema: {timestamp, service, host, trace_id, span_id, level, message, raw}.3. Reconcile timestamps (see below) and sort events by adjusted time.4. Correlate via unique IDs (trace_id, request_id, txn_id) and by IP/port/session where IDs are missing.5. Fill gaps using traces → metrics (spikes) → network logs; annotate uncertain ranges.6. Produce a merged timeline with confidence scores per event and export artifacts for forensics.

Reconciling timestamps:- Verify clock sync: check NTP/Chrony status and offsets per host; collect last sync time and drift.- If hosts report drift, compute per-host offset using NTP offsets or compare known anchor events (e.g., GC logs vs. upstream request timestamp) to estimate correction.- Prefer monotonic sequence within a host for ordering local events; use trace/span parent-child relationships for causal order across hosts.- Record and apply corrections deterministically; keep original timestamps unchanged and store adjusted timestamp and offset metadata.

Handling missing or corrupted logs:- Use redundancy: cross-check other sources (traces, metrics, LB logs, DB audit) to infer missing events.- Try partial parsing: salvage headers/IDs from corrupted entries.- If logs truncated, preserve the partial file and hash it; run file carving tools to extract fragments.- Mark inferred events distinctly and attach reasoning/evidence.- When critical data missing, escalate to preserve volatile sources (memory dump, live pcap) before restart.

Artifacts to preserve for later analysis and reporting:- Immutable exports of raw logs, traces, metric dumps, and network captures (with checksums).- Adjusted merged timeline plus provenance metadata (per-event source, offsets, confidence).- Configuration snapshots, deployment manifests, and commit IDs active at incident time.- Alert history, runbook/incident chat transcript, and on-call actions.- Forensic images or memory dumps if legal/PII concerns exist.- A one‑page incident summary and a technical appendix with evidence links.

This approach ensures reproducible timelines, auditable corrections, and clear evidence for postmortem and stakeholder reporting.

MediumTechnical

71 practiced

A monitoring alert shows large outbound traffic from several hosts and a customer reports sensitive files appearing online. As SRE on-call, walk through the immediate containment steps you would take to reduce blast radius and restore service while preserving forensic evidence. Include CLI/network actions, temporary ACLs, feature flag toggles, and guidance on what to snapshot or capture first.

Sample Answer

Situation: Multiple hosts show large outbound traffic and a customer reports sensitive files leaked online. Immediate goal: contain exfiltration, preserve forensic data, keep services healthy where possible.

Immediate actions (in order, with rationale):1. Triage & scope- Confirm alerts, list affected hosts (monitoring/asset DB).- Identify public endpoints receiving data (netflow/egress logs).

2. Fast containment (minimize blast radius, preserve evidence)- Put affected hosts into network isolation mode (quarantine VLAN / security group) rather than powering off. - AWS example: detach from public subnets / apply restrictive security group: aws ec2 modify-instance-attribute --instance-id i-... --groups sg-xxxx (sg with only ssh from SOC)- On-prem / Linux hosts: add temporary iptables/nft rule to block all outbound except to internal collectors:

bash

# block all outbound except to forensics collector IP 10.0.0.5
ssh ops@host
sudo iptables -I OUTPUT 1 -d 10.0.0.5 -j ACCEPT
sudo iptables -I OUTPUT 2 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
sudo iptables -I OUTPUT 3 -j REJECT

- Kubernetes: cordon nodes/pods, scale down risky workloads, or apply NetworkPolicy to deny egress:

bash

kubectl label node node-1 quarantine=true
kubectl patch networkpolicy deny-egress -p '{"spec":{"policyTypes":["Egress"],"egress":[]}}'

- If using cloud firewalls, quickly apply ACL to block suspicious outbound ports/IP ranges.

3. Preserve forensic evidence (do first before disruptive changes)- Capture volatile data on each host (memory, network connections, processes):

bash

# on host
sudo netstat -tunp > /tmp/netconns.txt
sudo ss -tunp >> /tmp/ss.txt
sudo ps auxwww > /tmp/ps.txt
sudo tcpdump -w /tmp/egress.pcap -s 0 port not 22 &
sudo gcore $(pgrep suspicious_process)  # if safe

- Snapshot disks and create AMI / volume snapshot (cloud) immediately: - AWS: create-snapshot or create-image; GCP: disk snapshot.- Collect system logs, application logs, auditd, bash_history, cloudtrail/cloud logging, flow logs (VPC flow logs).- Export copies to secure forensics bucket with write-once permissions.

4. Short-term service continuity- If quarantine breaks critical SLA, route traffic to healthy fallback instances or enable read-only mode/feature-flag sensitive features. - Toggle feature flags to disable file downloads or external sharing (LaunchDarkly/example).- Rate-limit outbound egress at edge (WAF or API gateway) to reduce exfiltration speed.

5. Evidence integrity & coordination- Record all commands, times, operator IDs in incident log.- Preserve timestamps and set storage to immutable when possible.- Notify security/forensics and escalate to incident commander; hand off preserved artifacts.

6. Follow-up steps- Once contained, perform deeper forensic analysis, rotate credentials, revoke keys/compromised tokens, rotate certificates, audit IAM, and patch vulnerability.- Run post-incident blameless review and implement permanent controls (e.g., egress filtering, DLP, least-privilege, monitoring of sensitive file access).

Key principles: isolate network egress quickly (prefer network-level blocks over host shutdown), capture volatile data first, snapshot immutable storage, use feature flags to reduce functionality without full outage, and preserve audit trail for investigators.

EasyTechnical

85 practiced

Define 'blast radius' in the context of distributed systems and give three concrete, actionable strategies SREs can use at runtime to reduce blast radius. Provide short examples such as network segmentation, rate-limiting, and feature flags and describe when each is most appropriate.

HardTechnical

80 practiced

Explain the legally defensible steps and documentation an SRE team must produce during a data privacy breach to comply with GDPR and CCPA notification requirements. Include timelines for notification, how to map and identify affected data subjects, specific logs/evidence to retain, and how to coordinate actions with Privacy and Legal teams.

Sample Answer

Situation summary (brief): As SRE you must both enable the technical forensic work and produce legally defensible documentation so Privacy/Legal can meet GDPR and CCPA obligations. Below is a practical, role-appropriate checklist with timelines, evidence to collect and retain, mapping steps, and coordination playbook.

1) Immediate actions (hours)- Contain & preserve: isolate affected systems (network segmentation, snapshots) but do not alter disks or volatile memory unless forensically captured. Use write-blockers or cloud snapshot APIs; preserve VM images and container artifacts.- Internal alerting: notify Incident Response Lead, Security, Privacy, and Legal within 1–3 hours (per IR runbook).- Triage note: create an immutable incident ticket with timestamped status updates (SIEM event ID, detection method).

2) Assessment & mapping (within 72 hours)- Scope data subjects & records: - Use data inventory/data-flow maps, DB schemas, logging of PII fields, encryption metadata, and DPIA outputs to identify affected data categories and estimate number of data subjects. - Query backups/snapshots with reproducible SQL/ETL queries or scripts; document queries and results (counts, hashes). - If full mapping is unknown, produce conservative estimates and document assumptions.- Risk assessment: with Privacy, determine likelihood of harm (high → data-subject notification required under GDPR “without undue delay” and under CCPA).

3) Regulatory timelines- GDPR: notify supervisory authority “without undue delay” and no later than 72 hours after becoming aware. If missed, document reasoned justification. Notify data subjects “without undue delay” when high risk to rights/freedoms.- CCPA/California breach laws: notify affected consumers “without unreasonable delay.” Under recent guidance, provide consumer notice promptly and Attorney General notification if >10k residents (or per state law; follow Legal’s interpretation). Many states require reporting within 30–45 days—treat 30 days as conservative maximum unless Legal advises otherwise.- Internal deliverables: initial summary to Legal/Privacy within 24 hours; preliminary regulator-ready brief within 72 hours.

4) Evidence and logs to collect & retain (forensically)- Immutable logs and evidence chain-of-custody: - SIEM/alert logs, IDS/IPS events - Network flow and packet captures (pcap) around event window - Host forensic images or snapshots (with checksums) - Systemd/journal, application logs, web server logs, DB transaction logs/binlogs - Authentication/authorization logs (IAM, SSO, MFA events) - Cloud provider audit logs (CloudTrail, Audit Logs), object storage access logs - KMS/key-management access logs, certificate logs - Change-management records, deployment pipelines, recent config commits - Backups and retention metadata - Screenshots/export of error messages, API responses, IDS alerts- For each artifact record: collector, timestamp, hash (SHA-256), who collected it, preservation location. Retain logs for minimum 1–2 years (adjust per Legal/regulatory requirements).

5) Documentation to produce (legally defensible)- Incident timeline: minute-level timeline with source evidences and references to log IDs/hashes.- Forensic report: scope, root cause hypothesis, methods, artifacts with hashes, who handled evidence (chain-of-custody).- Data impact statement: number and categories of affected data subjects, records, types of PII, encryption status, and exposure vector.- Risk assessment memo from Privacy (harm likelihood and mitigation).- Notification drafts and mailing lists approved by Legal (templates for supervisory authority, affected data subjects, press if needed).- Remediation plan and verification evidence (patches, configuration changes, test results).

6) Coordination with Privacy & Legal (how-to)- Single source of truth: maintain a shared incident folder (read-only for external counsel) with indexed artifacts and the immutable timeline.- Daily syncs (or more frequent) between SRE, Security, Privacy, Legal to share findings, agree on scope, and approve communications.- Lawful advice: Legal assesses obligations and drafts regulator/consumer notices; SRE supplies factual artifacts, counts, and technical mitigations.- Public/PR: route all external communications through Legal/Communications; SRE provides plain-language technical summary.- Regulatory correspondence: Legal leads submission; SRE provides annexes (forensic report, data impact) as requested.

7) Forensics & preservation best practices- Do not alter original evidence; always work on copies. Record hashes before and after transfer.- Timestamp everything; use NTP-synchronized times and record timezone.- If malware suspected, preserve memory dump and endpoint artifacts quickly.

8) Post-incident (30–90 days)- Produce final incident report, lessons learned, evidence index, and compliance artifacts.- Implement and document remediation, monitoring upgrades, and preventive controls.- Retain all records for statutory retention period and for potential regulatory audits.

Key principle: every technical assertion must be traceable to preserved evidence (log IDs, hashes, queries). Work closely with Privacy and Legal to err on the side of prompt notification when risk is uncertain, keeping a documented rationale for timing decisions.

Unlock Full Question Bank

Get access to hundreds of Security Incident Investigation and Remediation interview questions and detailed answers.

Join thousands of developers preparing for their dream job.