Security Incident Response and Operations Questions

Covers the practices, processes, and tooling for responding to security incidents and operating a security capability. Topics include the security incident lifecycle of preparation, detection, analysis, containment, eradication, recovery, and post incident review; development and execution of playbooks and runbooks tailored to threat types; severity classification and decision criteria for escalation; evidence preservation and forensic analysis and chain of custody; crisis communication to stakeholders and regulators; notification and regulatory compliance considerations; and coordination with legal, privacy, communications, and executive leadership. Also includes operational aspects of building and staffing a security operations center, on call schedules and escalation, ticketing and case management, leadership and coordination during major incidents, running blameless post incident reviews to identify systemic improvements, and integration of security incident learnings into engineering and operations.

EasyBehavioral

74 practiced

Tell me about a time you were on-call for a security-related outage. Describe the situation, your role, the actions you took during the incident, how you coordinated with other teams, and what concrete changes resulted from the post-incident review. Use the STAR format.

Sample Answer

Situation: During a weekend on-call shift as the SRE for our payment API, our monitoring alerted a sudden spike in 5xx errors and abnormal outbound traffic. Our WAF also flagged a series of unusual payloads — we suspected an active exploitation attempt against an input-parsing vulnerability.

Task: As the primary on-call SRE, I needed to contain the incident to protect customer data and restore service availability, while coordinating with security, dev, and product teams.

Action:- Immediately initiated the incident channel and declared a Sev-1. I set clear roles: myself (lead responder), a security engineer (for attack analysis), and the owning service’s tech lead.- Applied mitigation: disabled the vulnerable endpoint via feature flag and added a temporary WAF rule to block the malicious patterns to stop ongoing exploitation.- Throttled egress for the service to limit data exfiltration and scaled up read-only replicas to handle legitimate traffic.- Collected forensic logs and snapshots (preserving chain-of-custody) and handed them to security for deeper analysis.- Kept stakeholders updated every 30 minutes and documented decisions in the incident timeline.

Result:- Service returned to SLO within 45 minutes; no customer data confirmed leaked.- Post-incident (blameless) review identified gaps: no automated runbook for this class of attack, slow secret rotation, and insufficient endpoint test coverage.Concrete changes implemented:- Added a specific runbook and dashboard for injection/parse attacks with playbooked mitigation steps and automated runbook-triggered feature-flagging.- Deployed automated weekly secret rotation and tightened RBAC for service accounts.- Expanded integration tests and CI fuzzing for the endpoint and introduced a canary deployment gate for schema-parsing changes.- Tuned alerts to reduce noise and added a joint SRE/Sec on-call rotation for faster coordination.

This incident reinforced the value of rapid containment, clear coordination, and turning lessons into automated safeguards.

HardTechnical

71 practiced

Implement a function in Python that computes a composite security incident severity score from signals: error_rate (0-1), auth_fail_spike (0-1), outbound_bytes_anomaly (0-1), and intel_risk (0-1). Use configurable weights and return a normalized 0-100 score. Provide a sample input and expected output.

Sample Answer

Approach: compute a weighted sum of the four normalized signals, divide by the sum of weights to get a 0–1 normalized score, apply optional non-linear scaling (tunable) to emphasize high-risk contributions, and return 0–100. Validate inputs and handle missing or out-of-range values.

python

def compute_severity(error_rate,
                     auth_fail_spike,
                     outbound_bytes_anomaly,
                     intel_risk,
                     weights=None,
                     nonlinear=None):
    """
    Compute composite severity score (0-100).
    - signals: floats in [0,1]
    - weights: dict with keys matching signals; defaults to sensible SRE weights
    - nonlinear: None, 'sqrt' (compress), or 'square' (amplify)
    """
    # default weights
    default_w = {
        "error_rate": 0.4,
        "auth_fail_spike": 0.25,
        "outbound_bytes_anomaly": 0.2,
        "intel_risk": 0.15,
    }
    w = default_w.copy()
    if weights:
        w.update(weights)

    # validate signals
    signals = {
        "error_rate": error_rate,
        "auth_fail_spike": auth_fail_spike,
        "outbound_bytes_anomaly": outbound_bytes_anomaly,
        "intel_risk": intel_risk,
    }
    for k, v in signals.items():
        if v is None:
            # treat missing as 0 (configurable alternative: raise)
            signals[k] = 0.0
        elif not (0.0 <= v <= 1.0):
            # clamp to [0,1]
            signals[k] = max(0.0, min(1.0, float(v)))

    # weighted sum normalized by sum of weights
    total_weight = sum(w.values()) or 1.0
    weighted = sum(signals[k] * w[k] for k in signals) / total_weight

    # optional nonlinear transform to emphasize extremes
    if nonlinear == "square":
        weighted = weighted ** 2
    elif nonlinear == "sqrt":
        weighted = weighted ** 0.5

    # scale to 0-100 and return rounded float
    return round(weighted * 100, 2)

Key points:- Simple, explainable linear model with configurable weights.- Nonlinear option lets you amplify (square) or compress (sqrt) the final score.- Missing signals default to 0; out-of-range inputs are clamped.

Complexity: O(1) time and O(1) space.

Sample input and expected output:- Input: error_rate=0.7, auth_fail_spike=0.4, outbound_bytes_anomaly=0.9, intel_risk=0.2- Using defaults, compute: weighted = (0.7*0.4 + 0.4*0.25 + 0.9*0.2 + 0.2*0.15)/1.0 = (0.28+0.10+0.18+0.03)=0.59 Score = 59.0- compute_severity(0.7,0.4,0.9,0.2) -> 59.0

Notes/alternatives:- Use log/sigmoid transforms or per-signal thresholds to model saturation.- For alerting, map score bands to severity levels (info/warn/critical) and tune weights via historical labeled incidents.

MediumTechnical

60 practiced

Describe the process and tools an SRE would use to perform memory forensics on a compromised Linux host: how to acquire a memory image, what volatile artifacts to prioritize (process list, loaded modules, network sockets, credentials in memory), and how to analyze with tools like Volatility or Rekall.

Sample Answer

High-level approach: preserve evidence, acquire a trusted memory image with minimal alteration, prioritize volatile artifacts (processes, modules, network sockets, credentials), then analyze offline with Volatility or Rekall using appropriate Linux profiles/kallsyms.

Steps & tools

1. Preparation / safety- Isolate the host (network segmentation) to prevent lateral movement.- Document chain-of-custody, time, user, commands run.- Use a trusted forensics workstation and checksum all captures.

2. Memory acquisition (minimal footprint)- Preferred tools: - LiME (kernel module) → produces raw or LIME format; low footprint; write to local disk or pipe to remote host. - Microsoft AVML (Linux build) → small, single binary that captures raw memory and is easy to verify. - fmem / magma / dd if=/proc/kcore (less preferred; /proc/kcore may not reflect physical memory reliably).- Example remote capture (pipe to workstation; reduces local disk writes): sudo insmod lime.ko "path=/dev/stdout format=lime" | gzip -c > host1.mem.gz Then compute a SHA256 on the resulting file.

3. Live triage (if acquisition delayed) — run quick, trusted commands and log output:- ps aux --sort=-%mem (process list)- ss -tunap or netstat -tunap (network sockets / listening ports)- lsmod (loaded kernel modules)- lsof -nP (open files & sockets)- cat /proc/*/status, /proc/<pid>/maps (memory mappings)- grep -R --binary-files=without-match -I 'password\|secret' /proc/*/environ (careful; volatile)Save outputs and hashes.

4. Volatile artifacts to prioritize- Process list & hidden processes (pslist, psscan)- Loaded modules & kernel hooks (lsmod, modscan)- Network sockets & connections (netscan, linux_netstat, sockets)- Open file descriptors and loaded libraries (linux_lsof, proc_maps)- In-memory credentials/tokens (linux_cred, yarascan, regex/string searches)- Injected code, rootkits (malfind, malproc)- Kernel symbol table / kallsyms for correct analysis

5. Analysis with Volatility / Rekall- Ensure you have the correct profile or Linux symbol table: - Volatility 3 supports Linux; supply --kdbg or a Linux symbol file; Rekall can use /proc/kallsyms captured from the host.- Typical plugin workflow (Volatility 3 / Volatility 2 equivalents): - pslist / psscan to enumerate processes and find hidden ones - linux_lsmod or modules to list kernel modules - netscan / linux_netstat for sockets and connections - linux_proc_maps + linux_vmmap to view memory regions - linux_environ / linux_bash for environment variables/commands - linux_cred / yarascan / strings/regex to find tokens, passwords, SSH keys - malfind / devicetree checks for injected code or rootkits- Extract suspicious process memory, then run yara or strings and inspect for IOCs (IP, domains, filenames, base64 blobs).

6. Post-analysis- Correlate artifacts with logs (syslog, auditd, firewall) and network captures.- Export indicators (hashes, IPs, command-lines), document findings, preserve images and analysis notes.

Why this order: acquisition first preserves maximum volatility; live triage is limited and documented; offline analysis with Volatility/Rekall is repeatable and safer. Use checksums and isolated workstations to maintain forensic integrity.

MediumTechnical

63 practiced

Your company must notify a regulator within 72 hours of a data breach. As the SRE responsible for evidence and timelines, outline the steps to collect required information (scope, personal data impacted, mitigation), coordinate with legal and privacy, and provide a timeline and communication artifacts for the regulator.

Sample Answer

Situation: A production incident exposed customer data and we have a 72-hour regulatory notification requirement.

Task: As the SRE owner for evidence/timelines, I must gather authoritative facts about scope, impacted personal data, mitigation steps, and provide a regulator-ready timeline and artifacts while coordinating with Legal and Privacy.

Action:- Triage & preserve evidence (0–1 hour) - Isolate affected services, take immutable snapshots (logs, DB backups, container images) and preserve timestamps. Note who executed actions. - Capture monitoring alerts, pager/incident timestamps, and change events from CI/CD and IAM logs.- Determine scope (1–6 hours) - Query logs and databases with reproducible queries to enumerate affected accounts, records, IPs. Use audited scripts and store outputs in read-only storage. - Correlate with access logs, firewall/IDS, and SSO events to confirm exfiltration vs. exposure.- Classify personal data (6–12 hours) - Map affected data fields to privacy categories (name, email, national ID, health, financial). Provide counts and sample hashes (not raw PII) for validation.- Mitigation & containment (ongoing, document immediately) - Document immediate fixes (revoked keys, patched vulnerability, firewall rules, forced password resets), time applied, and responsible engineers.- Coordinate with Legal & Privacy (concurrent) - Daily syncs (or ad-hoc) to align on what is reportable, thresholds, and regulator language. Provide Legal with raw evidence, sanitized summaries, and a draft notification.- Produce timeline & artifacts (within 24 hours, refine) - A chronological timeline with UTC timestamps, author, and source (alert, log, command). Include: - Incident summary (what happened) - Scope summary (counts by data type) - Technical root-cause and remediation actions - Evidence bundle (logs, hashes, config diffs) with integrity checks (SHA256) - Risk assessment and steps to prevent recurrence - Deliverables: CSV of affected identifiers (hashed), timeline doc, remediation checklist, signed chain-of-custody for artifacts.

Result:- Provide Legal/Privacy a regulator-ready packet and a one-page executive summary within 48 hours, and complete evidence bundle with integrity proofs within 72 hours. I ensure all artifacts are reproducible, timestamped, and traceable so regulator questions can be answered without delays. Lessons learned: prebuilt incident-data playbooks, automation for scoped exports, and a standing Legal/Security runbook shorten future response times.

HardTechnical

60 practiced

Propose a low-overhead detection strategy for stealthy lateral movement inside Kubernetes clusters: which telemetry to collect (audit logs, process execs, network connections), heuristics to detect suspicious sequences (unusual RBAC usage, execs from pods to nodes), and automated containment actions that minimize service disruption.

Sample Answer

Situation: You’re designing a low-overhead detection + containment capability for stealthy lateral movement in Kubernetes that SREs can run without harming availability.

Telemetry (low-cost, high-signal):- Kubernetes audit logs (API server): record RBAC/grant/rolebinding, exec/port-forward/attach, pod creation → enable metadata-only mode to reduce payload.- Kubelet/container runtime execs & process events: capture process execs from containers (Falco/eBPF-based probes) — only alert on exec/attach events, not full syscall tracing.- Pod-to-pod / pod-to-node network flows: collect connection metadata (5-tuple + process label) via eBPF or service mesh sidecar stats (no full packet capture).- RBAC/identity context: map service accounts, tokens, and impersonation events from audit logs.- Image metadata & least-privilege labels: avoid heavyweight scanning; keep image provenance and critical CVE flags.

Detection heuristics (sequence-based, low false positives):- Unusual RBAC use: new ClusterRoleBinding for a low-privileged SA + subsequent API calls that require elevated permissions within short window (e.g., create/exec/delete).- Exec-from-pod to node/pivot behavior: container execs that spawn privileged CLIs, or attempts to access /var/run/docker.sock, hostPath mounts, or use hostNetwork.- Cross-namespace lateral access: pod A authenticates to service B in different namespace using service-account tokens it never used before.- Chained actions pattern: sequence of (credential change/binding) → (pod exec/attach) → (new pod creation with higher privileges) within T (e.g., 5–15 minutes).- Anomalous network flows: pod initiating connections to control-plane ports, kubelet API, or SSH to nodes; or unusual port scanning behavior.- Baseline/score model: assign weighted scores per event (RBAC change=5, exec to host=7, network to kubelet=6); alert when score > threshold to limit noise.

Automated containment (minimize disruption):- Pre-approved playbooks + gradated actions: 1. Soft: annotate pod as “investigating”, add network policy to block outbound traffic from suspect pod (via CNI policy engine) — immediate, minimal impact to other pods. 2. Moderate: create a short-lived emergency RoleBinding revocation for the implicated SA (use token revocation / delete secret), and cordon node if node-level compromise suspected. 3. Hard (manual approval or automation with high confidence): evict pod (graceful), taint node + cordon, rollback deployment to last-known-good image.- Use Admission/OPA and mutating webhooks to enforce auto-remediation safely (e.g., prevent new high-priv pods until human review).- Rate-limit automated actions and require multi-signal confirmation (e.g., RBAC change + exec + network to kubelet) before hard containment.

Operational considerations:- Tune thresholds per-cluster/service; keep telemetry retention short for noisy low-value events.- Ship alerts to a SOC/SRE runbook with tracing context (pod, sa, timeline) and automated forensic snapshot (kubectl cp logs, ps list) saved to immutable store.- Measure false positives by shadow-mode enforcement for 2–4 weeks before automatic revocations.

This approach balances detection capability with low overhead by focusing on metadata events, sequence heuristics, graded containment, and human-in-the-loop escalation to preserve availability.

Unlock Full Question Bank

Get access to hundreds of Security Incident Response and Operations interview questions and detailed answers.

Join thousands of developers preparing for their dream job.