Security Incident Response and Operations Questions

Covers the practices, processes, and tooling for responding to security incidents and operating a security capability. Topics include the security incident lifecycle of preparation, detection, analysis, containment, eradication, recovery, and post incident review; development and execution of playbooks and runbooks tailored to threat types; severity classification and decision criteria for escalation; evidence preservation and forensic analysis and chain of custody; crisis communication to stakeholders and regulators; notification and regulatory compliance considerations; and coordination with legal, privacy, communications, and executive leadership. Also includes operational aspects of building and staffing a security operations center, on call schedules and escalation, ticketing and case management, leadership and coordination during major incidents, running blameless post incident reviews to identify systemic improvements, and integration of security incident learnings into engineering and operations.

MediumTechnical

78 practiced

You detect large uploads from internal workstations to a personal cloud storage provider suspected of data exfiltration. Outline containment and eradication steps that preserve evidence and minimize business disruption. Include immediate network controls, endpoint actions, how to preserve cloud provider metadata, and communication with business owners.

Sample Answer

Situation overview- I detect large outbound uploads from internal workstations to a personal cloud provider suspected of data exfiltration. Priority: stop ongoing exfil, preserve evidence, minimize business impact, and notify stakeholders.

Immediate network containment (first 15–30 minutes)- Block or throttle egress to the specific cloud provider IPs/domains at the perimeter firewall/proxy; implement an allow/deny rule scoped to affected subnets to limit blast radius.- Deploy an inline IDS/IPS rule to drop matching sessions and mirror traffic to a capture appliance for forensic review.- If business-critical services share the same provider, use targeted blocks (IPs, URLs, SNI) rather than organization-wide cutoffs.

Endpoint actions- Isolate affected endpoints to network VLAN/quarantine while preserving powered-on state for live forensics (do not power off).- Remotely collect volatile artifacts: memory image, active network connections, running processes, open files, browser sessions, and endpoint logs using approved forensic tools.- Create full disk images where feasible; document chain-of-custody and timestamps.

Preserving cloud provider metadata- Capture outbound TLS metadata: SNI, certificate details, destination IPs and ports, and timing from network captures.- Preserve HTTP(S) headers and timestamps from proxy logs and the packet capture (PCAP).- Request provider logs via legal/compliance channels: account activity, upload object metadata (object names, sizes, hashes), auth tokens, IPs, and timestamps. Use the provider’s preservation or legal-hold features if available.- Hash and timestamp all collected artifacts; maintain a tamper-evident log.

Evidence handling and analysis- Centralize logs and artifacts in an evidence repository with access controls.- Correlate SIEM alerts, EDR telemetry, and network captures to map scope and data touched.- Identify data categories and sensitivity to inform notification and remediation priorities.

Eradication and remediation- Remove malicious tooling, revoke compromised credentials and API keys, rotate affected service accounts, and patch vulnerabilities used for exfil.- Restore endpoints from clean image if compromise confirmed; validate with endpoint scanning and monitoring.

Communication with business owners and stakeholders- Notify impacted business owners immediately with concise facts: systems affected, suspected data types, immediate mitigation steps, and recommended next actions.- Provide timelines for containment, evidence collection, and when normal operations may resume.- Coordinate with Legal, Privacy, and Incident Response for external reporting, regulatory obligations, and provider engagements.

Follow-up actions- Conduct root-cause analysis, implement longer-term controls (DLP, stricter egress policies, CASB), update playbooks, and run tabletop exercises.- Produce an incident report with lessons learned and measurable remediation status.

HardTechnical

80 practiced

Perform a root cause analysis for an incident where an attacker used credentials stored in a misconfigured secrets manager in a CI/CD pipeline to access production. Describe technical artifacts to collect from the pipeline and runners, the steps to remediate immediately and long-term (secrets rotation, policy changes), and process improvements to prevent future pipeline compromises.

Sample Answer

**Situation / Goal**Perform an RCA for an incident where attacker used credentials stored in a misconfigured secrets manager in CI/CD to access production. Below is a structured investigation, immediate containment, remediation (short + long term), and process improvements.

**Artifacts to collect**- CI/CD audit logs (pipeline runs, job start/stop times, triggering users/commits, IPs)- Secrets manager logs (access attempts, grant events, token issuance)- Runner/agent host logs (syslog, process lists, shell history, docker/container logs)- Build artifacts and workspace snapshots from impacted jobs- Git commit history & PRs tied to pipeline triggers- Network logs (firewall, proxy, cloud VPC flow logs) for suspicious egress- Cloud IAM/token metadata and active sessions- SIEM alerts, EDR telemetry from any involved hosts

**Immediate containment (first 0–24 hrs)**1. Revoke/disable the exposed secrets and any active tokens immediately.2. Quarantine affected runner instances (isolate network, snapshot disks).3. Block attacker IPs and revoke compromised service principals/keys.4. Pause/disable affected pipelines until secure configuration verified.5. Preserve all collected artifacts with integrity hashing and chain-of-custody.

**Remediation (short-term)**- Rotate all credentials/keys the attacker could use; enforce short-lived tokens.- Reprovision new service accounts with least privilege; revoke old ones.- Rebuild runners from trusted images; redeploy with hardened configs.

**Remediation (long-term)**- Enforce secrets never stored in plaintext in pipeline variables; use dedicated secrets manager with access policies and encryption.- Require short-lived ephemeral credentials (OIDC, STS) for CI tasks.- Implement MFA/conditional access for service account operations where supported.- Harden runner security: immutable runners, minimal tools, no persistent credentials.

**Policy & Controls**- Implement least-privilege IAM roles, deny-by-default policies, and audited role assumption.- Require code reviews/approvals for pipeline changes that alter secrets/config.- Enforce branch protection and signed commits for CI-triggered deployments.- Enable secrets manager access logging and automated alerts on anomalous accesses.

**Process improvements / Prevention**- Adopt OIDC-based CI-to-cloud authentication to avoid long-lived secrets.- Use ephemeral secrets brokers (HashiCorp Vault, cloud-native short-lived tokens).- Integrate CI/CD into devsecops: automated scans for plaintext secrets in commits, pipeline linting to detect misconfigurations.- Regular pipeline runner hardening and automated image rebuilds; periodic pentests and “red-team” pipeline exercises.- Create runbooks & tabletop exercises; ensure incident response drills include pipeline compromise scenarios.

**Why this addresses root cause**- Collecting artifacts links attacker actions to pipeline events and credentials.- Immediate rotation and isolation stop active access.- Moving to ephemeral credentials + policy/automation removes root cause: misconfigured persistent secrets in CI.

HardTechnical

68 practiced

During an ongoing high-severity breach, executive leadership requires immediate public statements and regulatory notifications. As Incident Manager, draft a 48-hour communications plan that coordinates technical updates, legal review, privacy considerations, PR messaging, and regulator/customer notifications. Describe who communicates what, the approval flow, and how to ensure technical accuracy while minimizing legal exposure.

Sample Answer

Situation and objectiveAs Incident Manager (Information Security Analyst), I need a 48-hour communications plan to provide timely public statements and regulator/customer notifications while preserving technical accuracy and minimizing legal exposure during a high-severity breach.

48-hour timeline (high-level)- Hour 0–2: Triage brief & message skeleton - Lead: Incident Manager (me) runs war room; produce a one-paragraph holding statement (facts-only). - Inputs: SOC triage summary, affected systems, scope estimate. - Review: Legal and Privacy on standby for red flags.

- Hour 2–8: Drafting and stakeholder alignment - Technical lead: provides a concise root-cause/status note (what happened, what’s affected, mitigation steps). - Privacy: annotates PII/PHI exposure assessment. - Legal: flags admission risks, regulatory timelines, mandatory content. - PR: crafts consumer-facing language from holding statement. - Approval flow: Incident Manager → Legal → CEO/Corp Comm sign-off (parallel review by Privacy & CTO for technical accuracy).

- Hour 8–24: Initial public/regulatory notifications - Public statement: PR publishes approved holding statement through corporate channels. - Regulator notifications: Legal files required notices (e.g., data protection authority, sector regulator) with Incident Manager supplying technical appendix. - Customer/email notice: Customer Ops/PR sends tailored notification; technical FAQ appended and maintained by Incident Manager.

- Hour 24–48: Technical update and follow-up - Hourly technical updates in war room; Incident Manager prepares technical update package. - Legal reviews updates for liability language; Privacy confirms remediation for exposed data. - PR issues an update with Q&A; Legal/Privacy appended as needed. - Regulatory follow-ups and evidence retention handled by Legal with logs and chain-of-custody prepared by me.

Who communicates what (roles)- Incident Manager (me): coordinates, prepares technical summaries, maintains FAQ, ensures chain-of-custody for evidence.- Technical lead/CTO: detailed remediation steps, systems impacted, ETA for fixes.- Legal: approval, regulator filings, language to limit admissions and preserve privilege.- Privacy Officer: exposure assessment, notification content for individuals.- PR/Corp Comm: external messaging, media handling, spokesperson.- Customer Ops/Account Mgmt: direct customer notices, support routing.

Approval flow and gating- All external messaging requires Legal approval first, then CEO/Corp Comm sign-off. Technical appendix requires Technical lead + Incident Manager sign-off for accuracy. Use short SLA for approvals (Legal: 60 min, Technical: 30 min) with escalation to GC/CTO.

Ensuring technical accuracy while minimizing legal exposure- Use facts-only, time-stamped statements: what happened, what systems, known impact, actions taken.- Separate technical appendix (for regulators/partners) from public statement; label as “for regulator review” and submit under privilege when applicable.- Avoid speculative causes or blame; use conditional language for unknowns and commit to updates.- Preserve privileged communications: route technical analyses through Legal-managed channels when intended for regulatory submission.- Keep an auditable log of who provided each fact and timestamp evidence (SIEM exports, packet captures) to support regulatory responses.

Metrics and checkpoints- Publish holding statement within 8 hours.- File regulator notices within statutory window (24–72 hours as required).- Hourly internal updates; external updates every 12–24 hours until contained.

ClosingThis plan balances speed with control: technical accuracy comes from direct technical sign-off; legal minimizes exposure by gating statements and using privileged channels for sensitive details.

MediumTechnical

63 practiced

Write a Python script or detailed pseudocode that reads a CSV file of suspicious IP addresses and queries a SIEM REST API to tag matching events with 'threat-intel-hit'. Describe how your implementation handles API rate limits, transient errors, retries with backoff, idempotency, and logging for auditing.

Sample Answer

**Approach (brief)** Read CSV of IPs, query SIEM for events matching each IP, then tag matching events with tag "threat-intel-hit". Use resilient HTTP client with exponential backoff, honoring Retry-After for rate limits, idempotent PATCH operations, and structured logging for audit.

**Sample Python script**

python

import csv, time, logging, requests
from urllib.parse import quote

# config
SIEM_BASE = "https://siem.example/api"
API_KEY = "REDACTED"
HEADERS = {"Authorization": f"Bearer {API_KEY}", "Content-Type": "application/json"}
TAG_BODY = {"tags": ["threat-intel-hit"]}

# logging
logging.basicConfig(filename='siem_tagging.log', level=logging.INFO,
                    format='%(asctime)s %(levelname)s %(message)s')

def backoff_sleep(attempt, retry_after=None):
    if retry_after:
        sleep = int(retry_after)
    else:
        sleep = min(60, (2 ** attempt) + (attempt * 0.1))
    time.sleep(sleep)

def request_with_retries(method, url, **kwargs):
    max_attempts = 5
    for attempt in range(1, max_attempts+1):
        resp = requests.request(method, url, headers=HEADERS, timeout=30, **kwargs)
        if resp.status_code in (200, 201, 204):
            return resp
        if resp.status_code == 429:
            retry_after = resp.headers.get("Retry-After")
            logging.warning("Rate limited: %s %s attempt=%d retry_after=%s", method, url, attempt, retry_after)
            backoff_sleep(attempt, retry_after)
            continue
        if 500 <= resp.status_code < 600:
            logging.warning("Transient SIEM error %d for %s attempt=%d", resp.status_code, url, attempt)
            backoff_sleep(attempt)
            continue
        # non-retryable
        logging.error("Non-retryable error %d: %s %s", resp.status_code, method, url)
        return resp
    raise RuntimeError(f"Max retries exceeded for {url}")

def find_events_for_ip(ip):
    q = quote(f"source.ip:{ip} OR destination.ip:{ip}")
    url = f"{SIEM_BASE}/search?query={q}"
    resp = request_with_retries("GET", url)
    resp.raise_for_status()
    return resp.json().get("events", [])

def tag_event(event_id):
    # Use idempotent PATCH/PUT; API should be designed to add tag if absent
    url = f"{SIEM_BASE}/events/{event_id}/tags"
    resp = request_with_retries("PUT", url, json=TAG_BODY)
    if resp.status_code in (200,201,204):
        logging.info("Tagged event %s", event_id)
    else:
        logging.error("Failed to tag %s status=%d body=%s", event_id, resp.status_code, resp.text)

def main(csv_path):
    with open(csv_path) as f:
        reader = csv.reader(f)
        for row in reader:
            ip = row[0].strip()
            logging.info("Processing IP %s", ip)
            try:
                events = find_events_for_ip(ip)
                for ev in events:
                    event_id = ev["id"]
                    # idempotency: check tags already present to avoid duplicate calls
                    if "threat-intel-hit" in ev.get("tags", []):
                        logging.info("Event %s already tagged", event_id); continue
                    tag_event(event_id)
            except Exception as e:
                logging.exception("Error processing %s: %s", ip, e)

if __name__ == "__main__":
    main("suspicious_ips.csv")

**Why this handles concerns**- Rate limits: Honors Retry-After and exponential backoff.- Transient errors: Retries 5 times for 5xx with backoff.- Idempotency: Uses PUT to add tags and checks existing tags before tagging to avoid duplicates.- Logging/audit: Structured file logs for each IP/event, warnings on rate limits and errors.- Security: API key stored in config/secret manager in real deployment; add batching and parallelism limited by rate budget.

HardSystem Design

61 practiced

Design an automated incident scoring system that computes a severity/business-impact score using telemetry (alerts, asset criticality, user role, threat context). Define the data model, weighting factors, aggregation method, how to handle conflicting or missing signals, and how the resulting score should inform automated vs human workflows.

Sample Answer

**Clarify requirements & goals**Compute a single incident severity/business-impact score from telemetry (alerts, asset criticality, user role, threat context) to drive automated playbooks vs human review. Must be explainable, auditable, and adjustable.

**Data model**- Incident { id, timestamp, alerts: [AlertID], assets: [AssetID], users: [UserID], context: {threat_intel, kill_chain_stage} }- Alert { id, type, confidence, source }- Asset { id, owner, criticality: [1-10], business_service }- User { id, role, privilege_level }- ThreatContext { reputation_score, ioc_match_count, active_campaign_score }

**Weighting factors**Base weight vector (configurable): W_alert_confidence, W_asset_criticality, W_user_privilege, W_threat_context. Example initial values: 0.35, 0.30, 0.20, 0.15.

**Aggregation method**1. Normalize components to 0–100.2. Compute component scores: - Alerts = max(Alert.confidence * alert_type_risk_multiplier) - Asset = mean(asset.criticality) * 10 - User = privilege_level_score - Threat = reputation_score * 1003. Final score = sum( W_i * component_i ) / sum(W_i)Explainability: store contribution breakdown.

**Handle missing/conflicting signals**- Missing: impute conservative defaults (e.g., unknown asset criticality → medium 5) and flag uncertainty.- Conflicting: use provenance and recency; prefer higher-confidence, recent signals. If conflicts remain, increase uncertainty factor and cap automated actions.

**Automation vs human workflow**- Score 0–30: low — auto-close or low-priority automated remediation.- 31–60: medium — automated enrichment (pull logs, run playbook sandbox) + human triage queue.- 61–85: high — immediate human analyst notify + staged automated containment (isolate network segments) with require human approval.- 86–100: critical — auto-containment (if safe), CIRT escalation, executive notification.

**Feedback & governance**- Continuous ML tuning from closed-loop labels (analyst dispositions), periodic review of weights, SLA metrics (false positives/negatives), and audit logging for all automated actions.

Unlock Full Question Bank

Get access to hundreds of Security Incident Response and Operations interview questions and detailed answers.

Join thousands of developers preparing for their dream job.