Focuses on the practical judgment, processes, and technical actions used to respond to active security incidents, contain attacker activity, eradicate threats, remediate affected systems, preserve evidentiary integrity, and restore services with minimal business impact. Coverage includes containment strategies from immediate short term isolation and network segmentation to longer term monitored observation and selective blocking of attacker infrastructure; trade offs between rapid containment that reduces blast radius and slower approaches that preserve forensic visibility to determine attacker objectives and scope; and prioritization of remediation steps such as removing attacker access, eradicating malware, applying patches, closing exploited vulnerabilities, resetting compromised credentials, rebuilding or hardening systems, and validating fixes through testing and monitoring. Also includes recovery procedures such as phased restoration, rollback to known good images, and integration with business continuity plans. Operational topics include defining decision boundaries and escalation paths for analyst actions versus management or change control approvals, assessing business impact and continuity trade offs, coordinating with system administrators, database teams, application owners, legal and business stakeholders, preserving evidence and maintaining chain of custody for forensic analysis, communicating status to stakeholders, and conducting post incident activities including root cause analysis, lessons learned, and updates to runbooks and controls.
MediumSystem Design
51 practiced
Design a monitoring and alerting approach to detect lateral movement in a large cloud environment (multi-account AWS). List data sources, example detection rules or heuristics, thresholds, enrichment fields to include, and how alerts should be triaged and routed to support rapid containment.
Sample Answer
Requirements & constraints:- Detect lateral movement across multi-account AWS quickly (minutes), low false positives, scalable to 100s of accounts, integrations with SIEM/IR playbooks, automated containment optional.High-level architecture:- Central ingest (AWS SQS/Kinesis) → Normalization layer → SIEM/Analytics (Splunk/Elastic/Sumo/Chronicle) + Threat Intel + UEBA → Alerting + Orchestration (PagerDuty/ServiceNow + Lambda/Step Functions for containment).Data sources:- AWS CloudTrail (management/API calls across accounts)- VPC Flow Logs & ENI metadata- AWS GuardDuty, Amazon Macie, AWS Config- IAM credential usage logs (AssumeRole, CreateAccessKey, Console/Login)- CloudWatch Logs (app/system), EKS/k8s audit logs- Host/agent telemetry (OS auth logs, process, netconn) via Fleet (OSQuery, Wazuh)- DNS logs, proxy logs, S3 access logs- Threat intel feeds (IP/ASN/file hashes)Example detection rules / heuristics:- Unusual cross-account AssumeRole: Principal from Account A assumes role in Account B outside baseline (frequency or geography) — alert.- New lateral movement pattern: EC2 instance initiates remote connections to other internal IPs it never contacted before + suspicious process spawn — alert.- Console login + immediate privilege escalation (CreatePolicy/AttachRolePolicy) — high-severity.- Mass credential use: access key used from multiple regions/IPs within short window (impossible travel) — alert.- New user creation followed by policy attachment and resource access within short time — alert.- Repeated failed access to resources across accounts then a success — suspicious brute/credential stuffing.- Abnormal data staging: S3 PUTs to uncommon buckets or new buckets with wide ACLs after internal movement — alert.Thresholds & tuning (starting points):- Cross-account AssumeRole: >1 unusual account in 24h OR geolocation distance >5000 km from prior auth — medium alert.- New internal IP contacts: >10 unique internal endpoints in 5 minutes from single host — high alert.- Privilege escalation within 10 minutes of first login from new source — high/critical.- Access key used from >3 distinct ASN in 1 hour — medium-high.Enrichment fields to include in alerts:- Principal ARN, source account, role/session name, user agent, source IP, ASN, geo, VPC/subnet, instance-id, ENI, process/parent process, destination resource (bucket, host, port), recent API calls, previous baseline behavior score, associated vulnerability tags, relevant threat-intel matches, timestamps, correlation id(s).Alert triage & routing for rapid containment:- Automated severity classification (low/med/high/critical) based on rule confidence + enrichment.- Route critical alerts immediately to on-call (PagerDuty) + create incident in ITSM with enriched context and recommended runbook steps.- Provide automated containment actions for high confidence events: revoke session tokens/assume-role sessions, disable user, detach role policies, quarantine EC2 (move to isolated SG), block IP via WAF/Network ACLs — executed via vetted Lambda/Runbook only after guardrails (multi-signer or approval).- Low/medium go to SOC queue in SIEM with playbook for investigation: gather host snapshots, sudo/ps listings, network captures, timeline of API calls.- Maintain a pre-built incident template including blast radius map (what accounts/resources touched), recommended first 3 actions, and owner.Operational considerations & trade-offs:- Prioritize high-fidelity detections to avoid alert fatigue; use UEBA to raise novelty scores.- Keep retention of raw telemetry sufficient for 30–90 days depending on compliance.- Regularly run red-team exercises and tune thresholds; create labeled incidents to feed ML models.- Balance automation vs human review: automated containment for high-confidence where rollback is trivial; otherwise require analyst approval.Metrics to monitor effectiveness:- Mean time to detect (MTTD), mean time to contain (MTTC), false positive rate, alerts per 1000 hosts, number of automated containments vs rollbacks.This approach combines centralized telemetry, contextual enrichment, behavioral heuristics and safe automation to detect and rapidly contain lateral movement across multi-account AWS.
HardTechnical
51 practiced
You must respond to an incident in a multi-tenant environment where isolating a single tenant's workload could impact paying customers. Create a decision framework to weigh containment options (tenant isolation, throttling, service-level restrictions), legal obligations, SLA considerations, and customer communication. Provide pragmatic procedural steps for executing whichever option you choose.
Sample Answer
Decision framework (prioritize safety, legal, and business impact):1. Clarify facts quickly- Scope: affected tenants, resources, interdependence- Severity: user-visible errors, data integrity, PII exposure, security breach- Velocity: is issue spreading or contained?- Constraints: legal/regulatory holds, existing SLAs/SLOs, maintenance windows2. Decision axes (score 0–5): user impact, revenue impact, legal/regulatory risk, blast radius risk, reversibility, operational complexity. Weight legal/regulatory higher when PII/security involved.3. Options & guiding rules- Tenant isolation (network/namespace/quota): choose when blast radius limited, reversible, and high-risk tenant causes measurable harm. Avoid if isolation will break critical shared services for many customers.- Throttling (rate limits, CPU/memory cgroups): prefer when resource exhaustion is source and graceful degradation is acceptable; low legal risk and reversible.- Service-level restrictions (feature flags, read-only mode): use for data consistency or to prevent state corruption; choose when preserving data integrity outweighs availability.- Emergency global actions (circuit breakers, full service pause): last resort when continued operation risks widespread corruption or legal exposure.4. Legal & SLA checks (must be consulted immediately)- If PII/security breach: invoke legal, compliance, data protection officer; preserve logs/chain-of-custody; do not modify data unless instructed.- Check SLA commitments and error budget: prefer options that minimize SLA breaches for highest-value customers; document any deviation.Procedural execution steps (pragmatic playbook)1. Triage (first 10–15 min)- Run quick diagnostics, gather logs/metrics, identify offending tenant(s)- Notify incident commander, legal/compliance on suspicion of breach2. Containment decision (15–30 min)- Score decision axes, choose least-impactful effective option- If unsure, apply throttling first (least disruptive) while validating isolation feasibility3. Execute containment (30–60 min)- Use automated runbooks: apply network policy to tenant namespace or adjust cgroups/CPU shares; deploy rate-limit configs; flip feature flag to read-only- Perform targeted health checks and smoke tests for major customers4. Communication (immediately and ongoing)- Internal: update incident channel with chosen action, rationale, rollback plan, owner, ETA- External: notify affected tenants (and broader customer base if relevant) with brief status, expected impact, mitigation steps, and escalation contacts; align messages with legal wording if breach- For high-SLA customers, offer dedicated updates and remediation timelines5. Stabilize and remediate (hours)- Monitor KPIs and error budgets; iterate containment if ineffective- Capture forensic data if required; avoid destructive changes6. Restore & verify (after root cause fixed)- Roll back containment in controlled phased manner; monitor for regressions- Validate data integrity and customer experience7. Post-incident- Run postmortem with RCA, impact analysis by tenant, SLA effects, legal follow-up- Update runbooks: automated isolation playbooks, alert tuning, customer communication templates- Propose changes: per-tenant resource limits, better telemetry, feature flags by customer, contractual SL adjustmentsRationale: prefer minimally invasive, reversible controls (throttling) first; escalate to isolation only when necessary. Legal and SLA constraints override pure technical preference. Automate repeatable containment steps and keep communication transparent and timely to preserve customer trust.
MediumTechnical
42 practiced
Explain the role of an error budget and SLOs in incident containment decision-making. Given a service with a tight error budget, how would you incorporate SLO impact when deciding between aggressive isolation actions that reduce blast radius but cause downtime versus more gradual mitigations that preserve availability?
Sample Answer
An SLO (service level objective) defines the acceptable level of service (e.g., 99.95% success over 30 days); the error budget is 1 − SLO and quantifies allowable failure before stakeholders consider the service unacceptable. In incident containment, these two act as decision levers: error budget constrains how much availability/latency you can sacrifice to contain an incident, and SLO impact maps containment actions to business risk.If the service has a tight (nearly exhausted) error budget, incorporate SLO impact like this:- Quantify: estimate how each action affects SLI (% errors, latency) and the time window. e.g., aggressive isolation reduces blast radius immediately but causes 5 minutes of full-service downtime (adds X% error over the SLO window); gradual mitigation keeps availability but risks progressive SLI degradation.- Prioritize safety vs. availability: if the incident risks long-tail or cascading failures that would burn far more of the budget over time, favor aggressive isolation despite short downtime. If the impact is bounded and low, prefer gradual mitigations.- Use a decision matrix: map actions → immediate SLO hit, expected recovery time, risk of cascade, business impact (customer-facing vs internal), and rollback complexity.- Communicate: notify stakeholders and runbook owners, record chosen action and rationale.Example: a control-plane bug causing increased error rate on 10% of requests. Aggressive isolation (take a failing cluster offline) causes 3 minutes of 100% errors for those users (spike in error budget) but prevents propagation to all clusters; gradual throttling might let errors creep to 50% of traffic over hours. Given a tight budget and high cascade risk, isolate. If budget tight but customer-facing SLA is contractual and downtime would breach legal SLO, prefer controlled, targeted mitigation and request immediate escalation.Always re-evaluate after containment: measure real SLI impact, adjust rollbacks, and document for postmortem and future runbook tuning.
HardTechnical
39 practiced
Design a roll-forward remediation approach versus a rollback approach for a complex distributed configuration flaw that was exploited in production. Provide decision criteria for choosing roll-forward vs rollback, testing/canary strategies for roll-forward, feature flags or throttles you would use, and safety nets for both approaches.
Sample Answer
Clarify constraints: impact (customer-visible vs internal), blast radius, time-to-fix, ability to patch config safely, dependency graph, and rollback feasibility (stateful migrations, irreversible side-effects).Decision criteria (choose rollback when):- Exploit requires immediate cessation and patch can’t be safely applied online.- State is corrupted or inconsistent and prior state is known-good and restorable.- Blast radius is small and rollback time < business impact tolerance.Choose roll-forward when:- Fix is patchable without reversing prior state, or rollback risks more disruption (migrated DB schemas, long rollback windows).- You can progressively harden/mitigate via feature flags/throttles and validate in canary.- Exploit is configuration-based and you can change behavior without losing state.Roll-forward testing / canary strategy:- Fast-path canary: route ~1% traffic to patched config in isolated cluster/availability zone with mirrored production data or sampled requests.- Dark-launch requests: mirror live traffic to patched instances (no response to users) to validate behavior and side-effects.- Progressive ramp: 1% → 5% → 25% → 100% with SLO-driven gates (error rate, latency, business metrics).- Automated rollback triggers: if any gate breaches, auto-throttle and revert percentages.Feature flags / throttles:- Kill-switch flag to disable vulnerable feature immediately.- Gradual rollout flag with percentage targeting and user segmentation (internal users first).- Rate-limiting throttle per-tenant and global circuit-breakers.- Scoped flags by AZ/region/service to limit blast radius.Safety nets for roll-forward:- Read-only mode for affected services to prevent writes that corrupt state.- Shadow backups and transactional logging to replay/repair if needed.- Health checks, synthetic transactions, and alerting tied to automated rollback.- Post-change audit trail and tracing to trace exploit vectors.Safety nets for rollback:- Pre-tested rollback scripts (idempotent), database backups and point-in-time recovery, and migration reversal procedures.- Deployment freeze windows and coordination plan for dependent teams.- Chaos-safe rollback rehearsals and runbooks with checklist-driven communications.Communication & governance:- Incident commander decides approach based on metrics and risk with Exec/PM alignment.- Timeboxed decisions (e.g., try roll-forward for X minutes then fallback to rollback).- Post-incident: Root cause, drift prevention (config as code, signing), and automated canary gates added to pipeline.
HardTechnical
38 practiced
A forensic image reveals attacker-modified binaries on many hosts. How would you determine whether monitoring/alerting data (agents, collectors) were compromised as well? Propose steps to verify alert integrity, collect independent telemetry, and rebuild trust in detection pipelines without losing critical evidence.
Sample Answer
Start by assuming monitoring/alerting is untrusted until proven otherwise. Work in three parallel tracks: verify alert integrity, collect independent telemetry, and rebuild trust while preserving evidence.1) Verify alert integrity- Preserve chain of custody: snapshot affected hosts and collectors (read-only forensic images), record hashes (SHA256), timestamps, and who accessed them.- Compare alert logs to immutable originals: check whether agent and collector binaries, configs, and alert records are cryptographically signed or timestamped. Validate signatures against known-good keys.- Look for tampering indicators: modified file mtime/ctime incongruities, unexpected process ancestry, altered log rotation, gaps in logs, or log entries with impossible timestamps.- Cross-check alert pipeline configuration (rules, filters, destinations) stored in source control or CI artifacts. Diff deployed configs against git-tagged baseline.2) Collect independent telemetry (avoid trusting the compromised path)- Deploy or enable out-of-band collectors from a clean build: use a hardened, minimal read-only image (air-gapped if possible) or live-boot rescue media to collect volatility memory, process lists, kernel modules, network connections, and disk metadata.- Ingest network-level telemetry: capture packet-level data from network taps/SPAN, firewall logs, DNS logs, and upstream cloud provider logs (flow logs, API audit logs) that agents can’t easily modify.- Pull host-inspector data via out-of-band channels: orchestration-plane APIs (cloud provider metadata, hypervisor consoles) and isolated management network to retrieve console logs, snapshots, and serial output.- Centralize copies in immutable storage (WORM/S3 with object lock) to prevent tampering.3) Rebuild trust in detection pipelines without destroying evidence- Quarantine compromised systems; do not reinstall monitoring on them until rebuilt from trusted images. Continue monitoring via independent collectors and network sensors to avoid visibility gaps.- Build new agent/collector images from reproducible builds in an isolated CI environment using signed artifacts. Rotate keys and credentials used by agents and pipelines.- Validate new pipeline by seeding known benign and malicious test events (canary alerts) and verifying end-to-end delivery and signature validation.- Apply layered detection: combine host-based, network-based, and cloud-provider telemetry; correlate independent sources to reduce single-point-of-failure trust.- Document every action in an immutable incident log and maintain forensic copies of original artifacts for investigators and legal needs.Practical checks and tools- Use SHA256, GPG signatures, and binary provenance (package manager metadata), compare with baseline hashes.- Use network pcap from taps, Zeek/Suricata logs, cloud audit logs, and SIEM retained raw events.- Run integrity tools from known-good media: AIDE/OSSEC in read-only mode, Volatility for memory.Trade-offs and priorities- Preserve evidence (forensic images + hashes) before active remediation.- Balance speed vs. completeness: bring independent visibility online quickly (network taps, cloud logs) while preparing full forensic captures.- Rebuild hosts from trusted images rather than attempting in-place remediation for compromised binaries.Outcome goal- Restore multi-source visibility and an attestable detection pipeline: signed artifacts, rotated credentials, independent telemetry correlation, and reproducible builds—so alerts can be trusted and future compromises detected earlier.
Unlock Full Question Bank
Get access to hundreds of Incident Containment and Remediation interview questions and detailed answers.
