Incident Response and Containment Questions
Managing security incidents from detection through recovery. Covers incident response process and playbooks, containment and remediation, data-breach investigation methodology, data-exfiltration detection and analysis, root-cause and post-incident analysis, and fraud and complex-attack investigation. The operational 'a compromise is happening, now what' discipline, distinct from broader production-outage incident management.
You are the lead responder for a multi-stage compromise: initial access via phishing or a public exploit, privilege escalation, lateral movement using living-off-the-land binaries or custom loaders, and staged data exfiltration. Draft a comprehensive containment, eradication, and recovery plan: prioritized containment options and their trade-offs, techniques to scope which hosts and accounts are affected, removal of cross-platform persistence, and validation that the attacker cannot re-establish access before declaring the incident closed.
Sample Answer
Direct answer
Contain based on where the attacker currently has the most leverage while scoping the full chain of hosts and accounts touched, remove every persistence mechanism across every affected platform (not just the one you found first), and only declare the incident closed once you've validated the attacker genuinely cannot re-establish access.
Structured elaboration
Prioritized containment with trade-offs. Early containment options include isolating the initial-access host, disabling the escalated-privilege account, or blocking the exfiltration destination; each has a different cost and speed. Disabling the compromised privileged account is usually fast and high-value since it removes the attacker's current leverage across everything that account can reach, while isolating individual hosts one at a time is slower and risks leaving other, undiscovered footholds active while you work through the list.
Scoping which hosts and accounts are affected. Living-off-the-land techniques (using built-in, legitimate system tools rather than custom malware) are specifically designed to blend in, so scoping requires correlating multiple weak signals rather than looking for one obvious indicator: unusual process parent-child relationships, authentication logs showing the escalated account's activity across hosts it doesn't normally touch, and network connections inconsistent with that host's normal baseline. Where lateral movement specifically uses SMB (moving between hosts via file-sharing protocols), look for unusual SMB session patterns between hosts that don't normally communicate directly.
Structured impact analysis. Work through initial-access vector (how did they get in), propagation mechanism (how did they move from there to everywhere else), and ultimate impact (what did they actually reach or take), in that order, since understanding propagation tells you where else to look for persistence, and understanding impact tells you what needs remediation versus what was merely reachable but untouched.
Removing cross-platform persistence. Living-off-the-land and custom-loader techniques often plant persistence across multiple mechanisms and platforms (a scheduled task on one host, a modified startup script on another, a backdoored account with unusual privileges), so eradication needs a checklist covering every platform in scope, not just the specific technique first discovered.
Validation before declaring closure. Confirm the attacker cannot re-establish access by checking that every credential they touched has been rotated, every persistence mechanism found has been removed and re-checked, and monitoring shows no further activity matching the original indicators over a meaningful observation window, not just the first quiet hour.
Worked example
An intrusion begins with a phishing email leading to initial access, followed by privilege escalation to a domain-level service account, lateral movement across a dozen hosts using SMB and a custom loader disguised as a legitimate process, and staged exfiltration to an external cloud bucket. Containment: the escalated service account is disabled first (fastest, highest-leverage action), followed by isolating the specific hosts showing the custom loader's process signature. Scoping: correlating authentication logs for the escalated account against SMB session logs reveals the full set of twelve affected hosts, more than the four initially suspected from EDR alerts alone. Persistence removal: the team finds scheduled tasks on Windows hosts and a modified cron entry on the two Linux hosts in scope, both removed and validated absent. Closure: the incident is only declared resolved after two weeks of clean monitoring specifically watching for the loader's known signature and the escalated account's original access pattern, not just the first 48 hours.
Trade-offs and pitfalls
Declaring victory after finding and removing the first persistence mechanism, without checking whether the attacker planted a second one on a different platform as a fallback, is a common and costly gap in multi-stage intrusions; sophisticated attackers frequently plant redundant persistence specifically anticipating that defenders will find and remove the first one. Underinvesting in the propagation-mechanism analysis (jumping straight from initial access to impact) also risks missing hosts the attacker reached that show no obvious damage yet but still carry a live persistence mechanism.
You confirm a Pass-the-Hash, Golden Ticket, or Kerberoasting attack in an Active Directory environment (forged Kerberos tickets granting persistent domain access). Outline detection and validation, containment of active misuse, remediation including the KRBTGT account reset and enterprise-wide credential rotation, replication considerations, and how you validate that forged tickets can no longer be reused before restoring trust.
Sample Answer
Direct answer
First confirm which technique is actually in play, since they aren't interchangeable: Golden Ticket and Kerberoasting revolve around forged or cracked Kerberos material, while Pass-the-Hash reuses a stolen NTLM password hash directly and involves no forged Kerberos ticket at all. Contain by identifying and disabling the misused accounts (and, for ticket-based attacks, the tickets themselves), then remediate: a full KRBTGT account reset (done twice, spaced apart, due to how ticket-granting-ticket trust works) addresses Golden Ticket forgery specifically, while Pass-the-Hash is remediated by rotating the password of every account whose hash was stolen, which invalidates that hash, not by touching KRBTGT at all. Either way, follow with enterprise-wide credential rotation and validate no forged ticket or stolen hash can still be used before declaring the domain trustworthy again.
Structured elaboration
Detection and validation. Golden Ticket and Pass-the-Ticket attacks typically show up as: Kerberos tickets with unusually long or non-standard lifetimes, ticket-granting-ticket requests for accounts that show no corresponding initial authentication event, and encryption types on ticket requests inconsistent with the domain's configured policy. Kerberoasting shows up as a spike in ticket-granting-service requests for accounts with servicePrincipalNames, often followed by offline password-cracking activity you won't see directly but can infer from the request pattern. Pass-the-Hash is a different mechanism entirely: no Kerberos ticket is forged, the attacker authenticates directly using a stolen NTLM password hash without ever needing to crack it. It shows up as NTLM (not Kerberos) authentication events for a privileged account from a host that account doesn't normally log into, often in rapid succession across several machines in a short window, the hallmark of an attacker relaying the same stolen hash to move laterally, and it especially targets environments where the same local-administrator password hash is reused across many machines.
Containment of active misuse. Identify which accounts and tickets are actually being misused right now (not just theoretically vulnerable) using the indicators above, and immediately disable or force sign-out on any account showing confirmed forged-ticket usage. Do this without tipping off the attacker that you know, if possible, since revealing detection to an attacker holding a Golden Ticket can prompt them to escalate or destroy evidence before you've fully scoped the compromise.
Remediation: the KRBTGT reset. A Golden Ticket is forged using the KRBTGT account's password hash, so resetting that single account's password invalidates every ticket forged with the old hash. Because Active Directory keeps both the current and previous KRBTGT password hash valid (to avoid breaking tickets issued moments before a reset), you must reset it twice, spaced apart (commonly recommended as at least several hours, ideally 24, to let replication and the natural expiry of legitimately issued tickets play out), to actually invalidate everything including tickets forged just before your first reset. This KRBTGT-specific remediation is for Golden Ticket forgery; it does nothing for a Pass-the-Hash compromise, since no Kerberos ticket or KRBTGT material was involved there. For Pass-the-Hash, remediation is simpler but still urgent: rotate the password of every account whose hash was stolen or plausibly reused, which immediately invalidates that hash for future reuse, and where the attack exploited a shared local-administrator password across many hosts, randomize local admin credentials per host going forward (for example via Microsoft's LAPS) so a single stolen hash can no longer unlock the whole fleet.
Enterprise-wide credential rotation. Alongside the KRBTGT reset, rotate credentials for any account confirmed or plausibly compromised during the intrusion, particularly privileged and service accounts, since the attacker may have harvested additional credentials beyond what the forged tickets alone represent.
Replication considerations. The KRBTGT reset must replicate to every domain controller before you can be confident the old hash is fully invalidated everywhere; verify replication status across all DCs rather than assuming a change made on one has already propagated, especially in environments with multiple sites or slow replication links.
Validation before restoring trust. Confirm no forged tickets are still being accepted anywhere by monitoring authentication logs across all domain controllers for any Kerberos activity using the old KRBTGT hash's expected patterns, and only then consider the domain's trust fully restored.
Worked example
Security monitoring flags an unusual Kerberos ticket with a ten-year lifetime granted to a low-privilege service account, which then authenticates as a domain admin, a classic Golden Ticket signature. Containment: the account showing the forged ticket's activity is immediately disabled, and the team scopes which systems that account touched in the last 24 hours using authentication logs. Remediation: the KRBTGT password is reset once immediately, and a second reset is scheduled 24 hours later after confirming replication across all six domain controllers completed following the first reset. In parallel, every privileged account is rotated, since a Golden Ticket attack implies the attacker already had domain-admin-level access at some point to obtain the KRBTGT hash in the first place. Validation: authentication logs across all DCs are monitored for the following week for any sign of ticket activity inconsistent with the new KRBTGT hash, and the incident is only closed once none appears.
Trade-offs and pitfalls
The most common and costly mistake is resetting KRBTGT only once and declaring victory, which leaves a window where tickets forged just before the reset remain valid until the second reset actually happens; skipping the second reset (often because teams don't realize it's necessary, or fear disruption) leaves the domain still compromised in a way that looks resolved. A second pitfall is underestimating scope: since obtaining the KRBTGT hash typically requires the attacker to have already had domain-admin-equivalent access, treating this as an isolated account compromise rather than a full domain compromise understates what needs to be rotated and validated.
During an authorized penetration test, you unexpectedly discover evidence of an active, unrelated compromise by a real attacker. What are your immediate obligations: how do you preserve evidence, what are your legal and ethical responsibilities given your engagement scope, how and when do you notify the client, and how do you coordinate with their incident response team without compromising either the finding or your own engagement's integrity?
Sample Answer
Direct answer
Preserve evidence of what you found without taking any action beyond what your engagement scope authorizes, notify the client immediately through the pre-agreed escalation channel rather than waiting until your engagement's normal reporting cadence, and coordinate directly with their incident response team while being explicit about where your authorized engagement ends and their own response begins.
Structured reasoning
Immediate obligations. The moment you find evidence of a real, unrelated compromise, your engagement's rules of engagement almost certainly don't authorize you to actively respond to it (contain it, remove it, or investigate it further), since that's outside the scope the client agreed to; your obligation shifts to preserving what you've already observed and notifying the client promptly, not continuing to dig deeper into territory you weren't authorized to test.
Legal and ethical responsibilities. Your contract likely has specific language about discovering unrelated findings during an engagement; even without explicit contract language anticipating this exact scenario, the ethical baseline is clear: you don't have standing authorization to act on this finding technically, and continuing to investigate beyond noting what you've already seen risks exceeding your actual authorization, however well-intentioned.
Notification timing and channel. This warrants immediate, out-of-band notification, not a note buried in your final engagement report delivered weeks later; contact the client's designated security or incident contact right away, through whatever emergency escalation path your engagement agreement specifies (or, if none exists, the most direct available channel), since every hour of delay is an hour the client's real incident continues unaddressed.
Coordinating with the client's IR team. Share exactly what you observed, how, and when, being precise about the boundary between your authorized testing activity and the newly-discovered, unrelated finding, so their team can distinguish your legitimate test traffic from the actual attacker's activity during their own investigation; offer to pause your engagement's active testing if it risks confusing their incident response, since continuing normal test activity during their live investigation could contaminate their evidence or trigger false alarms.
Worked example
During an authorized web-application penetration test, the tester discovers an active webshell on a server unrelated to the specific application in scope, clearly indicating a real, ongoing compromise by someone other than the testing team. The tester immediately stops testing activity on that specific server (outside engagement scope to investigate further) and documents exactly what was observed: the file path, timestamp, and how it was discovered. Within the hour, the tester contacts the client's designated security contact directly, rather than waiting for the engagement's weekly status update, explaining the finding and offering to pause broader testing activity if it would help the client's incident response team distinguish real attacker activity from the ongoing authorized test. The client's IR team takes over the technical investigation, and the pentest engagement's own final report separately documents the discovery and notification timeline as a distinct, time-stamped finding.
Trade-offs and pitfalls
Continuing to investigate the unrelated compromise out of professional curiosity or a desire to be more helpful, without explicit authorization to do so, risks exceeding your engagement's legal scope and potentially contaminating evidence the client's own IR team will need. Waiting to report the finding until your engagement's normal reporting cadence, rather than escalating immediately, is the single most damaging mistake here, since it needlessly extends the client's exposure to a real, active attacker.
You receive a high-severity alert (for example: a spike of failed logins followed by a successful admin login, or an encoded PowerShell command on a production host) indicating possible lateral movement or credential compromise. Within the first 15 to 30 minutes, walk through your triage: which logs and telemetry you check first and in what order, what you capture as evidence, initial containment actions you take, and which teams you notify.
Sample Answer
Direct answer
In the first 15 to 30 minutes, the priority is confirming scope and taking evidence-preserving containment action, in that order: check identity and endpoint telemetry first, capture what you see before it disappears, then contain based on confidence, and notify as soon as you have enough signal to say something useful.
Structured elaboration
Order of investigation for a credential-compromise or lateral-movement alert:
- Identity signals first. Check the authentication logs for the account in question: source IP, geolocation, MFA status, time of day relative to the user's normal pattern, and whether the "successful admin login" following failed attempts is consistent with a real user (travel, new device) or clearly anomalous.
- Endpoint telemetry second. Pull EDR data for any host the account touched around the alert window: running processes, especially anything matching the suspicious pattern (an encoded PowerShell command, an unusual parent-child process relationship), and any outbound network connections from that host.
- Network telemetry third. Check for lateral movement signals from the account or host: unusual SMB traffic, new connections to other internal hosts, or anything reaching out to an external IP with no legitimate business reason.
What to capture as evidence, before anything else changes: a snapshot of the current process list and network connections on any implicated host, the raw authentication log entries (not just a summary), and a copy of the specific alert with its full context. Do this before taking any containment action that might cause the process or connection to disappear.
Initial containment actions, roughly in order of aggressiveness: disable or force a password reset on the account if compromise looks credible; isolate the specific host at the EDR or network layer if there's endpoint-level evidence of compromise, not just an identity anomaly; and if lateral movement across multiple hosts is confirmed, consider a broader network segmentation action rather than isolating one host at a time.
Who to notify within this window: your incident lead or on-call security manager immediately once you've confirmed this is a real incident (not a false positive), and the system owner of any affected host so they're aware before you take containment action that might affect their service, unless the risk of tipping off an insider is a specific concern.
Worked example
An alert shows ten failed logins on an admin account, followed by a successful login from an unfamiliar country, followed by an EDR alert for a suspicious process on a file server that same account accessed. In order: pull the raw authentication log entries and confirm the geolocation and device fingerprint don't match the user's normal pattern (rules out "they're just traveling"); pull the EDR process tree on the file server and find the suspicious process is an encoded PowerShell command spawning a network connection to an unfamiliar external IP; snapshot the process list and network connections before doing anything else. Given both identity and endpoint evidence now corroborate each other, disable the compromised account immediately (low business cost, high containment value) and isolate the file server at the network layer while notifying the incident lead and the file server's system owner, all within the first 20 minutes.
Trade-offs and pitfalls
The most common mistake under time pressure is jumping straight to containment before confirming the alert is real, which causes unnecessary business disruption on a false positive; the opposite mistake, spending too long gathering evidence before containing a clearly credible compromise, gives the attacker more time to cause damage. The order above (identity, then endpoint, then network, capture-before-contain) is designed to get you to a confident containment decision as fast as possible without either extreme. The exact same triage process applies even when the alert source turns out to be a misconfiguration (an admin endpoint accidentally exposed to the internet) rather than an active attacker; the difference is urgency and communication tone, not method, since you don't yet know which one it is when you start.
That is every published Incident Response and Containment question for Penetration Tester so far. Browse the other topics in this category, or practice this one interactively.