Incident Response and Containment Questions

Direct answer

Scope which builds and services actually consumed the compromised artifact using build metadata and an SBOM if you have one, revoke and rotate build credentials immediately, rebuild from a verified-clean state rather than trusting the existing artifacts, and coordinate disclosure to anyone downstream who may have consumed what you shipped.

Structured elaboration

Scoping affected builds and services. Use build logs, artifact hashes, and (if available) a software bill of materials to determine exactly which builds included the compromised dependency or were produced by the compromised pipeline stage, rather than assuming only the specifically-flagged build is affected; a compromised shared dependency or build agent can silently affect every build that touched it during the compromise window.

Revoking and rotating build credentials. Any credential the compromised pipeline stage had access to (signing keys, registry push credentials, cloud deployment credentials) should be treated as potentially exposed and rotated, since a compromised CI/CD stage is a high-value target specifically because of what it's trusted to do.

Rebuilding from a trusted state. Don't simply redeploy the existing, potentially-compromised artifacts; rebuild from source using a verified-clean pipeline, re-signing and re-verifying integrity (checksums, signatures) before those artifacts go anywhere near production.

Coordinating disclosure. If downstream teams, other services, or external customers consumed the compromised artifact, they need to know, including what specifically was affected and what action they should take (redeploy from the corrected build, rotate their own credentials if they trusted something signed by your compromised keys); this needs to happen promptly even though it's an uncomfortable conversation, since delaying it only extends how long the compromised artifact stays trusted elsewhere.

Where the initial finding is itself a forensic investigation into the build system (rather than an obvious, already-confirmed compromise), scoping needs to happen carefully and in parallel with containment: preserve build logs and pipeline state before making changes that might overwrite the evidence of exactly how the compromise happened, while still moving quickly on credential rotation given the stakes. Where an SBOM already exists, use it to quickly answer "which of our services include this dependency" rather than manually auditing every service's dependency tree from scratch, which is dramatically slower at any meaningful scale. Where the root cause traces back to committed credentials in a public repository (rather than a compromised third-party dependency), the same rotate-and-rebuild pattern applies, but with an added step: audit the repository's commit history for how long the credential was exposed and whether it shows signs of having actually been used by someone outside your organization.

Worked example

A dependency used across a dozen internal services is found to have been compromised, injecting malicious code into any build that included it during a specific two-week window. Using the organization's SBOM, the team quickly identifies exactly seven of the twelve services that included the affected version during that window, rather than needing to manually inspect all twelve. Build credentials the pipeline used during that window (signing keys, registry push tokens) are rotated immediately. All seven affected services are rebuilt from source using the corrected dependency version, re-signed, and redeployed, while the five unaffected services are confirmed clean and left alone rather than unnecessarily rebuilt. Two of the affected services are consumed by an external partner, who is notified with specifics on what was affected and what artifact version they should now trust.

Trade-offs and pitfalls

Redeploying the existing, already-built artifacts after just rotating a credential (rather than rebuilding from a verified-clean source) is a dangerous shortcut, since the artifact itself, not just the credential, may carry the injected compromise. Delaying downstream disclosure to avoid an uncomfortable conversation or reputational risk is a common but costly mistake, since every day the compromised artifact remains trusted elsewhere is a day the actual harm can continue growing.

Direct answer

Sequence the response to stop the attacker fastest with the least disruption: revoke active sessions everywhere first, then decide between an immediate organization-wide reset versus targeted resets based on how confidently you've scoped which accounts are actually compromised, enforce MFA for affected accounts, and keep hunting for accounts you haven't found yet.

Structured elaboration

Session revocation first. Revoking active sessions for all confirmed and suspected-compromised accounts is usually the fastest, lowest-disruption first step: it immediately cuts off an attacker's current access without requiring the user to do anything, unlike a password reset which requires the legitimate user to go through a recovery flow.

Targeted versus organization-wide reset. If you can confidently scope exactly which accounts were compromised (for example, you have a clean list of who clicked a specific phishing link, or a clear list of accounts hit by credential-stuffing based on login patterns), a targeted reset limits disruption to only those users. If the scope is uncertain, for example the phishing campaign's reach isn't fully known or the credential-stuffing attempt's success rate against reused passwords is unclear, an organization-wide reset trades broader disruption for confidence that you haven't missed anyone.

MFA enforcement. For affected accounts specifically, and ideally as a broader policy improvement coming out of this incident, enforcing MFA closes the specific gap that let a stolen password alone grant access, and should happen alongside or immediately after the reset, not as a separate later project.

Hunting for accounts you haven't found yet. Don't stop once you've handled the accounts you already know about; use the confirmed indicators (specific source IPs, timing patterns, the phishing campaign's known targets) to search more broadly for accounts showing the same pattern that weren't yet flagged.

Sequencing to limit both attacker access and user disruption. Revoke sessions immediately (low disruption, high value), decide on reset scope based on confidence (targeted if scoped well, broad if not), and roll out MFA enforcement in the same action as the reset so users aren't asked to change their password twice in one week.

For detecting credential stuffing specifically, a useful concrete signal: source IPs attempting logins across an unusually large number of distinct usernames within a short window (for example, more than ten distinct usernames from one IP in ten minutes), which is a pattern normal user behavior essentially never produces and automated credential-stuffing tools do reliably.

Worked example

A phishing campaign harvests credentials from an estimated 40 employees, but the exact list isn't fully confirmed since some employees may have entered credentials without reporting it. Given the uncertainty in scope, the team revokes active sessions organization-wide immediately (low cost, closes the door fast) and opts for an organization-wide password reset rather than a targeted one, given the risk of missing an unreported victim. MFA enforcement is rolled into the same reset flow so affected users only go through one disruptive step. In parallel, the security team searches authentication logs for any login pattern matching the phishing campaign's known timing and source IPs that wasn't already on the confirmed-victim list, finding three additional affected accounts.

Trade-offs and pitfalls

Choosing a targeted reset when your scoping confidence is actually low is the most common mistake, since it leaves compromised accounts you haven't identified yet fully accessible; the cost of a broader reset (user friction, help-desk load) is usually worth paying when scoping confidence isn't high. A second common gap is treating MFA enforcement as a follow-up project rather than bundling it into the immediate response, which leaves the same gap open for a repeat attack using freshly reset but still MFA-less credentials.

Direct answer

In the first 15 to 30 minutes, the priority is confirming scope and taking evidence-preserving containment action, in that order: check identity and endpoint telemetry first, capture what you see before it disappears, then contain based on confidence, and notify as soon as you have enough signal to say something useful.

Structured elaboration

Order of investigation for a credential-compromise or lateral-movement alert:

1. Identity signals first. Check the authentication logs for the account in question: source IP, geolocation, MFA status, time of day relative to the user's normal pattern, and whether the "successful admin login" following failed attempts is consistent with a real user (travel, new device) or clearly anomalous.
2. Endpoint telemetry second. Pull EDR data for any host the account touched around the alert window: running processes, especially anything matching the suspicious pattern (an encoded PowerShell command, an unusual parent-child process relationship), and any outbound network connections from that host.
3. Network telemetry third. Check for lateral movement signals from the account or host: unusual SMB traffic, new connections to other internal hosts, or anything reaching out to an external IP with no legitimate business reason.

What to capture as evidence, before anything else changes: a snapshot of the current process list and network connections on any implicated host, the raw authentication log entries (not just a summary), and a copy of the specific alert with its full context. Do this before taking any containment action that might cause the process or connection to disappear.

Initial containment actions, roughly in order of aggressiveness: disable or force a password reset on the account if compromise looks credible; isolate the specific host at the EDR or network layer if there's endpoint-level evidence of compromise, not just an identity anomaly; and if lateral movement across multiple hosts is confirmed, consider a broader network segmentation action rather than isolating one host at a time.

Who to notify within this window: your incident lead or on-call security manager immediately once you've confirmed this is a real incident (not a false positive), and the system owner of any affected host so they're aware before you take containment action that might affect their service, unless the risk of tipping off an insider is a specific concern.

Worked example

An alert shows ten failed logins on an admin account, followed by a successful login from an unfamiliar country, followed by an EDR alert for a suspicious process on a file server that same account accessed. In order: pull the raw authentication log entries and confirm the geolocation and device fingerprint don't match the user's normal pattern (rules out "they're just traveling"); pull the EDR process tree on the file server and find the suspicious process is an encoded PowerShell command spawning a network connection to an unfamiliar external IP; snapshot the process list and network connections before doing anything else. Given both identity and endpoint evidence now corroborate each other, disable the compromised account immediately (low business cost, high containment value) and isolate the file server at the network layer while notifying the incident lead and the file server's system owner, all within the first 20 minutes.

Trade-offs and pitfalls

The most common mistake under time pressure is jumping straight to containment before confirming the alert is real, which causes unnecessary business disruption on a false positive; the opposite mistake, spending too long gathering evidence before containing a clearly credible compromise, gives the attacker more time to cause damage. The order above (identity, then endpoint, then network, capture-before-contain) is designed to get you to a confident containment decision as fast as possible without either extreme. The exact same triage process applies even when the alert source turns out to be a misconfiguration (an admin endpoint accidentally exposed to the internet) rather than an active attacker; the difference is urgency and communication tone, not method, since you don't yet know which one it is when you start.

Direct answer

Restore in phases, validating at each step before moving to the next, with the stateful tier (a database) handled far more carefully than the stateless tier given the risk of data loss or corruption; prefer roll-forward over rollback when the exploited flaw was a configuration issue that a forward fix addresses more cleanly than reverting.

Structured elaboration

Phases of restoration. Bring back the most isolated, least-risk components first (internal tooling, non-customer-facing services) to validate the overall restoration process works before touching customer-facing or data-critical systems; then restore the stateless application tier, which can typically be redeployed fresh with minimal risk since it holds no persistent state of its own; and handle the stateful tier last and most carefully, since restoring it incorrectly (from a stale backup, or without reconciling any writes that happened during the incident) can cause data loss or inconsistency that's much harder to reverse than a stateless service issue.

Validation checks at each phase. Confirm the restored component is behaving correctly (health checks, smoke tests) before routing real traffic to it, and specifically for the stateful tier, verify data consistency and integrity against expectations (row counts, checksums, or application-level consistency checks) before considering it fully restored, not just that the database process is running.

Rollback criteria. Define in advance what would trigger backing out of a restoration phase, for example error rates exceeding a threshold or a data-consistency check failing, so the team has a pre-agreed trigger rather than debating it in the moment under pressure.

Special considerations for the stateful tier. Unlike a stateless service, you generally can't just redeploy a clean copy and move on; you need to reconcile what data existed before the incident, what (if anything) changed during it, and whether any legitimate writes happened during the containment window that need to be preserved or replayed, which is a fundamentally harder problem than restoring a stateless tier.

Roll-forward versus rollback for an exploited configuration flaw. Rollback (reverting to the previous configuration) is appropriate when the previous state was known-good and simple to restore; roll-forward (deploying a corrected configuration rather than reverting) is preferable when the flaw was introduced further back than your easy rollback point, or when reverting would also undo legitimate, wanted changes made since the flawed configuration was introduced. Roll-forward requires more confidence in the fix (since you're moving to a new state rather than a previously-proven one), so it typically warrants canary testing the corrected configuration on a small slice of traffic before applying it fleet-wide.

Worked example

After containing an incident where an exploited configuration flaw in a load balancer allowed unauthorized access, the team restores in phases: internal monitoring tooling first (low risk, validates the general restoration process works), then the stateless API tier (redeployed fresh from a known-good build, validated with smoke tests before receiving real traffic), and finally the primary database, where the team specifically verifies no unauthorized writes occurred during the compromise window using audit logs before considering the data tier restored. For the load-balancer configuration flaw itself, since the flawed setting was introduced several deployments ago and several legitimate configuration changes have happened since (which a simple rollback would also undo), the team chooses to roll forward with a corrected configuration, canary-testing it on 5% of traffic for 30 minutes before applying it fleet-wide.

Trade-offs and pitfalls

Restoring the stateful tier with the same speed and confidence as a stateless service is the most common and costly mistake, since a database restored incorrectly can silently lose or corrupt data in a way that's far harder to detect and reverse than a misbehaving stateless service that a health check would catch immediately. Choosing rollback reflexively, without checking whether legitimate changes since the flawed state would also be lost, can create a second, self-inflicted problem on top of the original incident.

Direct answer

At this scale, fast detection needs automated quality and safety monitoring rather than relying on user reports, mitigation needs pre-built rollback and fallback paths that can trigger without human intervention for the fastest-moving failure modes, and the audit trail needs to capture enough of the model's actual inputs and outputs to support a real regulatory review, not just aggregate metrics.

Structured elaboration

Fast detection. At 100 million inferences a day, waiting for user complaints is far too slow; automated monitoring needs to track output-quality signals (confidence distributions, refusal rates, safety-classifier flags) in near real time and alert on statistically significant deviations from baseline, ideally within minutes rather than hours.

Automated mitigations. Pre-built rollback to a previous known-good model version, or fallback to a simpler, more conservative model or a rule-based response for the specific failure pattern detected, should be triggerable automatically for well-understood failure signatures, with human review for anything novel or ambiguous, mirroring the same auto-versus-human-gate logic used in traditional security automation: reversible, well-understood actions can run automatically, while anything with broader impact or uncertainty escalates to a human.

Forensic data capture with cross-region replication. Capturing prompts, retrieval context, and outputs at this volume requires careful engineering (sampling strategies for routine traffic, full capture triggered around detected anomalies) balanced against storage cost and privacy considerations; cross-region replication ensures this data survives a regional outage or failover event, which matters both for continuity of investigation and because a serious incident and an infrastructure failure can plausibly coincide.

Immutable audit trail for regulatory review. The audit trail needs to demonstrate, after the fact, exactly what the model was given, what it produced, and what automated or human decisions were made in response, stored in a way that can't be quietly altered after the fact (append-only storage, cryptographic integrity checks), since regulatory review of an AI system's behavior increasingly expects this level of evidentiary rigor rather than aggregate dashboards alone.

Key trade-offs. Full capture of every prompt and output at this scale is expensive and raises its own privacy considerations, so most designs sample routine traffic while capturing fully around detected anomalies; automated rollback reduces response latency dramatically but requires very high confidence in the detection signal to avoid false-positive rollbacks disrupting service unnecessarily; and cross-region replication of potentially sensitive prompt data needs to respect data-residency requirements, which can conflict with a simple "replicate everywhere" approach.

flowchart TD
    A[Inference requests] --> B[Model serving, multi-region]
    B --> C[Safety and quality classifiers]
    C --> D{Anomaly detected?}
    D -->|no| E[Sampled prompt/output logging]
    D -->|yes, known pattern| F[Automatic rollback or fallback model]
    D -->|yes, novel pattern| G[Human review]
    E --> H[Immutable audit store, cross-region replicated]
    F --> H
    G --> H
    F --> I[Serving resumes on prior known-good model]

Worked example

A large-scale LLM-serving platform detects a sudden spike in a safety classifier's flag rate for one specific category of harmful output, concentrated on requests routed through one particular model version deployed an hour earlier. Automated detection catches the deviation within minutes based on the classifier-flag baseline. Given this matches a pre-defined, well-understood failure signature (a specific classifier threshold breach tied to a recent deployment), the system automatically rolls back that model version to the prior one without waiting for human approval, while flagging the incident for review. Prompt and output logs around the anomaly window are captured in full (rather than the standard sampled rate) and replicated across regions, providing the detailed evidentiary basis needed for the post-incident review and any subsequent regulatory inquiry into what specifically went wrong.

Trade-offs and pitfalls

Relying on user reports or manual dashboard review as the primary detection mechanism at this scale means real harm accumulates for far longer than automated monitoring would allow; the investment in automated, near-real-time quality and safety signals is what actually makes fast detection possible at 100 million inferences a day. Over-broad automatic rollback, triggered on a detection signal that isn't actually reliable enough, is the main risk on the mitigation side, and requires the same discipline around confidence thresholds and blast-radius awareness used in any automated containment system.