Secure Software Delivery: DevSecOps, Pipeline, and Supply Chain Security Questions

A supply chain attack targets the trust an organization places in something it did not write itself, meaning attackers get their code running inside a target not by breaching the target directly, but by compromising something upstream the target already trusts and pulls in automatically.

How attackers compromise the chain

Attackers commonly target three points: a dependency (publishing a malicious package, or compromising a legitimate maintainer's account to push a malicious update to an already-trusted package), the CI/CD pipeline itself (compromising the build system so the backdoor is inserted during compilation, invisible in the source repository), or a vendor's software update mechanism (compromising a trusted vendor's own build or distribution infrastructure so their legitimate customers receive a malicious update signed with the vendor's own real credentials).

Two historical examples

The SolarWinds SUNBURST attack (2020) is the canonical build-system-compromise example: attackers compromised SolarWinds' own build environment and inserted a backdoor (via malware named SUNSPOT) into the Orion software's build process itself, so the resulting update, signed with SolarWinds' own legitimate certificate, was silently backdoored and distributed to roughly 18,000 organizations who trusted the update as coming from their vendor. The tj-actions/changed-files compromise (2025) is a more recent example targeting the CI/CD supply chain specifically: a popular, widely-used GitHub Action was compromised, leaking secrets from over 23,000 repositories that used it in their workflows, illustrating how a single compromised third-party CI dependency can cascade across a huge number of otherwise-unrelated organizations.

Indicators of compromise to hunt for

Unexpected changes to a build's dependency lockfile with no corresponding pull request explaining why, a dependency's published version jumping in a way inconsistent with its normal release cadence, outbound network connections from a build process to an unfamiliar destination during a build (a strong signal of exfiltration or command-and-control activity), and a signed artifact whose signing identity doesn't match the expected CI pipeline that should have produced it.

Proactive controls across procurement and engineering

On the procurement side, evaluate and periodically re-assess third-party dependencies and CI actions the same way you would a vendor, including checking maintenance activity and known security history. On the engineering side, pin dependencies (including third-party CI actions) to an immutable commit hash rather than a mutable version tag, so a compromised upstream can't silently change what you're pulling without you explicitly updating the pin, and generate SBOMs and require signed, verified artifacts so an unexpected or unsigned component is caught automatically rather than relying on someone noticing manually.

Trade-offs

Pinning to an immutable hash instead of a convenient version tag adds real maintenance overhead (every legitimate update requires explicitly bumping the pin rather than automatically tracking a moving tag), but that overhead is exactly what closes the specific attack this control is meant to prevent: an upstream compromise silently changing what a moving tag points to without any action on your side.

Unauthorized use of an HSM (hardware security module) root key is one of the most severe possible findings in a signing pipeline, since the root key is typically the trust anchor everything else in the signing chain ultimately derives from; the response has to assume the worst about scope until evidence narrows it.

Forensic investigation

Start with the HSM's own access and operation logs (most HSMs log every cryptographic operation performed, including which key, what operation, and from which authenticated client), correlating the timeline of unauthorized use against known-legitimate signing operations to identify exactly which operations were NOT initiated by an expected, authorized pipeline. Cross-reference against network logs and authentication logs for the systems that have legitimate access to the HSM, looking for an unexpected authentication source or an authentication pattern (time of day, request volume) inconsistent with normal pipeline behavior.

Assessing scope and impact

Every artifact signed using the root key (or a key derived from it) during the window of unauthorized access has to be treated as potentially untrustworthy, not just the specific artifact that first drew attention; this means enumerating every signature produced during that window against the artifact registry and treating each one as needing re-verification or re-signing. If the root key signs intermediate keys rather than artifacts directly (a common PKI pattern), the scope assessment has to extend to everything trusted transitively through any intermediate key the root key issued or could have issued during the compromise window.

Recovery and key rotation, preserving trust where possible

The root key itself must be revoked and replaced; because it's a root of trust, this cascades: every intermediate certificate it issued needs to be re-issued from the new root, and every previously-signed artifact that relied on the old root's trust chain needs re-signing or an explicit, published transition plan customers and downstream consumers can follow (a documented key-rotation event, with the old root's revocation and the new root's public key published through the same trusted channel customers already use to verify your signatures). Where feasible, maintain the OLD root as revoked-but-documented (rather than silently disappearing) so downstream systems that cached the old root can be updated deliberately rather than suddenly failing verification with no explanation.

Trade-offs

Treating every signature from the compromise window as suspect, rather than trying to selectively determine which specific signings were the attacker's versus legitimate, is the conservative and correct choice here, even though it means re-signing artifacts that may well have been signed legitimately during that same window; the alternative, trying to cherry-pick which signings to trust, risks leaving a genuinely attacker-signed artifact in circulation because it was mistakenly judged legitimate.

A pre-merge security-gate checklist for pull requests needs to combine what can be automated (and therefore should never depend on a human remembering to check it) with what genuinely still needs a human's judgment, since collapsing that distinction either overloads reviewers with mechanical checks or under-automates things a human shouldn't have to catch manually every time.

Automated checks

Secret scanning on the diff, SCA on any changed dependency, SAST on changed files, and IaC misconfiguration scanning on any changed infrastructure definition, all running automatically on every PR without requiring a reviewer to remember to trigger them.

Manual reviews

A human reviewer still needs to assess things automation can't reliably judge: whether the CHANGE's actual business logic introduces an authorization gap (does this new endpoint correctly check the caller has permission for the specific resource, a check that's context-dependent in a way a generic scanner can't fully verify), and whether a third-party contribution's overall intent looks legitimate, beyond just what the automated scans flagged.

Required approvals and artifact verification

At least one reviewer with context on the affected area (not just any available reviewer) should approve before merge, and, for changes affecting the delivery pipeline itself or producing a released artifact, the artifact's signature and SBOM generation should be verified as part of the same gate, so a build that skips signing is caught here rather than discovered later at deploy time.

Considerations for third-party contributions

A pull request from an external, untrusted contributor needs the automated checks to run in an isolated context that doesn't expose organization secrets to the PR's own code during the CI run (the poisoned-pipeline-execution risk discussed elsewhere in this topic), and the manual review for an external contribution should weight more heavily toward understanding what the change actually does, since an external contributor hasn't built up the same track record of trust as an internal team member.

Enforcing without slowing developers down

The automated checks should complete fast enough (minutes, using the incremental/changed-file scanning techniques discussed elsewhere) that they don't become the bottleneck; the manual-review requirement should be scoped to a single required approver for routine changes, reserving a heavier, multi-reviewer requirement specifically for changes touching the pipeline's own security-relevant configuration or a genuinely high-risk area of the codebase, rather than applying the heaviest review bar uniformly to every single change regardless of its actual risk.

Trade-offs

A checklist this granular (separating automated from manual, and tiering the manual-review requirement by actual risk) takes more upfront design work than a single, uniform 'get one approval and pass CI' rule; that design work pays off by keeping the gate proportionate, catching genuinely risky changes with real scrutiny while not imposing that same heavy scrutiny on routine, low-risk changes that would otherwise slow the whole team down unnecessarily.

Threat-modeling the pipeline itself, rather than the application it ships, means treating the pipeline as a genuinely privileged system in its own right, since it routinely holds the credentials needed to deploy to production and often runs code it doesn't fully trust.

Assets and trust boundaries

The key assets are: the source repository (what determines what gets built), the CI runners (which execute arbitrary build-defined code), the secrets and signing keys the pipeline holds, and the artifact registry and deployment target (the ultimate destination an attacker wants to reach). Trust boundaries sit wherever untrusted or less-trusted input meets a more-trusted execution context: an external contributor's pull request meeting a runner that has repository write access or secrets access is the sharpest boundary, since a malicious PR is attacker-controlled input running inside a context that may hold real credentials.

Likely attackers and top threats

A plausible attacker profile ranges from an external contributor submitting a malicious pull request, to an attacker who has compromised a legitimate contributor's credentials, to an attacker who has compromised a third-party dependency or CI action the pipeline trusts. The top five threats, roughly in order of how often they show up in real incidents: (1) runner compromise via a malicious build step, letting an attacker read whatever secrets that runner had; (2) supply-chain poisoning via a compromised dependency or third-party action; (3) a malicious pull request triggering a workflow with elevated permissions (the poisoned-pipeline-execution pattern, particularly via triggers like pull_request_target that grant a fork's PR access to secrets); (4) leaked long-lived credentials granting broader access than a short-lived credential would; (5) a compromised signing key letting an attacker produce artifacts that appear legitimately trusted.

Mitigations, prioritized by impact and effort

Highest impact, lowest effort: pin all third-party actions to an immutable commit SHA (addresses threat 2 directly, cheap to implement, no ongoing operational cost). Next: avoid or tightly restrict dangerous trigger patterns like pull_request_target for anything that doesn't strictly need it (addresses threat 3, requires an audit of existing workflows but no new infrastructure). Then: move to ephemeral, least-privilege runners and short-lived, scoped credentials (addresses threats 1 and 4, moderate effort since it may require re-architecting how credentials are issued). Highest effort, addressing the residual risk in threat 5: keyless, OIDC (OpenID Connect)-backed signing removes the long-lived signing key from the picture entirely, closing that threat at the cost of migrating existing signing infrastructure.

Trade-offs

Prioritizing by impact-versus-effort rather than tackling every threat with equal urgency means some real risk (the signing-key compromise threat) stays partially open longer, since it's genuinely the most expensive to fully close; that's an honest, deliberate sequencing choice rather than an oversight, made explicit so stakeholders understand exactly what residual risk remains at each stage of the rollout.

A policy-as-code architecture evaluating SCA, SAST, and secrets findings together, rather than as three independent gates, needs a clear model for how policies are authored, how they're tested before they affect anyone, and how an emergency bypass is possible without becoming a silent, unaudited backdoor.

Architecture

flowchart TB
  PR[Pull request] --> Collect[Collect SCA + SAST + secrets findings]
  Collect --> Eval[Policy evaluation engine]
  Eval -->|blocking violation| Block[Block merge]
  Eval -->|advisory violation| Ticket[Open ticket, allow merge]
  Eval -->|no violation| Allow[Allow merge]
  Block -->|documented exception| Bypass[Emergency bypass, logged + time-boxed]

Authoring, testing, and versioning policies

Policies live in their own version-controlled repository, authored as code and reviewed through the same pull-request process as any other change, with the security team as a required reviewer for changes to the policy logic itself. Every policy change needs its own test suite (does it correctly flag the violation it's meant to catch, and correctly pass the case it's meant to allow) run in CI before the policy change itself can merge, exactly the same discipline applied to the application code the policy governs.

Enforcement: blocking versus advisory

A finding's disposition (block the merge outright, versus allow the merge but open a tracked ticket) is itself a policy decision, not a blanket rule; CRITICAL secrets and CRITICAL, reachable SCA findings default to blocking, while lower-severity or lower-confidence findings across all three categories default to advisory, giving visibility without stopping delivery for findings that don't yet warrant it.

Emergency bypasses and audit trails

An emergency bypass (letting a genuinely time-critical fix merge despite a blocking policy violation) needs to require an explicit, logged action from an authorized approver, be automatically time-boxed (the bypass itself expires, and the underlying violation must still be resolved by a defined deadline), and generate its own audit entry distinct from a normal merge, so a security review can later see exactly which merges bypassed policy, when, why, and whether the underlying issue was subsequently fixed as promised.

Handling emergency bypasses becoming routine

Track the RATE of emergency bypasses as its own metric; a bypass mechanism that's used rarely, for genuine emergencies, is healthy, while one used routinely is a signal that either the policy thresholds are miscalibrated (too strict for real development velocity) or the team has started treating the bypass as a normal path around the gate rather than an exception, either of which needs to be addressed at the policy level, not by simply tightening the bypass approval process further.

Trade-offs

Requiring a full test suite for every policy change adds friction to updating the policy itself, which could tempt someone to skip testing 'just this once' for an urgent policy fix; that temptation is exactly why the same PR-and-CI discipline applied to application code should apply here without exception, since an untested policy change risks either blocking legitimate work with a bug in the policy logic, or, worse, silently failing to catch what it was supposed to catch.