Secure Software Delivery: DevSecOps, Pipeline, and Supply Chain Security Questions

A supply chain attack targets the trust an organization places in something it did not write itself, meaning attackers get their code running inside a target not by breaching the target directly, but by compromising something upstream the target already trusts and pulls in automatically.

How attackers compromise the chain

Attackers commonly target three points: a dependency (publishing a malicious package, or compromising a legitimate maintainer's account to push a malicious update to an already-trusted package), the CI/CD pipeline itself (compromising the build system so the backdoor is inserted during compilation, invisible in the source repository), or a vendor's software update mechanism (compromising a trusted vendor's own build or distribution infrastructure so their legitimate customers receive a malicious update signed with the vendor's own real credentials).

Two historical examples

The SolarWinds SUNBURST attack (2020) is the canonical build-system-compromise example: attackers compromised SolarWinds' own build environment and inserted a backdoor (via malware named SUNSPOT) into the Orion software's build process itself, so the resulting update, signed with SolarWinds' own legitimate certificate, was silently backdoored and distributed to roughly 18,000 organizations who trusted the update as coming from their vendor. The tj-actions/changed-files compromise (2025) is a more recent example targeting the CI/CD supply chain specifically: a popular, widely-used GitHub Action was compromised, leaking secrets from over 23,000 repositories that used it in their workflows, illustrating how a single compromised third-party CI dependency can cascade across a huge number of otherwise-unrelated organizations.

Indicators of compromise to hunt for

Unexpected changes to a build's dependency lockfile with no corresponding pull request explaining why, a dependency's published version jumping in a way inconsistent with its normal release cadence, outbound network connections from a build process to an unfamiliar destination during a build (a strong signal of exfiltration or command-and-control activity), and a signed artifact whose signing identity doesn't match the expected CI pipeline that should have produced it.

Proactive controls across procurement and engineering

On the procurement side, evaluate and periodically re-assess third-party dependencies and CI actions the same way you would a vendor, including checking maintenance activity and known security history. On the engineering side, pin dependencies (including third-party CI actions) to an immutable commit hash rather than a mutable version tag, so a compromised upstream can't silently change what you're pulling without you explicitly updating the pin, and generate SBOMs and require signed, verified artifacts so an unexpected or unsigned component is caught automatically rather than relying on someone noticing manually.

Trade-offs

Pinning to an immutable hash instead of a convenient version tag adds real maintenance overhead (every legitimate update requires explicitly bumping the pin rather than automatically tracking a moving tag), but that overhead is exactly what closes the specific attack this control is meant to prevent: an upstream compromise silently changing what a moving tag points to without any action on your side.

A secret is any credential that grants access to something (an API key, a database password, a private signing key, an OAuth token) where possession of the value itself is the entire authorization check. General configuration (a timeout value, a feature flag, a log level, a service's hostname) describes how the system should behave but grants no access on its own.

Concrete examples

• Secrets: a database connection password, an AWS access key, a third-party API token, a TLS private key, a webhook signing secret.
• Configuration: a request-timeout value, a retry count, a feature-flag boolean, a public API base URL, a log-verbosity level.

Why secrets need special handling

The practical distinction is blast radius on exposure: if a configuration value leaks (say, into a public log or a committed file), nothing bad happens by itself; at worst it reveals an implementation detail. If a secret leaks the same way, whoever finds it can immediately act as your application, which might mean reading a production database, calling a paid third-party API on your account, or impersonating your service to another system entirely. Secrets also need a lifecycle configuration values don't: rotation, expiry, and revocation, since a leaked secret needs to be actively invalidated, while a leaked configuration value just needs to be corrected.

Three practical controls

1. Store secrets in a dedicated secrets manager (HashiCorp Vault, AWS Secrets Manager, or equivalent), never in environment variables baked into a config file or committed alongside application configuration. This keeps secrets out of version control by construction and gives you a single place to rotate and audit access.
2. Inject secrets at runtime via short-lived, scoped credentials (an OIDC (OpenID Connect)-issued token exchanged for a temporary cloud credential) rather than long-lived static keys checked into a pipeline's configuration. A short-lived credential that leaks has a bounded window of usefulness to an attacker; a static key that leaks is valid until someone notices and manually revokes it.
3. Run automated secret scanning on every commit and every build artifact, so a secret that accidentally ends up in code, a log line, or a built container image is caught and flagged for rotation before it reaches a public or widely-accessible location, rather than relying on developers to notice manually.

The trade-off

Treating every string that looks sensitive as a secret (over-classifying configuration as secret) adds unnecessary rotation and access-control overhead to values that don't need it; the practical test is always 'does possessing this value alone grant access to something', and applying that test consistently is what keeps the secrets-management overhead proportionate to the actual risk.

A pre-merge security-gate checklist for pull requests needs to combine what can be automated (and therefore should never depend on a human remembering to check it) with what genuinely still needs a human's judgment, since collapsing that distinction either overloads reviewers with mechanical checks or under-automates things a human shouldn't have to catch manually every time.

Automated checks

Secret scanning on the diff, SCA on any changed dependency, SAST on changed files, and IaC misconfiguration scanning on any changed infrastructure definition, all running automatically on every PR without requiring a reviewer to remember to trigger them.

Manual reviews

A human reviewer still needs to assess things automation can't reliably judge: whether the CHANGE's actual business logic introduces an authorization gap (does this new endpoint correctly check the caller has permission for the specific resource, a check that's context-dependent in a way a generic scanner can't fully verify), and whether a third-party contribution's overall intent looks legitimate, beyond just what the automated scans flagged.

Required approvals and artifact verification

At least one reviewer with context on the affected area (not just any available reviewer) should approve before merge, and, for changes affecting the delivery pipeline itself or producing a released artifact, the artifact's signature and SBOM generation should be verified as part of the same gate, so a build that skips signing is caught here rather than discovered later at deploy time.

Considerations for third-party contributions

A pull request from an external, untrusted contributor needs the automated checks to run in an isolated context that doesn't expose organization secrets to the PR's own code during the CI run (the poisoned-pipeline-execution risk discussed elsewhere in this topic), and the manual review for an external contribution should weight more heavily toward understanding what the change actually does, since an external contributor hasn't built up the same track record of trust as an internal team member.

Enforcing without slowing developers down

The automated checks should complete fast enough (minutes, using the incremental/changed-file scanning techniques discussed elsewhere) that they don't become the bottleneck; the manual-review requirement should be scoped to a single required approver for routine changes, reserving a heavier, multi-reviewer requirement specifically for changes touching the pipeline's own security-relevant configuration or a genuinely high-risk area of the codebase, rather than applying the heaviest review bar uniformly to every single change regardless of its actual risk.

Trade-offs

A checklist this granular (separating automated from manual, and tiering the manual-review requirement by actual risk) takes more upfront design work than a single, uniform 'get one approval and pass CI' rule; that design work pays off by keeping the gate proportionate, catching genuinely risky changes with real scrutiny while not imposing that same heavy scrutiny on routine, low-risk changes that would otherwise slow the whole team down unnecessarily.

The workflow below runs Trivy's filesystem scanner against a Node.js repository on every pull request, uploads the full findings as a SARIF artifact for later triage, and then fails the job specifically when a CRITICAL-severity vulnerability is found, using two separate Trivy invocations so the artifact upload always succeeds even when the gate fails.

name: dependency-scan

on:
  pull_request:
    branches: [main]

jobs:
  sca:
    runs-on: ubuntu-latest
    permissions:
      contents: read
    steps:
      - uses: actions/checkout@v4

      - name: Run Trivy filesystem scan
        uses: aquasecurity/trivy-action@v0.24.0
        with:
          scan-type: fs
          scan-ref: .
          format: sarif
          output: trivy-results.sarif
          severity: CRITICAL,HIGH
          exit-code: '0'

      - name: Upload SARIF report
        uses: actions/upload-artifact@v4
        with:
          name: trivy-sarif-report
          path: trivy-results.sarif

      - name: Fail on CRITICAL findings
        uses: aquasecurity/trivy-action@v0.24.0
        with:
          scan-type: fs
          scan-ref: .
          severity: CRITICAL
          exit-code: '1'
          format: table

Why two scan steps instead of one

A single Trivy invocation with exit-code: 1 would fail the job immediately on a CRITICAL finding, which means the SARIF upload step never runs, since GitHub Actions stops the job on a failing step by default. Running the SARIF-producing scan first with exit-code: 0 (never fails) guarantees the full report is always archived as an artifact for triage, and the second, narrower CRITICAL-only scan is what actually gates the merge. This two-step pattern is the practical answer to 'how do I both fail the build AND keep the evidence', which a single combined step cannot do without extra continue-on-error plumbing that obscures the actual gating logic.

Verified

This workflow parses as valid YAML (verified with PyYAML): one job, four steps, pull_request trigger on main; on parsing as a YAML boolean key is a known PyYAML quirk that does not affect GitHub's own workflow parser, which treats on as a reserved workflow keyword regardless. Checked aquasecurity/trivy-action's published tags via the GitHub API: the action only publishes v-prefixed tags (v0.24.0, v0.25.0, etc.); an unprefixed @0.24.0 reference does not exist and GitHub Actions would fail to resolve it at run time with "unable to find version". The action reference has been corrected to @v0.24.0 in both steps above, and actions/checkout@v4 and actions/upload-artifact@v4 were also confirmed to exist as real tags.

Trade-offs

The severity: CRITICAL,HIGH on the first scan (broader than the gate) is deliberate: it means the SARIF artifact captures HIGH-severity findings for visibility even though they don't block the merge, giving the team a queue of near-term work without slowing anyone down today.

A security gate is a checkpoint in the CI/CD pipeline where a security or compliance finding can affect whether a change proceeds, and the central design choice for any gate is whether it's a hard gate (blocking) or a soft gate (advisory).

Hard versus soft gates

A hard gate blocks the merge or deploy outright when its condition fires; nothing proceeds until the finding is fixed or an explicit, documented exception is granted. A soft gate surfaces the finding (a warning, an automatically-created ticket) but allows the change to proceed regardless, trading immediate enforcement for preserved velocity.

When to adopt each

Adopt a hard gate for findings where the confidence is high and the risk of shipping is genuinely severe: a confirmed secret in a commit, or a CRITICAL, reachable vulnerability with a known exploit, since the cost of blocking (a short delay while it's fixed) is clearly lower than the cost of shipping it. Adopt a soft gate for anything with a meaningfully higher false-positive rate or lower-confidence signal (a MEDIUM-severity SAST finding whose reachability is unclear), where blocking would impose a real velocity cost against uncertain actual risk.

Measuring the impact on security posture and velocity

For security posture, track whether soft-gate findings are actually getting resolved over a reasonable time window (not just generated and ignored), since a soft gate that never leads to a fix is providing visibility without any actual risk reduction. For velocity, track the time added to a typical pull request by the hard gates specifically (since those are the ones that can actually stop work), watching for the gate becoming a growing source of delay as the codebase and its dependency surface grow over time.

The trade-off in practice

A program dominated by hard gates ships the most defensible security posture on paper but risks the alert-fatigue and gate-erosion pattern discussed elsewhere in this topic if the gates aren't well-tuned; a program dominated by soft gates preserves velocity but risks findings quietly accumulating in an unenforced backlog with no real pressure to fix them. The healthiest programs use a small number of well-tuned hard gates for the highest-confidence, highest-severity findings, and route everything else through soft gates with a real, tracked follow-through mechanism, rather than treating 'hard gate' as the default setting for every category of finding regardless of its actual confidence and severity.