Secure Software Delivery: DevSecOps, Pipeline, and Supply Chain Security Questions

Auto-blocking on a raw CVSS (Common Vulnerability Scoring System) threshold and auto-creating a ticket both fail in the same direction if applied blindly: CVSS alone doesn't distinguish a theoretical, unreachable finding from an actively-exploited one, so a policy built purely on that score either blocks too aggressively or lets real risk slide through as 'just a ticket'.

Pros and cons of each

Auto-blocking on CVSS threshold: strong guarantee (nothing above the threshold ever ships without explicit action), but high false-positive cost, since CVSS alone doesn't account for whether the vulnerable code path is actually reachable in this specific application; blocking on raw CVSS routinely stops merges for findings that pose no practical risk in context, which is exactly the pattern that erodes developer trust in the gate.

Allow-merge-plus-ticket: preserves developer velocity and avoids blocking on unreachable findings, but risks a genuinely dangerous, actively-exploited vulnerability shipping to production while its ticket sits in a backlog with no enforced urgency, especially if the ticket has no real deadline or escalation path.

Considering developer productivity, attack window, and context-aware exploitability

The honest resolution isn't picking one policy over the other uniformly, but making the blocking decision a function of BOTH severity and reachability/exploitability together: a CRITICAL, actively-exploited, reachable finding blocks the merge outright regardless of any velocity cost, since the attack-window cost of shipping it clearly outweighs the delay cost of fixing it now; a HIGH-CVSS but unreachable or theoretical finding becomes a tracked, SLA-bound ticket rather than a blocker, since blocking here buys little real risk reduction for a real velocity cost.

A balanced policy

Block automatically only on the combination of high severity AND either confirmed reachability or known active exploitation; everything else routes to a ticket with a severity-tiered SLA (days for the highest tier, weeks for lower tiers) and an escalation path if the SLA is missed, so 'just a ticket' still carries real enforced urgency rather than becoming an indefinitely-deferred backlog item.

An exception process

Even a rule this well-calibrated will occasionally need a genuine, documented exception (a finding that's technically reachable but mitigated by a compensating control elsewhere the automated policy can't see); the exception needs an owner, a stated reason, a re-review date, and visibility to the security team, following the same pattern used for every other exception mechanism discussed in this topic.

Trade-offs

This combined-criteria policy is more sophisticated to implement (it requires reachability analysis, not just a CVSS lookup) than either pure-blocking or pure-ticketing alone, but it's the calibration that actually distinguishes real risk from noise, which is the entire point of moving past CVSS-only gating in the first place.

A compromised runner that pushed a backdoored image to production is a scenario where the response has to move fast on containment while preserving enough evidence to actually understand what happened, since those two goals can pull in opposite directions if handled carelessly.

Detection

The telltale signs are usually: a build that ran at an unexpected time or from an unexpected trigger, a build step that doesn't match what the pipeline definition in the repository actually specifies (indicating the runner's execution diverged from the declared pipeline, not just that the pipeline itself was malicious), or a signed image whose signing identity doesn't match the expected pipeline. Correlating the timeline of the suspicious build against the runner's own system logs (if the runner logs its own activity outside the build job itself) often reveals the injected step.

Containment, across CI and production

Immediately revoke the compromised runner's credentials and any secrets it had access to, and quarantine (do not terminate yet, if forensics are needed) the runner instance itself rather than the whole runner pool, to limit disruption to unrelated jobs. In the registry, prevent the backdoored image from being pulled further (mark it untrusted or remove the tag pointing to it) without necessarily deleting it yet, since the image itself is evidence. In production/Kubernetes, identify every workload currently running that backdoored image (matched by digest, not just tag, since a tag can be reused) and begin rolling those workloads back to the last known-good, signed image.

Evidence collection

Preserve the runner's build logs, the exact backdoored image and its layers, and the signing metadata (which identity signed it, when) before any remediation step that would overwrite or delete this evidence; a forensic snapshot of the compromised runner's filesystem, if feasible, before it's destroyed, is valuable for understanding exactly how the injection happened.

Remediation

Revoke the signing key or certificate that was used to sign the backdoored image if there's any indication the signing process itself, not just the runner, was compromised; rebuild a clean image from a verified-good commit on a runner confirmed uncompromised, sign it fresh, and redeploy. Rotate any secret the compromised runner had access to, on the assumption it may have been exfiltrated even if there's no direct evidence it was.

Long-term controls

Move to ephemeral, single-use runners so no future compromise can persist across jobs, add build-step verification that compares the runner's actual executed steps against the pipeline definition declared in source control (catching exactly this kind of divergence automatically going forward), and require signature verification at the Kubernetes admission layer so an unsigned or unexpectedly-signed image can never reach production even if a similar compromise recurs.

Trade-offs

Quarantining rather than immediately terminating the compromised runner preserves forensic value but leaves a compromised system technically still present in the environment for a short window; that trade is worth it specifically when understanding the root cause (to prevent recurrence) matters as much as stopping the immediate damage, though a runner posing an active, ongoing threat (still actively exfiltrating data) would tip the balance toward immediate termination instead.

A security gate is a checkpoint in the CI/CD pipeline where a security or compliance finding can affect whether a change proceeds, and the central design choice for any gate is whether it's a hard gate (blocking) or a soft gate (advisory).

Hard versus soft gates

A hard gate blocks the merge or deploy outright when its condition fires; nothing proceeds until the finding is fixed or an explicit, documented exception is granted. A soft gate surfaces the finding (a warning, an automatically-created ticket) but allows the change to proceed regardless, trading immediate enforcement for preserved velocity.

When to adopt each

Adopt a hard gate for findings where the confidence is high and the risk of shipping is genuinely severe: a confirmed secret in a commit, or a CRITICAL, reachable vulnerability with a known exploit, since the cost of blocking (a short delay while it's fixed) is clearly lower than the cost of shipping it. Adopt a soft gate for anything with a meaningfully higher false-positive rate or lower-confidence signal (a MEDIUM-severity SAST finding whose reachability is unclear), where blocking would impose a real velocity cost against uncertain actual risk.

Measuring the impact on security posture and velocity

For security posture, track whether soft-gate findings are actually getting resolved over a reasonable time window (not just generated and ignored), since a soft gate that never leads to a fix is providing visibility without any actual risk reduction. For velocity, track the time added to a typical pull request by the hard gates specifically (since those are the ones that can actually stop work), watching for the gate becoming a growing source of delay as the codebase and its dependency surface grow over time.

The trade-off in practice

A program dominated by hard gates ships the most defensible security posture on paper but risks the alert-fatigue and gate-erosion pattern discussed elsewhere in this topic if the gates aren't well-tuned; a program dominated by soft gates preserves velocity but risks findings quietly accumulating in an unenforced backlog with no real pressure to fix them. The healthiest programs use a small number of well-tuned hard gates for the highest-confidence, highest-severity findings, and route everything else through soft gates with a real, tracked follow-through mechanism, rather than treating 'hard gate' as the default setting for every category of finding regardless of its actual confidence and severity.

A supply chain attack targets the trust an organization places in something it did not write itself, meaning attackers get their code running inside a target not by breaching the target directly, but by compromising something upstream the target already trusts and pulls in automatically.

How attackers compromise the chain

Attackers commonly target three points: a dependency (publishing a malicious package, or compromising a legitimate maintainer's account to push a malicious update to an already-trusted package), the CI/CD pipeline itself (compromising the build system so the backdoor is inserted during compilation, invisible in the source repository), or a vendor's software update mechanism (compromising a trusted vendor's own build or distribution infrastructure so their legitimate customers receive a malicious update signed with the vendor's own real credentials).

Two historical examples

The SolarWinds SUNBURST attack (2020) is the canonical build-system-compromise example: attackers compromised SolarWinds' own build environment and inserted a backdoor (via malware named SUNSPOT) into the Orion software's build process itself, so the resulting update, signed with SolarWinds' own legitimate certificate, was silently backdoored and distributed to roughly 18,000 organizations who trusted the update as coming from their vendor. The tj-actions/changed-files compromise (2025) is a more recent example targeting the CI/CD supply chain specifically: a popular, widely-used GitHub Action was compromised, leaking secrets from over 23,000 repositories that used it in their workflows, illustrating how a single compromised third-party CI dependency can cascade across a huge number of otherwise-unrelated organizations.

Indicators of compromise to hunt for

Unexpected changes to a build's dependency lockfile with no corresponding pull request explaining why, a dependency's published version jumping in a way inconsistent with its normal release cadence, outbound network connections from a build process to an unfamiliar destination during a build (a strong signal of exfiltration or command-and-control activity), and a signed artifact whose signing identity doesn't match the expected CI pipeline that should have produced it.

Proactive controls across procurement and engineering

On the procurement side, evaluate and periodically re-assess third-party dependencies and CI actions the same way you would a vendor, including checking maintenance activity and known security history. On the engineering side, pin dependencies (including third-party CI actions) to an immutable commit hash rather than a mutable version tag, so a compromised upstream can't silently change what you're pulling without you explicitly updating the pin, and generate SBOMs and require signed, verified artifacts so an unexpected or unsigned component is caught automatically rather than relying on someone noticing manually.

Trade-offs

Pinning to an immutable hash instead of a convenient version tag adds real maintenance overhead (every legitimate update requires explicitly bumping the pin rather than automatically tracking a moving tag), but that overhead is exactly what closes the specific attack this control is meant to prevent: an upstream compromise silently changing what a moving tag points to without any action on your side.

Policy-as-code means security and compliance rules are written and enforced as version-controlled, testable code, evaluated automatically against a proposed change, rather than living as a document a human has to remember to check.

A simple example rule

In plain terms: deny any pull request whose commit contains a secret-scanning finding, or an SCA finding at HIGH severity or above. Expressed as a Rego-style rule:

deny contains msg if {
	input.sca_findings[_].severity == "HIGH"
	msg := "blocked: high-severity SCA finding present"
}

deny contains msg if {
	count(input.secret_findings) > 0
	msg := "blocked: secret detected in commit"
}

Tools that enforce this kind of rule

Open Policy Agent (OPA), often paired with Gatekeeper for Kubernetes admission control, and HashiCorp Sentinel (commonly used with Terraform Cloud/Enterprise) are two widely-adopted engines for exactly this pattern: writing the policy once as code, then evaluating it automatically wherever a relevant change is proposed, whether that's a pull request, a Terraform plan, or a Kubernetes deployment request.

Why this is preferable to a document

A written policy document requires a human to remember to check it and to interpret it consistently; policy-as-code evaluates the exact same logic every time, on every proposed change, without depending on any individual reviewer's memory or interpretation, and because it's code, it can be tested (does this policy correctly block the case it's meant to block, and correctly allow the case it's meant to allow) the same way any other piece of software is tested before it ships.

Trade-offs

Writing policy as code requires someone with the skill to write and test that code, which is a real barrier compared to writing a plain-language policy document; the payoff is that the enforced behavior and the documented intent can never drift apart the way a document and actual practice quietly can over time, since the code IS the enforcement.