Secure Software Delivery: DevSecOps, Pipeline, and Supply Chain Security Questions

Findings from SAST and DAST tools arrive in different shapes (different severity scales, different identifiers for the same underlying weakness, different confidence signals), and a system that just concatenates them produces a flood of duplicate-looking tickets that erodes trust in the whole pipeline.

Data model

A canonical finding record needs at minimum: a normalized CVE or CWE identifier where one exists, the affected package or file path, a normalized severity on one consistent scale (rather than trusting each tool's own scale directly), the evidence (the specific line, request, or dependency that triggered the finding), the source tool, and a stable fingerprint used for deduplication.

Normalization rules

Each tool's native severity gets mapped onto one shared scale (for example CRITICAL/HIGH/MEDIUM/LOW) via a per-tool mapping table maintained by the security team, since 'HIGH' from one SAST tool and 'HIGH' from a DAST tool do not necessarily represent the same real-world risk. Package and path identifiers get normalized to a canonical form (resolving symlinks, stripping tool-specific path prefixes) so the same file reported by two different tools maps to the same record.

Deduplication and conflicting severities

The fingerprint for deduplication should combine the normalized location (file plus line range, or package plus version) with the normalized weakness category (CWE), not the raw tool-specific rule ID, since two different tools often use different rule IDs for the same underlying weakness. When two tools disagree on severity for what fingerprints as the same finding, take the higher of the two by default (a false negative from erring toward caution is cheaper than a false negative from erring toward dismissal) but surface both scores in the merged record so a human reviewer can see the disagreement rather than having it silently resolved.

Enrichment and prioritization

Enrich each normalized finding with exploitability metadata from exploit-db or the NVD's own exploit-maturity field, and assign priority as a function of normalized severity, exploitability, and (where available) whether the code path is actually reachable at runtime; assignment to an owner follows code-ownership data (the same signal a CODEOWNERS file already encodes) rather than a generic security-team queue, since the person who owns the code can act on the finding fastest.

Trade-offs

This system is worth building once an organization is running enough distinct scanners that manual triage across tool-specific dashboards has become the bottleneck; for a single team running one SAST and one SCA tool, the normalization and dedup machinery is more overhead than the problem it solves, and a simpler per-tool triage process is the right call instead.

This function exchanges a GitHub Actions OIDC (OpenID Connect) token for short-lived AWS credentials via STS, then uses those credentials, not the pipeline's own identity, to fetch a secret from Secrets Manager.

import os
import boto3
from botocore.exceptions import ClientError


def fetch_secret_via_oidc(secret_id, role_arn, region_name="us-east-1",
                          session_name="github-actions-secret-fetch"):
    oidc_token = os.environ.get("GITHUB_OIDC_TOKEN")
    if not oidc_token:
        raise RuntimeError(
            "GITHUB_OIDC_TOKEN is not set. In the workflow, request it with "
            "curl against $ACTIONS_ID_TOKEN_REQUEST_URL and export the value."
        )

    sts = boto3.client("sts", region_name=region_name)
    try:
        assumed = sts.assume_role_with_web_identity(
            RoleArn=role_arn,
            RoleSessionName=session_name,
            WebIdentityToken=oidc_token,
            DurationSeconds=900,
        )
    except ClientError as e:
        raise RuntimeError(f"OIDC role assumption failed: {e.response['Error']['Code']}") from e

    creds = assumed["Credentials"]
    secrets_client = boto3.client(
        "secretsmanager", region_name=region_name,
        aws_access_key_id=creds["AccessKeyId"],
        aws_secret_access_key=creds["SecretAccessKey"],
        aws_session_token=creds["SessionToken"],
    )

    try:
        response = secrets_client.get_secret_value(SecretId=secret_id)
    except ClientError as e:
        code = e.response["Error"]["Code"]
        if code == "ResourceNotFoundException":
            raise RuntimeError(f"Secret {secret_id!r} does not exist") from e
        if code == "AccessDeniedException":
            raise RuntimeError(f"Role {role_arn!r} lacks GetSecretValue on {secret_id!r}") from e
        raise

    if "SecretString" not in response:
        raise RuntimeError("Secret is binary; this helper only handles string secrets")
    return response["SecretString"]

Error handling

Three distinct failure modes are handled explicitly rather than letting a generic exception surface: a missing OIDC token environment variable (a workflow-configuration error, caught before any AWS call), a failed role assumption (wrapped with the specific STS error code so the log line is actionable rather than a generic traceback), and a Secrets Manager call that fails for a specific, distinguishable reason (the secret doesn't exist, versus the role lacks permission), each producing a targeted error message rather than one catch-all.

Permission assumptions

The role's trust policy must restrict the sub claim to this specific repository and branch, not merely the OIDC provider's aud, otherwise any repository under the same GitHub organization could assume the same role. The role's permission policy should grant secretsmanager:GetSecretValue on the one specific secret ARN needed, nothing broader, and DurationSeconds is set to the minimum practical value (900 seconds) since the credential only needs to live for this one build step.

Verified

Executed with unittest.mock standing in for the AWS client (no real AWS account available in this sandbox): a happy-path test confirms the correct STS and Secrets Manager calls and return value; a missing-token test confirms the early, explicit RuntimeError; an access-denied test confirms the specific error is caught and re-raised with an actionable message. All three passed.

A supply chain attack targets the trust an organization places in something it did not write itself, meaning attackers get their code running inside a target not by breaching the target directly, but by compromising something upstream the target already trusts and pulls in automatically.

How attackers compromise the chain

Attackers commonly target three points: a dependency (publishing a malicious package, or compromising a legitimate maintainer's account to push a malicious update to an already-trusted package), the CI/CD pipeline itself (compromising the build system so the backdoor is inserted during compilation, invisible in the source repository), or a vendor's software update mechanism (compromising a trusted vendor's own build or distribution infrastructure so their legitimate customers receive a malicious update signed with the vendor's own real credentials).

Two historical examples

The SolarWinds SUNBURST attack (2020) is the canonical build-system-compromise example: attackers compromised SolarWinds' own build environment and inserted a backdoor (via malware named SUNSPOT) into the Orion software's build process itself, so the resulting update, signed with SolarWinds' own legitimate certificate, was silently backdoored and distributed to roughly 18,000 organizations who trusted the update as coming from their vendor. The tj-actions/changed-files compromise (2025) is a more recent example targeting the CI/CD supply chain specifically: a popular, widely-used GitHub Action was compromised, leaking secrets from over 23,000 repositories that used it in their workflows, illustrating how a single compromised third-party CI dependency can cascade across a huge number of otherwise-unrelated organizations.

Indicators of compromise to hunt for

Unexpected changes to a build's dependency lockfile with no corresponding pull request explaining why, a dependency's published version jumping in a way inconsistent with its normal release cadence, outbound network connections from a build process to an unfamiliar destination during a build (a strong signal of exfiltration or command-and-control activity), and a signed artifact whose signing identity doesn't match the expected CI pipeline that should have produced it.

Proactive controls across procurement and engineering

On the procurement side, evaluate and periodically re-assess third-party dependencies and CI actions the same way you would a vendor, including checking maintenance activity and known security history. On the engineering side, pin dependencies (including third-party CI actions) to an immutable commit hash rather than a mutable version tag, so a compromised upstream can't silently change what you're pulling without you explicitly updating the pin, and generate SBOMs and require signed, verified artifacts so an unexpected or unsigned component is caught automatically rather than relying on someone noticing manually.

Trade-offs

Pinning to an immutable hash instead of a convenient version tag adds real maintenance overhead (every legitimate update requires explicitly bumping the pin rather than automatically tracking a moving tag), but that overhead is exactly what closes the specific attack this control is meant to prevent: an upstream compromise silently changing what a moving tag points to without any action on your side.

A new scanner producing a wall of findings on day one is the single most predictable way to lose developer trust in a security program, so the triage approach has to separate 'this tool is noisy' from 'this specific finding is wrong' and fix the right one.

A systematic triage approach

Step one, establish a baseline before touching anything. Run the scanner against the existing codebase once and snapshot every current finding as "pre-existing, not newly introduced". This baseline is not a permanent suppression; it is a starting point that separates the debt the codebase already carried from anything a new PR introduces going forward.

Step two, cluster before triaging individually. Group findings by rule ID and inspect the top five to ten rules by volume first; a single overly broad rule is often responsible for the majority of the noise, and fixing that one rule's configuration has more leverage than triaging a thousand individual findings one at a time.

Step three, validate a sample of each cluster with a developer who owns that code, not the security team alone; the person who wrote the code can usually say in seconds whether a given pattern is actually exploitable in context, which the scanner cannot know without that context.

Step four, tune rather than blanket-suppress. For a rule confirmed to be a true false-positive pattern in this codebase (for example, a SQL-construction pattern the scanner flags that is actually parameterized through an ORM it doesn't recognize), adjust the rule's configuration or add a scoped suppression with a comment explaining why, rather than disabling the rule org-wide, which would also hide the rare true positive the same rule would have caught elsewhere.

Validating you haven't suppressed a real finding

After each tuning pass, run the scanner against a small set of KNOWN-VULNERABLE test fixtures (intentionally-vulnerable code samples representing the rule's true-positive case) to confirm the tuned rule still fires on the pattern it's supposed to catch, not just that it stopped firing on the false positive. Track the suppression list itself as a metric over time: a shrinking or stable list after the first tuning pass is healthy; a list that keeps growing every week is a sign the team has started suppressing rather than fixing, which defeats the purpose of the gate entirely.

Where automation fits

Automate what does not need human judgment: the baseline snapshot itself (a scheduled job diffing today's findings against the last approved baseline), nightly re-runs against the known-vulnerable test fixtures so a tuning regression is caught the next morning rather than at the next manual audit, and tracking the suppression list's size and age over time so a growing list raises itself as an alert rather than waiting for someone to notice. Keep the actual judgment calls (is this specific pattern really safe in context) manual, since that is exactly the step a developer with real code context does better than a script.

The trade-off

This process trades a slower initial rollout (weeks of tuning before the gate becomes meaningfully blocking) for a program that developers still trust six months later. Skipping straight to a hard-blocking gate on day one produces faster nominal coverage and a much higher chance the whole program gets quietly bypassed once developers learn how.

Secrets end up in Git history or CI artifacts through a small number of well-known paths, and each one has its own concrete, checkable prevention control rather than a single blanket fix.

Leakage vectors and controls

A developer commits a secret directly into source code (a hardcoded API key, a .env file checked in by accident). Prevention: a pre-commit hook running a secret-detection tool (gitleaks or trufflehog) that blocks the commit locally before it ever reaches the remote, plus a server-side push-protection check as a backstop for the developer who bypasses or doesn't have the local hook installed.

A secret leaks into CI build logs (an environment variable printed by a verbose build step, or a script echoing its own inputs for debugging). Prevention: enable the CI provider's built-in log masking for any value sourced from a secrets store (most CI systems mask a value automatically once it's referenced as a secret), and explicitly discourage set -x or verbose debug flags in scripts that handle secret-bearing environment variables.

A secret gets baked into a built artifact or container image (a config file with a real credential copied into the image during a multi-stage build, rather than injected at runtime). Prevention: scan every built artifact and container layer for secret patterns before it's published, and enforce a runtime-injection pattern (secrets manager, sidecar, or environment variable set at deploy time) rather than baking any credential into the image at build time.

Historical leakage, meaning a secret that was committed and later removed, but still exists in the Git history of an old commit. Prevention: run a periodic (for example nightly) scan of the FULL Git history, not just the current HEAD, since deleting a file in a later commit does not remove it from history; and gate any change that widens a repository's exposure (making a private repository public, adding an external collaborator, connecting a new third-party integration) on that full-history scan passing clean first, so a secret from years ago cannot be newly exposed the moment access to the repository widens.

If a secret is found in a log or artifact

The response is always rotate first, investigate second: revoke and reissue the credential immediately, since the window between detection and rotation is exactly the window an attacker could exploit, and only afterward investigate how it leaked and whether it was actually accessed by anyone unauthorized. A secret found by the historical scan gets the same treatment: it must be treated as compromised and rotated, not merely scrubbed from history, since history rewrites don't help once a repository has been cloned elsewhere, and the safe assumption is that any exposed secret has already been seen.

Trade-offs

Pre-commit hooks add a small amount of friction to every commit (a moment's pause while the scan runs) in exchange for catching the leak at the cheapest possible point, before it ever reaches a shared remote; skipping local hooks in favor of only a server-side check still catches the leak, but only after it has already reached the remote repository, which several tools and CI systems may have already cloned or cached by the time it's caught.