CI/CD Pipeline Design and Architecture Questions

A single workflow that builds for two architectures, generates an SBOM, scans, and signs needs each step to operate on the SAME image digest, so the SBOM accurately describes what was scanned, and what was scanned is exactly what gets signed.

name: build-sign-publish

on:
  push:
    branches: [main]

permissions:
  contents: read
  packages: write
  id-token: write   # required for cosign keyless (OIDC) signing

env:
  IMAGE: ghcr.io/${{ github.repository }}

jobs:
  build-scan-sign:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Set up QEMU
        uses: docker/setup-qemu-action@v3

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3

      - name: Log in to registry
        uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Build and push multi-arch image
        id: build
        uses: docker/build-push-action@v6
        with:
          context: .
          platforms: linux/amd64,linux/arm64
          push: true
          tags: ${{ env.IMAGE }}:${{ github.sha }}
          cache-from: type=gha
          cache-to: type=gha,mode=max

      - name: Generate SBOM with syft
        uses: anchore/sbom-action@v0
        with:
          image: ${{ env.IMAGE }}@${{ steps.build.outputs.digest }}
          format: cyclonedx-json
          output-file: sbom.cdx.json

      - name: Scan image with Trivy
        uses: aquasecurity/trivy-action@v0.24.0
        with:
          image-ref: ${{ env.IMAGE }}@${{ steps.build.outputs.digest }}
          severity: CRITICAL
          exit-code: '1'
          format: table

      - name: Install cosign
        uses: sigstore/cosign-installer@v3

      - name: Sign image keylessly
        run: cosign sign --yes "${IMAGE}@${DIGEST}"
        env:
          IMAGE: ${{ env.IMAGE }}
          DIGEST: ${{ steps.build.outputs.digest }}

      - name: Upload SBOM
        uses: actions/upload-artifact@v4
        with:
          name: sbom
          path: sbom.cdx.json

Why the digest, not the tag, threads through every step

Every downstream step (SBOM generation, scanning, signing) references ${{ steps.build.outputs.digest }}, the immutable content digest the build step itself produced, rather than the mutable :${{ github.sha }} tag; this guarantees the SBOM describes exactly what was scanned and exactly what gets signed, closing the gap a tag-based reference would leave open if the tag were somehow overwritten between steps.

Secure credential handling

Registry authentication uses the automatically-issued GITHUB_TOKEN, scoped to this repository and this workflow run, rather than a long-lived personal access token; cosign signing uses id-token: write permission to obtain a short-lived OIDC (OpenID Connect) token exchanged for a Sigstore certificate, meaning no signing key material is stored as a secret at all.

Verified

Parsed with PyYAML: valid YAML, ten steps confirmed in order, and permissions.id-token confirmed present as write, which cosign's keyless signing requires to obtain its OIDC token from the GitHub Actions runtime. Also checked every third-party action reference against the GitHub API's published tags: docker/setup-qemu-action@v3, docker/setup-buildx-action@v3, docker/login-action@v3, docker/build-push-action@v6, anchore/sbom-action@v0, and sigstore/cosign-installer@v3 all resolve to real tags. aquasecurity/trivy-action@0.24.0 did NOT resolve; the action only publishes v-prefixed tags, so the reference has been corrected to @v0.24.0 above.

Trade-offs

cache-to: type=gha,mode=max speeds up subsequent builds meaningfully but stores build-layer cache in GitHub's own Actions cache, which has its own size limits and retention policy; for a very large image, cache eviction under those limits can occasionally force a slower, cold rebuild, which is an acceptable trade for the typical case where caching saves far more time than it costs.

The schema needs to support point lookups by commit (a direct equality match) and a range-plus-filter query (unsigned artifacts older than a cutoff), which argues for two different indexes rather than relying on the primary key alone.

CREATE TABLE artifacts (
    artifact_id     TEXT PRIMARY KEY,
    version         TEXT NOT NULL,
    build_id        TEXT NOT NULL,
    commit_sha      TEXT NOT NULL,
    builder         TEXT NOT NULL,
    created_at      TEXT NOT NULL,
    size_bytes      INTEGER NOT NULL,
    signed          INTEGER NOT NULL DEFAULT 0,
    provenance_blob TEXT
);

CREATE INDEX idx_artifacts_commit_sha ON artifacts(commit_sha);
CREATE INDEX idx_artifacts_signed_created_at ON artifacts(signed, created_at);

The two queries

-- Find all artifacts built from commit X
SELECT artifact_id, version, build_id
FROM artifacts
WHERE commit_sha = 'abc123';

-- List unsigned artifacts older than 30 days
SELECT artifact_id, version, created_at
FROM artifacts
WHERE signed = 0
  AND created_at < date('now', '-30 days');

Indexing choices

The single-column index on commit_sha directly serves the first query as an index lookup rather than a full table scan, which matters once the table holds millions of rows across a busy build pipeline. The second query filters on TWO columns together (signed and created_at), so a composite index with signed as the leading column is the right choice: signed is low-cardinality (only two values) but highly selective for this specific query, since unsigned artifacts are expected to be a small minority in a healthy pipeline, and created_at as the second column lets the database narrow the age range within that already-small unsigned subset using the same index, rather than needing a separate index per column and then intersecting the results.

Verified

Executed against an in-memory SQLite database: created the schema and both indexes, inserted three sample rows (two sharing a commit SHA, one signed, two unsigned with different ages), and confirmed both queries return the expected rows. EXPLAIN QUERY PLAN confirmed the first query uses idx_artifacts_commit_sha and the second uses idx_artifacts_signed_created_at, rather than falling back to a full table scan.

Trade-offs

Storing provenance_blob as an unstructured JSON column keeps the schema flexible for evolving attestation formats without a migration every time the provenance schema changes, at the cost of not being able to efficiently query or index into specific fields inside that JSON without either a generated column or moving to a database with native JSON-path indexing; for a system that needs to query on specific provenance fields frequently, promoting those fields to their own indexed columns would be the next evolution of this schema.

For 200 microservices moving from a git commit to a production-ready image, the pipeline needs to run each check at the point where its cost of running is lowest and its signal is most useful, then promote the SAME built artifact forward rather than rebuilding at each stage.

Stage-by-stage walkthrough

flowchart LR
  Commit[Git commit] --> SCA[Static code analysis]
  SCA --> Test[Unit tests]
  Test --> Build[Build container image]
  Build --> SBOM[Generate SBOM]
  SBOM --> Scan[Vulnerability scan]
  Scan --> Sign[Sign image]
  Sign --> Dev[Promote to dev]
  Dev -->|same digest| Staging[Promote to staging]
  Staging -->|same digest, approval| Prod[Promote to production]

1. Static code analysis and unit tests run first, before a container is even built, since they're the cheapest checks and catch the most common class of bug fastest.
2. Build the image once. This is the single build that will be promoted through every subsequent environment; nothing gets rebuilt at staging or production, which is what makes 'the artifact you tested is the artifact you deploy' actually true rather than an assumption.
3. Generate the SBOM against that exact built image, capturing precisely what's in it.
4. Scan the image for vulnerabilities using the SBOM as the input, so the scan is checking exactly what was built, not a re-derived approximation.
5. Sign the image, binding its digest to this specific build's identity.
6. Promote the SAME signed, scanned artifact by digest (never by a mutable tag) through dev, staging, and production, with an automated gate at each promotion step re-verifying the signature and checking whether any NEW vulnerability has been disclosed against this image's dependencies since the last check (since a clean scan yesterday doesn't guarantee a clean scan today if a new CVE was published in the interim).

Gating criteria for promotion

Promotion from staging to production should require the signature verification to pass, no new CRITICAL vulnerability to have appeared since the build-time scan, and (for this scale of change, 200 microservices) an optional manual approval gate specifically for production, even when every automated check passes, giving a human a final checkpoint for a change affecting a service with real production traffic.

Incremental rollout into an existing pipeline

Rolling this out across 200 already-existing microservices should start with the least risky, lowest-traffic service first, validating the whole signed, scanned promotion chain works end to end before mandating it org-wide, then expanding service by service rather than flipping every pipeline over simultaneously, since a bug in the new promotion logic discovered against one low-traffic service is far cheaper than discovering it against all 200 at once.

Tooling

A concrete stack here might be GitHub Actions or Tekton as the orchestrator, Trivy or Snyk for scanning, Syft for SBOM generation, and cosign for signing; the specific tool choices matter less than the discipline of promoting one signed artifact by digest rather than rebuilding at each stage.

Trade-offs

Promoting by digest rather than rebuilding at each environment adds a small amount of pipeline complexity (the registry and deployment tooling both need to reference an immutable digest rather than a convenient, mutable tag like latest or staging), but it's what actually guarantees the artifact tested in staging is byte-for-byte the same one running in production, which rebuilding at each stage cannot guarantee even with identical source.

For a regulated financial customer, the three approaches trade off differently on exactly the dimensions that regulation cares most about: auditability, revocation speed, and operational maturity required to run them safely.

The three approaches

A, a centralized Vault with dynamic credentials. HashiCorp Vault (or an equivalent) issues short-lived, scoped credentials on demand rather than storing static secrets; every issuance is logged centrally, and a compromised credential expires on its own within minutes even if nobody notices the compromise. Operational complexity is the highest of the three: running Vault itself well (unsealing, high availability, backend storage) is a real operational commitment, and every pipeline needs a supported authentication method into it (OIDC (OpenID Connect), AppRole, or similar). Developer experience has the highest upfront cost of the three: a team has to integrate its pipeline with Vault's auth method before it can fetch a single secret, but once that integration exists, day-to-day use is transparent (a developer never sees or handles the actual credential value at all).

B, platform-native sealed secrets or cluster secret stores. Secrets are encrypted at rest and only decryptable by the specific cluster or platform they're deployed to (Kubernetes sealed-secrets, or a cloud-native equivalent). Operational complexity is lower than running Vault, since the platform already exists and this uses its native mechanism, but rotation is typically a manual or semi-automated process rather than the always-short-lived credentials of approach A, and auditability depends heavily on the platform's own audit logging maturity. Developer experience is generally the easiest of the three to adopt, since it reuses tooling (kubectl, the platform's own CLI) developers already use for everything else, at the cost of the weaker rotation story above.

C, encrypted variables stored in the CI system itself. The lowest operational complexity of the three (no additional infrastructure to run), but the weakest security guarantees: the CI system itself becomes a single point of both storage and access control, credentials are typically long-lived, and audit trail quality varies widely by CI provider. Developer experience is the simplest of all three to set up (paste a value into the CI system's own secrets UI, reference it by name), which is exactly why teams default to it even though it's the weakest option on every other dimension.

Recommendation for a regulated financial customer

Approach A. Dynamic, short-lived credentials directly satisfy the kind of access-review and least-privilege requirements a financial regulator will ask about (every credential issuance is individually logged and every credential expires whether or not it's ever explicitly revoked), and the centralized audit trail is exactly the evidence an auditor wants to see. The higher operational cost, including the steeper initial developer-experience cost of integrating every pipeline with Vault's auth method, is the honest trade-off: this customer needs the operational maturity to run Vault (or accept a managed Vault offering) reliably, including its own high-availability and disaster-recovery story, since an outage in the secrets layer becomes an outage in every pipeline that depends on it.

What would change the recommendation

For a smaller, less-regulated customer with a single small platform team, approach B or even C might be the right call precisely because the operational cost of running Vault well, and the developer-experience cost of onboarding every pipeline to it, would exceed the actual risk reduction it buys; the recommendation is a function of the customer's regulatory obligations and operational maturity, not a universal ranking of the three options.

A provenance attestation is fundamentally a signed statement, so the implementation has two halves that both have to be exactly right: building a deterministic payload to sign, and signing it in a way a verifier can independently reproduce and check.

package main

import (
	"crypto"
	"crypto/rand"
	"crypto/rsa"
	"crypto/sha256"
	"crypto/x509"
	"encoding/base64"
	"encoding/json"
	"encoding/pem"
	"fmt"
	"os"
	"time"
)

type FileDigest struct {
	Path   string `json:"path"`
	Sha256 string `json:"sha256"`
}

// Field order is fixed by the struct definition, which is what makes
// json.Marshal deterministic for this payload: the same logical content
// always produces the same bytes to sign and to re-verify against.
type Attestation struct {
	Inputs    []FileDigest `json:"inputs"`
	Outputs   []FileDigest `json:"outputs"`
	Timestamp string       `json:"timestamp"`
	BuilderID string       `json:"builderId"`
	Signature string       `json:"signature"`
}

func signPayload(payload []byte, priv *rsa.PrivateKey) (string, error) {
	digest := sha256.Sum256(payload)
	sig, err := rsa.SignPKCS1v15(rand.Reader, priv, crypto.SHA256, digest[:])
	if err != nil {
		return "", fmt.Errorf("signing attestation payload: %w", err)
	}
	return base64.StdEncoding.EncodeToString(sig), nil
}

func loadPrivateKey(pemPath string) (*rsa.PrivateKey, error) {
	data, err := os.ReadFile(pemPath)
	if err != nil {
		return nil, fmt.Errorf("reading key file: %w", err)
	}
	block, _ := pem.Decode(data)
	if block == nil {
		return nil, fmt.Errorf("no PEM block found in %s", pemPath)
	}
	key, err := x509.ParsePKCS1PrivateKey(block.Bytes)
	if err != nil {
		return nil, fmt.Errorf("parsing private key: %w", err)
	}
	return key, nil
}

func buildAttestation(inputsPath, outputsPath, keyPath string) (*Attestation, error) {
	inputsRaw, err := os.ReadFile(inputsPath)
	if err != nil {
		return nil, fmt.Errorf("reading inputs file: %w", err)
	}
	var inputs []FileDigest
	if err := json.Unmarshal(inputsRaw, &inputs); err != nil {
		return nil, fmt.Errorf("parsing inputs JSON: %w", err)
	}

	outputsRaw, err := os.ReadFile(outputsPath)
	if err != nil {
		return nil, fmt.Errorf("reading outputs file: %w", err)
	}
	var outputs []FileDigest
	if err := json.Unmarshal(outputsRaw, &outputs); err != nil {
		return nil, fmt.Errorf("parsing outputs JSON: %w", err)
	}

	builderID := os.Getenv("BUILDER_ID")
	if builderID == "" {
		return nil, fmt.Errorf("BUILDER_ID environment variable is not set")
	}

	priv, err := loadPrivateKey(keyPath)
	if err != nil {
		return nil, err
	}

	unsigned := Attestation{
		Inputs:    inputs,
		Outputs:   outputs,
		Timestamp: time.Now().UTC().Format(time.RFC3339),
		BuilderID: builderID,
	}

	payload, err := json.Marshal(unsigned)
	if err != nil {
		return nil, fmt.Errorf("canonicalizing payload: %w", err)
	}

	sig, err := signPayload(payload, priv)
	if err != nil {
		return nil, err
	}

	unsigned.Signature = sig
	return &unsigned, nil
}

func main() {
	if len(os.Args) != 4 {
		fmt.Fprintln(os.Stderr, "usage: provenance <inputs.json> <outputs.json> <private_key.pem>")
		os.Exit(2)
	}
	att, err := buildAttestation(os.Args[1], os.Args[2], os.Args[3])
	if err != nil {
		fmt.Fprintf(os.Stderr, "error: %v\n", err)
		os.Exit(1)
	}
	out, _ := json.MarshalIndent(att, "", "  ")
	fmt.Println(string(out))
}

Deterministic JSON serialization

The struct's field order is fixed at compile time by its definition, so json.Marshal always emits fields in the same order for the same logical content; this is why the payload is built as a typed Attestation struct rather than a map[string]interface{}, since Go's JSON encoding of a map sorts keys alphabetically by default but a struct preserves declaration order, either of which is deterministic on its own, but mixing the two within one payload risks a verifier reconstructing a different byte layout than the signer used. The signature field itself is excluded from the signed payload (set to empty string, or omitted, at signing time) since the signature obviously can't be part of what it's signing over.

Verified

Compiled with go build. Generated a real RSA keypair with OpenSSL, ran the program to produce a signed attestation, then wrote an independent Go verifier that re-reads the attestation, strips the signature field, re-marshals the remaining struct exactly as the signer did, and calls rsa.VerifyPKCS1v15 against the public key: verification succeeded. A tamper test, flipping one character in an input's SHA-256 digest and re-running verification, correctly failed with a signature-mismatch error, confirming the scheme actually detects tampering rather than passing regardless of content.

Trade-offs

Using PKCS1v15 padding and a single RSA key here is simple and fully supported by the Go standard library alone (no external dependency), matching the question's constraint; a production system would more likely use Sigstore's keyless signing (short-lived, OIDC (OpenID Connect)-backed certificates) to avoid the operational burden of protecting a long-lived private key file: that file has to be generated once and then protected for its entire lifetime, encrypted at rest, access-restricted, rotated on a schedule, and revoked immediately if it is ever exposed, and if it does leak, whoever holds it can forge valid attestations indefinitely, or at least until the leak is discovered and the key revoked, with no built-in record of who actually produced a given signature. Keyless signing removes that persisted secret entirely: at sign time the CI runner exchanges a short-lived OIDC token (proof that this exact signing request came from this exact workflow run) for a certificate from a public certificate authority (Sigstore's Fulcio), signs with an ephemeral key that is discarded the moment signing completes, and records the signature alongside the certificate and a transparency-log entry (Rekor) that anyone can later audit. There is no long-lived key file to steal, back up, or rotate, at the cost of depending on the OIDC identity provider and Sigstore's infrastructure being reachable at sign time. Container image signing, discussed separately in this topic, makes the identical trade-off: a persisted key you must guard forever versus an ephemeral, OIDC-backed identity with nothing to steal.