CI/CD Pipeline Design and Architecture Questions

A single workflow that builds for two architectures, generates an SBOM, scans, and signs needs each step to operate on the SAME image digest, so the SBOM accurately describes what was scanned, and what was scanned is exactly what gets signed.

name: build-sign-publish

on:
  push:
    branches: [main]

permissions:
  contents: read
  packages: write
  id-token: write   # required for cosign keyless (OIDC) signing

env:
  IMAGE: ghcr.io/${{ github.repository }}

jobs:
  build-scan-sign:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Set up QEMU
        uses: docker/setup-qemu-action@v3

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3

      - name: Log in to registry
        uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Build and push multi-arch image
        id: build
        uses: docker/build-push-action@v6
        with:
          context: .
          platforms: linux/amd64,linux/arm64
          push: true
          tags: ${{ env.IMAGE }}:${{ github.sha }}
          cache-from: type=gha
          cache-to: type=gha,mode=max

      - name: Generate SBOM with syft
        uses: anchore/sbom-action@v0
        with:
          image: ${{ env.IMAGE }}@${{ steps.build.outputs.digest }}
          format: cyclonedx-json
          output-file: sbom.cdx.json

      - name: Scan image with Trivy
        uses: aquasecurity/trivy-action@v0.24.0
        with:
          image-ref: ${{ env.IMAGE }}@${{ steps.build.outputs.digest }}
          severity: CRITICAL
          exit-code: '1'
          format: table

      - name: Install cosign
        uses: sigstore/cosign-installer@v3

      - name: Sign image keylessly
        run: cosign sign --yes "${IMAGE}@${DIGEST}"
        env:
          IMAGE: ${{ env.IMAGE }}
          DIGEST: ${{ steps.build.outputs.digest }}

      - name: Upload SBOM
        uses: actions/upload-artifact@v4
        with:
          name: sbom
          path: sbom.cdx.json

Why the digest, not the tag, threads through every step

Every downstream step (SBOM generation, scanning, signing) references ${{ steps.build.outputs.digest }}, the immutable content digest the build step itself produced, rather than the mutable :${{ github.sha }} tag; this guarantees the SBOM describes exactly what was scanned and exactly what gets signed, closing the gap a tag-based reference would leave open if the tag were somehow overwritten between steps.

Secure credential handling

Registry authentication uses the automatically-issued GITHUB_TOKEN, scoped to this repository and this workflow run, rather than a long-lived personal access token; cosign signing uses id-token: write permission to obtain a short-lived OIDC (OpenID Connect) token exchanged for a Sigstore certificate, meaning no signing key material is stored as a secret at all.

Verified

Parsed with PyYAML: valid YAML, ten steps confirmed in order, and permissions.id-token confirmed present as write, which cosign's keyless signing requires to obtain its OIDC token from the GitHub Actions runtime. Also checked every third-party action reference against the GitHub API's published tags: docker/setup-qemu-action@v3, docker/setup-buildx-action@v3, docker/login-action@v3, docker/build-push-action@v6, anchore/sbom-action@v0, and sigstore/cosign-installer@v3 all resolve to real tags. aquasecurity/trivy-action@0.24.0 did NOT resolve; the action only publishes v-prefixed tags, so the reference has been corrected to @v0.24.0 above.

Trade-offs

cache-to: type=gha,mode=max speeds up subsequent builds meaningfully but stores build-layer cache in GitHub's own Actions cache, which has its own size limits and retention policy; for a very large image, cache eviction under those limits can occasionally force a slower, cold rebuild, which is an acceptable trade for the typical case where caching saves far more time than it costs.

Scaling a policy-as-code enforcement mechanism to evaluate at both PR time and Kubernetes admission time, across an enterprise, means the evaluation engine itself becomes performance-critical infrastructure: a slow policy check at admission time blocks every single deployment until it responds.

Caching strategies for performance

Cache the result of evaluating an unchanged policy set against an unchanged input (the same image digest, the same set of policies) rather than re-evaluating from scratch on every admission request; since policies change far less often than deployments happen, a cache keyed on (policy-set version, input hash) can serve the overwhelming majority of repeated evaluations (the same base image redeployed many times across many services) without re-running the actual policy logic each time. For PR-time evaluation, cache per-commit results so re-running CI on an unchanged commit (a re-triggered pipeline) doesn't redundantly re-evaluate policies against the exact same input.

Testability and versioning of policies

Every policy needs its own automated test suite (does it correctly flag its intended violation, does it correctly pass its intended allowed case) that runs before the policy itself is deployed to the evaluation engine, exactly as discussed for the governance model above; at scale, this testing needs to run FAST, since a slow policy test suite becomes its own bottleneck for how quickly a legitimate policy fix can ship.

Mitigating false positives at scale

At enterprise scale, a false positive isn't just one annoyed developer, it's potentially thousands of blocked deployments across every team simultaneously if the false-positive policy is a shared, org-wide rule; this argues for a staged rollout for every new or changed policy (advisory mode across the whole org first, observing the real-world false-positive rate at scale before promoting to blocking), rather than trusting that a policy which passed its unit tests will also behave correctly against the full diversity of real production traffic.

Safe rollout strategy for changing or removing policies in production

Roll out a policy CHANGE the same way you'd roll out any other risky production change: to a small percentage of evaluation requests first (a canary of the policy engine's own traffic), monitoring the resulting allow/deny rate for an unexpected shift before expanding to full traffic. Removing a policy entirely needs its own deliberate process too, since a policy that's been silently relied upon (teams built workarounds assuming it would always block a certain pattern) can have removal cause unexpected effects if removed without notice.

Trade-offs

Caching by (policy-set version, input hash) introduces a small window where a policy change might not immediately apply to an in-flight cached decision, depending on cache invalidation timing; the mitigation is invalidating the entire cache immediately whenever the policy-set version changes, accepting a brief spike in evaluation load right after a policy update in exchange for correctness the rest of the time.

A provenance attestation is fundamentally a signed statement, so the implementation has two halves that both have to be exactly right: building a deterministic payload to sign, and signing it in a way a verifier can independently reproduce and check.

package main

import (
	"crypto"
	"crypto/rand"
	"crypto/rsa"
	"crypto/sha256"
	"crypto/x509"
	"encoding/base64"
	"encoding/json"
	"encoding/pem"
	"fmt"
	"os"
	"time"
)

type FileDigest struct {
	Path   string `json:"path"`
	Sha256 string `json:"sha256"`
}

// Field order is fixed by the struct definition, which is what makes
// json.Marshal deterministic for this payload: the same logical content
// always produces the same bytes to sign and to re-verify against.
type Attestation struct {
	Inputs    []FileDigest `json:"inputs"`
	Outputs   []FileDigest `json:"outputs"`
	Timestamp string       `json:"timestamp"`
	BuilderID string       `json:"builderId"`
	Signature string       `json:"signature"`
}

func signPayload(payload []byte, priv *rsa.PrivateKey) (string, error) {
	digest := sha256.Sum256(payload)
	sig, err := rsa.SignPKCS1v15(rand.Reader, priv, crypto.SHA256, digest[:])
	if err != nil {
		return "", fmt.Errorf("signing attestation payload: %w", err)
	}
	return base64.StdEncoding.EncodeToString(sig), nil
}

func loadPrivateKey(pemPath string) (*rsa.PrivateKey, error) {
	data, err := os.ReadFile(pemPath)
	if err != nil {
		return nil, fmt.Errorf("reading key file: %w", err)
	}
	block, _ := pem.Decode(data)
	if block == nil {
		return nil, fmt.Errorf("no PEM block found in %s", pemPath)
	}
	key, err := x509.ParsePKCS1PrivateKey(block.Bytes)
	if err != nil {
		return nil, fmt.Errorf("parsing private key: %w", err)
	}
	return key, nil
}

func buildAttestation(inputsPath, outputsPath, keyPath string) (*Attestation, error) {
	inputsRaw, err := os.ReadFile(inputsPath)
	if err != nil {
		return nil, fmt.Errorf("reading inputs file: %w", err)
	}
	var inputs []FileDigest
	if err := json.Unmarshal(inputsRaw, &inputs); err != nil {
		return nil, fmt.Errorf("parsing inputs JSON: %w", err)
	}

	outputsRaw, err := os.ReadFile(outputsPath)
	if err != nil {
		return nil, fmt.Errorf("reading outputs file: %w", err)
	}
	var outputs []FileDigest
	if err := json.Unmarshal(outputsRaw, &outputs); err != nil {
		return nil, fmt.Errorf("parsing outputs JSON: %w", err)
	}

	builderID := os.Getenv("BUILDER_ID")
	if builderID == "" {
		return nil, fmt.Errorf("BUILDER_ID environment variable is not set")
	}

	priv, err := loadPrivateKey(keyPath)
	if err != nil {
		return nil, err
	}

	unsigned := Attestation{
		Inputs:    inputs,
		Outputs:   outputs,
		Timestamp: time.Now().UTC().Format(time.RFC3339),
		BuilderID: builderID,
	}

	payload, err := json.Marshal(unsigned)
	if err != nil {
		return nil, fmt.Errorf("canonicalizing payload: %w", err)
	}

	sig, err := signPayload(payload, priv)
	if err != nil {
		return nil, err
	}

	unsigned.Signature = sig
	return &unsigned, nil
}

func main() {
	if len(os.Args) != 4 {
		fmt.Fprintln(os.Stderr, "usage: provenance <inputs.json> <outputs.json> <private_key.pem>")
		os.Exit(2)
	}
	att, err := buildAttestation(os.Args[1], os.Args[2], os.Args[3])
	if err != nil {
		fmt.Fprintf(os.Stderr, "error: %v\n", err)
		os.Exit(1)
	}
	out, _ := json.MarshalIndent(att, "", "  ")
	fmt.Println(string(out))
}

Deterministic JSON serialization

The struct's field order is fixed at compile time by its definition, so json.Marshal always emits fields in the same order for the same logical content; this is why the payload is built as a typed Attestation struct rather than a map[string]interface{}, since Go's JSON encoding of a map sorts keys alphabetically by default but a struct preserves declaration order, either of which is deterministic on its own, but mixing the two within one payload risks a verifier reconstructing a different byte layout than the signer used. The signature field itself is excluded from the signed payload (set to empty string, or omitted, at signing time) since the signature obviously can't be part of what it's signing over.

Verified

Compiled with go build. Generated a real RSA keypair with OpenSSL, ran the program to produce a signed attestation, then wrote an independent Go verifier that re-reads the attestation, strips the signature field, re-marshals the remaining struct exactly as the signer did, and calls rsa.VerifyPKCS1v15 against the public key: verification succeeded. A tamper test, flipping one character in an input's SHA-256 digest and re-running verification, correctly failed with a signature-mismatch error, confirming the scheme actually detects tampering rather than passing regardless of content.

Trade-offs

Using PKCS1v15 padding and a single RSA key here is simple and fully supported by the Go standard library alone (no external dependency), matching the question's constraint; a production system would more likely use Sigstore's keyless signing (short-lived, OIDC (OpenID Connect)-backed certificates) to avoid the operational burden of protecting a long-lived private key file: that file has to be generated once and then protected for its entire lifetime, encrypted at rest, access-restricted, rotated on a schedule, and revoked immediately if it is ever exposed, and if it does leak, whoever holds it can forge valid attestations indefinitely, or at least until the leak is discovered and the key revoked, with no built-in record of who actually produced a given signature. Keyless signing removes that persisted secret entirely: at sign time the CI runner exchanges a short-lived OIDC token (proof that this exact signing request came from this exact workflow run) for a certificate from a public certificate authority (Sigstore's Fulcio), signs with an ephemeral key that is discarded the moment signing completes, and records the signature alongside the certificate and a transparency-log entry (Rekor) that anyone can later audit. There is no long-lived key file to steal, back up, or rotate, at the cost of depending on the OIDC identity provider and Sigstore's infrastructure being reachable at sign time. Container image signing, discussed separately in this topic, makes the identical trade-off: a persisted key you must guard forever versus an ephemeral, OIDC-backed identity with nothing to steal.

At the scale of pipelines feeding multiple Kubernetes clusters across regions plus third-party SaaS integrations, the central design problem is that a single secrets store becomes both a latency bottleneck and a disaster-recovery single point of failure if every cluster fetches every secret on demand from one central location.

Architecture

flowchart TB
  Vault[Central Vault cluster, primary region]
  Vault -->|replicated, read-only| VaultDR[Vault DR replica, secondary region]
  Vault -->|per-region cache, short TTL| CacheA[Regional cache A]
  Vault -->|per-region cache, short TTL| CacheB[Regional cache B]
  CacheA --> ClusterA[K8s cluster, region A]
  CacheB --> ClusterB[K8s cluster, region B]
  Pipelines[CI/CD pipelines] -->|ephemeral OIDC-based creds| Vault
  SaaS[Third-party SaaS integrations] -->|scoped, rotated tokens| Vault

Signing and trust model

Each pipeline and each cluster authenticates to Vault using its own workload identity (an OIDC (OpenID Connect) token from the CI provider, or a Kubernetes service-account token via Vault's Kubernetes auth method) rather than a shared static credential, so no single leaked credential grants access across every consumer. Vault issues short-lived, dynamically-generated credentials scoped narrowly to what that specific pipeline or cluster needs, and every issuance is logged.

Replication versus on-demand retrieval, and caching

Secrets replicate to a per-region cache with a short TTL (minutes, not hours) rather than every pod in every cluster calling Vault directly on every access; this bounds both latency (a regional cache answers in single-digit milliseconds versus a cross-region call to Vault) and blast radius (a compromised regional cache exposes only that region's cached subset, not the whole secret store), while the short TTL keeps the cached copy from drifting too far from the source of truth after a rotation.

Audit logging, disaster recovery, and safe decommissioning

Every credential issuance, cache refresh, and secret access gets logged centrally regardless of which region served the request, giving one unified audit trail rather than N regional ones that have to be manually reconciled. Disaster recovery for the secrets layer itself means Vault's own storage backend is replicated to a standby region with a documented failover procedure, since an outage in secret issuance becomes an outage in every dependent pipeline and cluster. Decommissioning a rotated secret means the old value is invalidated at the source (Vault) and the short cache TTL guarantees every regional cache naturally expires the stale copy within minutes, without needing to explicitly purge every cache individually.

Trade-offs

The regional caching layer trades a small window of potential staleness (a secret rotated centrally takes up to one TTL period to propagate everywhere) for dramatically better latency and resilience to a transient network partition between a region and the central Vault cluster; for a secret where even a few minutes of staleness after rotation is unacceptable (an emergency, compromised-credential rotation), the design needs an explicit cache-invalidation push rather than waiting on the TTL to expire naturally.

For 200 microservices moving from a git commit to a production-ready image, the pipeline needs to run each check at the point where its cost of running is lowest and its signal is most useful, then promote the SAME built artifact forward rather than rebuilding at each stage.

Stage-by-stage walkthrough

flowchart LR
  Commit[Git commit] --> SCA[Static code analysis]
  SCA --> Test[Unit tests]
  Test --> Build[Build container image]
  Build --> SBOM[Generate SBOM]
  SBOM --> Scan[Vulnerability scan]
  Scan --> Sign[Sign image]
  Sign --> Dev[Promote to dev]
  Dev -->|same digest| Staging[Promote to staging]
  Staging -->|same digest, approval| Prod[Promote to production]

1. Static code analysis and unit tests run first, before a container is even built, since they're the cheapest checks and catch the most common class of bug fastest.
2. Build the image once. This is the single build that will be promoted through every subsequent environment; nothing gets rebuilt at staging or production, which is what makes 'the artifact you tested is the artifact you deploy' actually true rather than an assumption.
3. Generate the SBOM against that exact built image, capturing precisely what's in it.
4. Scan the image for vulnerabilities using the SBOM as the input, so the scan is checking exactly what was built, not a re-derived approximation.
5. Sign the image, binding its digest to this specific build's identity.
6. Promote the SAME signed, scanned artifact by digest (never by a mutable tag) through dev, staging, and production, with an automated gate at each promotion step re-verifying the signature and checking whether any NEW vulnerability has been disclosed against this image's dependencies since the last check (since a clean scan yesterday doesn't guarantee a clean scan today if a new CVE was published in the interim).

Gating criteria for promotion

Promotion from staging to production should require the signature verification to pass, no new CRITICAL vulnerability to have appeared since the build-time scan, and (for this scale of change, 200 microservices) an optional manual approval gate specifically for production, even when every automated check passes, giving a human a final checkpoint for a change affecting a service with real production traffic.

Incremental rollout into an existing pipeline

Rolling this out across 200 already-existing microservices should start with the least risky, lowest-traffic service first, validating the whole signed, scanned promotion chain works end to end before mandating it org-wide, then expanding service by service rather than flipping every pipeline over simultaneously, since a bug in the new promotion logic discovered against one low-traffic service is far cheaper than discovering it against all 200 at once.

Tooling

A concrete stack here might be GitHub Actions or Tekton as the orchestrator, Trivy or Snyk for scanning, Syft for SBOM generation, and cosign for signing; the specific tool choices matter less than the discipline of promoting one signed artifact by digest rather than rebuilding at each stage.

Trade-offs

Promoting by digest rather than rebuilding at each environment adds a small amount of pipeline complexity (the registry and deployment tooling both need to reference an immutable digest rather than a convenient, mutable tag like latest or staging), but it's what actually guarantees the artifact tested in staging is byte-for-byte the same one running in production, which rebuilding at each stage cannot guarantee even with identical source.