CI/CD Pipeline Design and Architecture Questions

Direct answer

Trunk-based development (short-lived branches merging to a single trunk frequently, often multiple times a day) keeps the pipeline simple and CI throughput high, at the cost of needing strong automated testing and usually feature flags to hide incomplete work. GitFlow-style long-lived feature and release branches give more isolation for large, risky changes, at the cost of expensive merges, more parallel pipeline runs to maintain, and slower, more complex releases.

Structured elaboration

With trunk-based development, every merge to main is small and frequent, so the pipeline's job is straightforward: validate each small change quickly and keep main always releasable. CI throughput tends to be high (many small, fast pipeline runs) and merge conflicts are rare because branches don't live long enough to diverge much. The cost is that you can't hide an in-progress, half-built feature behind a long-lived branch; you need feature flags to merge incomplete work into main safely, and your test suite has to be trustworthy enough that a fast merge-time pipeline can actually catch regressions, because there's no lengthy release-branch stabilization period to catch what the automated checks missed.

With GitFlow (or similar long-lived branch models: develop, release branches, hotfix branches), each branch effectively needs its own pipeline configuration and its own build/test runs, multiplying the CI surface area. Merges from a long-lived feature branch back into develop or main are larger and more likely to conflict, and the pipeline has to handle merge-back validation as a first-class event, not just individual commits. The benefit is genuine isolation for large or risky work, and a natural place (the release branch) to stabilize before a release without disrupting ongoing development on develop.

Rollback complexity differs too: with trunk-based development and small frequent merges, a bad change is usually one small commit, so reverting it (or rolling forward with a fix) is fast and low-risk. With long-lived release branches, a bad release can bundle many changes together, making it harder to isolate and revert just the offending one.

Trigger strategy: monorepo versus multi-repo. In a monorepo, a single trunk-based merge event still has to trigger only the pipelines for the services actually affected by that commit, so the trigger layer needs path-based or dependency-graph-based filtering; without it, a trunk-based monorepo pipeline naively rebuilds and retests everything on every merge, which quickly destroys the fast-feedback benefit trunk-based development is supposed to provide. In a multi-repo layout, each repository's own push/PR trigger is already scoped to that one service for free, but a change to a shared library published from one repository doesn't automatically re-trigger every dependent repository's pipeline the way a single monorepo merge event would; that has to be handled explicitly, typically via a webhook from the shared library's publish step or an automated dependency-bump PR into each consumer, which is inherently slower and less atomic than the monorepo case. This is one reason teams doing trunk-based development at scale with many interdependent services often lean toward a monorepo: it keeps the 'one merge, one coordinated trigger fan-out' property that GitFlow-style long-lived branches and multi-repo layouts both make harder to get for free.

Worked example

A team of 40 engineers shipping a consumer product with strong test coverage and feature-flag infrastructure adopts trunk-based development: everyone merges small changes to main multiple times a day, CI runs in under 10 minutes, and incomplete features ship dark behind flags until they're ready to turn on. Contrast a team maintaining an on-premise enterprise product with quarterly releases and customers who need release notes and a stabilization window: a release-branch model fits better, because the business process (not just the pipeline) genuinely needs a period where only bug fixes land before a release ships.

Trade-offs and pitfalls

The most common mistake is picking trunk-based development because it's the trendier answer without having the test coverage or feature-flag discipline to back it up, which just means broken code lands on main more often. The opposite mistake is defaulting to long-lived branches out of habit when the team's actual release cadence and risk profile would be better served by trunk-based development with flags; the tell is a team that dreads 'merge day' because branches have diverged so far that the merge itself is the risky event, not the code.

A provenance attestation is fundamentally a signed statement, so the implementation has two halves that both have to be exactly right: building a deterministic payload to sign, and signing it in a way a verifier can independently reproduce and check.

package main

import (
	"crypto"
	"crypto/rand"
	"crypto/rsa"
	"crypto/sha256"
	"crypto/x509"
	"encoding/base64"
	"encoding/json"
	"encoding/pem"
	"fmt"
	"os"
	"time"
)

type FileDigest struct {
	Path   string `json:"path"`
	Sha256 string `json:"sha256"`
}

// Field order is fixed by the struct definition, which is what makes
// json.Marshal deterministic for this payload: the same logical content
// always produces the same bytes to sign and to re-verify against.
type Attestation struct {
	Inputs    []FileDigest `json:"inputs"`
	Outputs   []FileDigest `json:"outputs"`
	Timestamp string       `json:"timestamp"`
	BuilderID string       `json:"builderId"`
	Signature string       `json:"signature"`
}

func signPayload(payload []byte, priv *rsa.PrivateKey) (string, error) {
	digest := sha256.Sum256(payload)
	sig, err := rsa.SignPKCS1v15(rand.Reader, priv, crypto.SHA256, digest[:])
	if err != nil {
		return "", fmt.Errorf("signing attestation payload: %w", err)
	}
	return base64.StdEncoding.EncodeToString(sig), nil
}

func loadPrivateKey(pemPath string) (*rsa.PrivateKey, error) {
	data, err := os.ReadFile(pemPath)
	if err != nil {
		return nil, fmt.Errorf("reading key file: %w", err)
	}
	block, _ := pem.Decode(data)
	if block == nil {
		return nil, fmt.Errorf("no PEM block found in %s", pemPath)
	}
	key, err := x509.ParsePKCS1PrivateKey(block.Bytes)
	if err != nil {
		return nil, fmt.Errorf("parsing private key: %w", err)
	}
	return key, nil
}

func buildAttestation(inputsPath, outputsPath, keyPath string) (*Attestation, error) {
	inputsRaw, err := os.ReadFile(inputsPath)
	if err != nil {
		return nil, fmt.Errorf("reading inputs file: %w", err)
	}
	var inputs []FileDigest
	if err := json.Unmarshal(inputsRaw, &inputs); err != nil {
		return nil, fmt.Errorf("parsing inputs JSON: %w", err)
	}

	outputsRaw, err := os.ReadFile(outputsPath)
	if err != nil {
		return nil, fmt.Errorf("reading outputs file: %w", err)
	}
	var outputs []FileDigest
	if err := json.Unmarshal(outputsRaw, &outputs); err != nil {
		return nil, fmt.Errorf("parsing outputs JSON: %w", err)
	}

	builderID := os.Getenv("BUILDER_ID")
	if builderID == "" {
		return nil, fmt.Errorf("BUILDER_ID environment variable is not set")
	}

	priv, err := loadPrivateKey(keyPath)
	if err != nil {
		return nil, err
	}

	unsigned := Attestation{
		Inputs:    inputs,
		Outputs:   outputs,
		Timestamp: time.Now().UTC().Format(time.RFC3339),
		BuilderID: builderID,
	}

	payload, err := json.Marshal(unsigned)
	if err != nil {
		return nil, fmt.Errorf("canonicalizing payload: %w", err)
	}

	sig, err := signPayload(payload, priv)
	if err != nil {
		return nil, err
	}

	unsigned.Signature = sig
	return &unsigned, nil
}

func main() {
	if len(os.Args) != 4 {
		fmt.Fprintln(os.Stderr, "usage: provenance <inputs.json> <outputs.json> <private_key.pem>")
		os.Exit(2)
	}
	att, err := buildAttestation(os.Args[1], os.Args[2], os.Args[3])
	if err != nil {
		fmt.Fprintf(os.Stderr, "error: %v\n", err)
		os.Exit(1)
	}
	out, _ := json.MarshalIndent(att, "", "  ")
	fmt.Println(string(out))
}

Deterministic JSON serialization

The struct's field order is fixed at compile time by its definition, so json.Marshal always emits fields in the same order for the same logical content; this is why the payload is built as a typed Attestation struct rather than a map[string]interface{}, since Go's JSON encoding of a map sorts keys alphabetically by default but a struct preserves declaration order, either of which is deterministic on its own, but mixing the two within one payload risks a verifier reconstructing a different byte layout than the signer used. The signature field itself is excluded from the signed payload (set to empty string, or omitted, at signing time) since the signature obviously can't be part of what it's signing over.

Verified

Compiled with go build. Generated a real RSA keypair with OpenSSL, ran the program to produce a signed attestation, then wrote an independent Go verifier that re-reads the attestation, strips the signature field, re-marshals the remaining struct exactly as the signer did, and calls rsa.VerifyPKCS1v15 against the public key: verification succeeded. A tamper test, flipping one character in an input's SHA-256 digest and re-running verification, correctly failed with a signature-mismatch error, confirming the scheme actually detects tampering rather than passing regardless of content.

Trade-offs

Using PKCS1v15 padding and a single RSA key here is simple and fully supported by the Go standard library alone (no external dependency), matching the question's constraint; a production system would more likely use Sigstore's keyless signing (short-lived, OIDC (OpenID Connect)-backed certificates) to avoid the operational burden of protecting a long-lived private key file: that file has to be generated once and then protected for its entire lifetime, encrypted at rest, access-restricted, rotated on a schedule, and revoked immediately if it is ever exposed, and if it does leak, whoever holds it can forge valid attestations indefinitely, or at least until the leak is discovered and the key revoked, with no built-in record of who actually produced a given signature. Keyless signing removes that persisted secret entirely: at sign time the CI runner exchanges a short-lived OIDC token (proof that this exact signing request came from this exact workflow run) for a certificate from a public certificate authority (Sigstore's Fulcio), signs with an ephemeral key that is discarded the moment signing completes, and records the signature alongside the certificate and a transparency-log entry (Rekor) that anyone can later audit. There is no long-lived key file to steal, back up, or rotate, at the cost of depending on the OIDC identity provider and Sigstore's infrastructure being reachable at sign time. Container image signing, discussed separately in this topic, makes the identical trade-off: a persisted key you must guard forever versus an ephemeral, OIDC-backed identity with nothing to steal.

Direct answer

When the CI/CD control plane itself suffers catastrophic data loss, the immediate priority is restoring some minimal ability to ship urgent changes, using whatever fallback exists, while the full restoration leans on the fact that pipelines are defined as code and can be reconstructed from version control rather than depending on the lost control-plane state. Designing for this scenario in advance means concrete RPO (recovery point objective, how much recent state you're willing to lose) and RTO (recovery time objective, how long restoration is allowed to take) targets and periodic drills, not just a documented plan nobody's tested.

Structured elaboration

Immediate response. First, confirm the actual scope of the loss (is it truly the control plane's own state, or a symptom of something else); then assess what's actually needed right now: can a genuinely urgent fix be shipped through some manual or degraded path (a documented emergency deploy procedure that doesn't depend on the normal pipeline) while the control plane itself is being restored. This is not the moment to start designing a new process; it's the moment to execute whatever emergency path was defined in advance.

Failover options. If a secondary or standby control-plane instance exists (as part of the HA design), failing over to it is the fastest path back to normal operation. If not, the fallback is restoring from backup, or, for the most urgent single changes, an entirely manual process (direct deploy access, gated by extra human review given the reduced safety net) as a bridge until the pipeline itself is back.

Full restoration leveraging pipeline-as-code. Because pipeline definitions live in version control (not solely in the control plane's own database), restoring the control plane doesn't require recovering the pipeline logic itself from a backup of the control plane; it can be reconstructed by pointing a restored (or newly provisioned) control plane at the existing Git repositories. What genuinely needs to be recovered from backup is the control plane's own operational state: credentials, job history, and any configuration that isn't expressed as pipeline-as-code.

Designing DR in advance. A concrete RPO (how much recent state, like the last few minutes of job history or credential changes, you're willing to lose) and RTO (how long restoration is allowed to take) target forces the actual design decisions: an RPO of 15 minutes means state needs to be replicated or backed up at least that frequently, not just backed up nightly; an RTO of 2 hours means the restoration procedure (not just the backup) needs to actually be tested and known to complete within that window, not estimated optimistically.

Artifact storage DR, specifically. Pipeline definitions being safely recoverable from Git covers the logic side of the platform, but not the other genuinely stateful piece the question calls out: the artifacts themselves (container images, build outputs, packages) sitting in a registry or artifact store. That needs its own explicit DR design, not an assumption that the pipeline-as-code story covers it: cross-region or cross-provider replication of the registry, or at minimum a documented, tested process to re-derive the currently-deployed artifacts by re-running the pipeline against the exact commit that produced them. That re-derivation only works if every build already practices version traceability: embedding the source commit SHA and a build number into the artifact's own recorded version at build time (for example 1.4.0+a1b2c3d, where a1b2c3d is the commit that produced it), rather than deploying under a mutable tag like latest that carries no link back to source. With that discipline already in place before the outage, 'what commit produced the artifact running in production right now' is a lookup against the deployed version string, not a guess, so rebuilding from that exact commit reliably reproduces the lost artifact; without it, losing the artifact store is unrecoverable because there is no record of what to rebuild. Artifact storage gets its own RPO (how many recent pushes you're willing to lose, which can differ from the control plane's own RPO) and RTO (how long until artifacts are pullable again), each validated by an actual restore test, since losing the artifact store means production can't be redeployed even if the control plane and pipeline definitions come back perfectly intact.

DR drills. A documented DR plan that's never been executed is a hypothesis, not a verified capability; periodic drills (actually failing over to the standby, or actually restoring from backup into an isolated environment and confirming pipelines run correctly) are what convert the plan into something you can trust when it matters, and they reliably surface gaps (a credential that wasn't included in the backup, a manual step nobody remembered) that a plan reviewed only on paper wouldn't catch.

Worked example

A control-plane outage strikes; the on-call engineer confirms via the documented runbook that this is a real control-plane data-loss event, not a transient blip, and immediately triggers failover to the standby instance (part of the HA design), restoring build capability within the RTO's target window. In parallel, they follow the documented emergency-deploy procedure to ship one specific, already-reviewed, urgent fix manually while the standby is stabilizing, since that fix couldn't wait even for the fast failover path. Full restoration reconstructs pipeline definitions from the existing Git repositories (no data lost there, since they were never solely stored in the control plane), while the operational state (credentials, recent job history) is restored from a backup taken within the 15-minute RPO window. A DR drill run the previous quarter had already surfaced and fixed a gap where one credential type wasn't included in the automated backup, which is exactly the kind of gap a drill catches that a paper plan doesn't. The same drill also exercises the separately-replicated artifact registry (a secondary-region mirror with its own 30-minute RPO), confirming production images stay pullable even when the primary registry region is the one that's down.

Trade-offs and pitfalls

The most common mistake is a DR plan that exists only as documentation and has never actually been executed, which reliably has gaps (a missed credential, an assumption about restoration time that turns out to be wildly optimistic) that only surface during a real incident, when discovering them is far more costly. The second is setting an RPO/RTO target without actually validating the backup and restoration mechanism can meet it, which means the target is aspirational rather than a real, tested commitment.

Direct answer

Push-based CI/CD deployment (the CI server actively pushes changes into the target environment) and pull-based GitOps (an in-cluster agent reconciles state by pulling from Git) differ mainly on where deployment credentials live, how auditable and self-correcting the system is, and how much operational complexity you take on. Neither is universally better; the right choice, and often a hybrid, depends on your security posture, team structure, and existing tooling investment.

Structured elaboration

Security. Push-based deployment means the CI system needs credentials with write access to the target environment, which makes the CI system itself a high-value target: compromise the pipeline, and you can potentially reach production. Pull-based GitOps keeps deployment credentials scoped inside the environment itself, and the CI system never needs write access to production at all, only to the Git repository (or artifact registry) that the reconciler watches, which meaningfully shrinks the blast radius of a compromised CI pipeline.

Traceability and auditability. GitOps gets a strong default here: every change to the desired state is a Git commit, so git log on the config repository is a complete, tamper-evident audit trail of every deployment decision. Push-based pipelines can achieve similar auditability, but it takes more deliberate logging and isn't automatic the way Git history is.

Rollback semantics. GitOps rollback is a Git revert, which the reconciler picks up and applies automatically; push-based rollback usually means re-running a previous pipeline deployment step, which works fine but depends on the pipeline itself still being able to run (an outage in the CI system can block a push-based rollback in a way it wouldn't block a GitOps one, since the reconciler doesn't depend on CI availability to apply an already-committed rollback).

Developer autonomy and operational overhead. Push-based pipelines are conceptually simpler for a team already comfortable with CI/CD: one system, one place to look. GitOps adds real operational complexity (a reconciler to run and monitor, a separate config repository to manage, drift-detection semantics to understand) that pays off mainly at scale, when the security and self-healing benefits outweigh the added moving parts for a single small team.

Concrete integration pattern: CI builds, GitOps deploys. A common hybrid keeps CI push-based for the build side and GitOps for deployment. Concretely: CI builds and publishes an image identified by an immutable digest, exactly as in a push-based pipeline. To communicate that new version to the GitOps tooling, CI does not call the cluster; instead, the last step of the CI pipeline opens an automated commit or pull request against the GitOps config repository that bumps the image tag/digest referenced in the relevant Kubernetes manifest (or, for Flux specifically, an image-automation controller can watch the registry directly and open that commit itself, with no CI involvement at all, using its own image policy). Either way, the reconciler (ArgoCD or Flux) picks up the new commit on its normal poll or webhook-triggered sync and applies it; CI's credentials never touch the cluster.

Detecting and handling drift. The reconciler continuously runs a diff between the live cluster state and the state declared in Git, on a poll interval (typically tens of seconds) or triggered by a webhook from the config repository. When live state diverges from declared state (a manual kubectl edit, a rollout triggered outside Git, another controller mutating a resource), the reconciler marks the application 'OutOfSync' and, depending on its configured sync policy, either auto-heals by reapplying the declared state (the common production default) or only alerts and waits for a human to manually trigger a sync, for environments where auto-reverting a live change is considered too risky to do unattended. Either policy needs monitoring on the reconciler's own sync/health status, not just on the application, because a reconciler that's down or stuck itself won't correct drift even though the config repository looks correct.

When hybrid makes sense. This pattern gives you GitOps's security and drift-correction benefits for the sensitive part (who can change what's running in production) without needing GitOps for the build side, where push-based tooling is simpler and perfectly adequate. Migrating a team fully reliant on push-based deployment to this hybrid typically starts with the highest-value, most security-sensitive services first, proving the pattern before rolling it out broadly.

Worked example

For a mid-size enterprise: internal tooling teams with lower deployment risk and a strong existing push-based pipeline might reasonably keep push-based deployment, since the operational overhead of adding GitOps isn't clearly worth it for their risk profile. A team running customer-facing production services in a regulated environment is a stronger fit for GitOps: the audit trail is close to free (it's just Git history), credentials never leave the cluster, and drift correction catches unauthorized manual changes automatically, which a compliance team will specifically value.

Trade-offs and pitfalls

The most common mistake is adopting GitOps everywhere reflexively because it's the more modern-sounding pattern, without weighing the real added operational complexity against a team's actual risk profile and scale; for a small team with low-risk internal services, push-based CI/CD can be the simpler, entirely adequate choice. The second is a genuine gap worth naming: GitOps's drift-correction is a double-edged sword, since an emergency manual fix applied directly to production during an incident will be silently reverted by the reconciler unless the fix is also committed to the config repository, which is a real operational surprise for a team encountering it for the first time.

Direct answer

Below is a declarative Jenkinsfile for a Java Maven project that checks out the code, validates it, runs unit tests, archives the built JAR, and publishes JUnit results.

Structured elaboration

A declarative pipeline for this is straightforward: an agent selecting a machine that has both Linux and Maven available, a small number of sequential stages (Validate, Test, Package), and a post block that always publishes test results and prints a short build summary regardless of whether the build passed, so failures are visible in the same place successes are.

Worked example

pipeline {
    agent { label 'linux && maven' }

    options {
        timestamps()
        timeout(time: 20, unit: 'MINUTES')
    }

    stages {
        stage('Checkout') {
            steps {
                checkout scm
            }
        }
        stage('Validate') {
            steps {
                sh 'mvn -B validate'
            }
        }
        stage('Test') {
            steps {
                sh 'mvn -B test'
            }
        }
        stage('Package') {
            steps {
                sh 'mvn -B package -DskipTests'
                archiveArtifacts artifacts: 'target/*.jar', fingerprint: true
            }
        }
    }

    post {
        always {
            junit testResults: 'target/surefire-reports/*.xml', allowEmptyResults: true
            echo "Build #${env.BUILD_NUMBER} on branch ${env.BRANCH_NAME}: result=${currentBuild.currentResult}, duration=${currentBuild.durationString}"
        }
    }
}

The -DskipTests on the package step is deliberate: tests already ran in the Test stage, so re-running them during packaging would waste time without adding signal. allowEmptyResults: true on the junit step keeps the pipeline from erroring out if, for some reason, no test reports were produced, while still surfacing a clear signal that something's off. The echo line after it is the required short build summary: it always runs (because it's in post { always {} }) and reads currentBuild.currentResult (reliable at this point in the pipeline, unlike earlier stages where the final result isn't determined yet) plus durationString, so anyone scanning the console log gets a one-line answer to 'did this pass, and how long did it take' without hunting through the full log.

A natural extension of this same pattern: after packaging, build a Docker image tagged with the Git commit SHA (docker build -t myservice:${env.GIT_COMMIT} .), authenticate to a registry using Jenkins' credentials binding (withCredentials([usernamePassword(credentialsId: 'ecr-creds', ...)]) or an AWS credentials plugin), and push the image, keeping the credential reference scoped to just the push step rather than exported for the whole pipeline.

Trade-offs and pitfalls

A common mistake is running mvn test as part of the package goal instead of as its own stage, which hides test failures inside a generic 'package failed' error instead of a clear 'tests failed' signal with JUnit results attached. Another is forgetting allowEmptyResults or an equivalent guard, which turns 'the test runner didn't produce a report' (a tooling bug) into a hard pipeline failure indistinguishable from a real test failure, making the failure harder to triage.