CI/CD Pipeline Design and Architecture Questions

Direct answer

This question is testing whether you own mistakes honestly and turn them into concrete, lasting process or tooling improvements, not whether you've never caused an outage. A strong answer names a real trigger, a real diagnosis process, and a specific change that prevents the same class of mistake, not just this exact one.

Structured elaboration

The interviewer is listening for: a credible, specific trigger (what pipeline change, and why did it cause the outage, described precisely enough to show you actually understood the mechanism, not just 'a bad deploy happened'); a real diagnosis narrative (how you or the team figured out the pipeline change was the cause, including any false leads you initially chased); a concrete mitigation in the moment (what you actually did to restore service, distinct from the longer-term fix); and, most importantly, a specific systemic change afterward that would have caught this class of problem earlier, not just fixed this one instance.

A weak answer stops at 'we rolled back and it was fine,' which describes the immediate mitigation but skips the part that actually demonstrates growth: what changed about the pipeline, the review process, or the testing strategy so the same shape of mistake is now caught automatically, before it ever reaches production again.

Worked example

A credible shape: 'A pipeline change I made added a new deployment step that skipped the smoke-test gate for a specific service, because I'd mentally modeled it as low-risk. It shipped a config change that silently broke the service's connection pool sizing under production load, which we didn't see in staging because staging's traffic volume never exercised the pool exhaustion path. We noticed within 15 minutes via error-rate alerting, rolled back to the previous deployment, and the immediate incident was over quickly. Afterward, I removed the smoke-test exception for that service (the actual mistake: assuming any service could be safely exempted from the standard gate), and separately added a load-shaped smoke test that exercises realistic concurrency, not just a single health-check request, specifically because staging's low-traffic smoke test wouldn't have caught this class of bug either.' This is credible because the mechanism is specific, the diagnosis is described honestly (including that staging didn't catch it, which is a real and common gap), and the fix addresses the actual root cause (an exemption that shouldn't have existed) rather than a surface-level patch.

Trade-offs and pitfalls

The most common weak answer blames the deployment or the tooling ('the pipeline just broke') rather than owning the specific decision that caused it, which reads as deflecting responsibility rather than demonstrating the self-awareness the question is actually probing for. A second common gap is describing a detailed incident but a vague, generic follow-up ('we improved our testing'), when a strong answer names the exact gap the incident revealed and the exact change that closed it.

Direct answer

A strong answer here has a specific, concrete before-and-after: what was actually slow or fragile before, exactly what technical and process changes you made, and a real measurement of impact, not a vague 'I made CI faster' story.

Structured elaboration

The interviewer is listening for four things. First, a genuine problem, described specifically enough to be credible: not 'the pipeline was slow' but something like 'PR feedback took 35 minutes because every PR ran the full 1,200-test integration suite serially.' Second, a deliberate diagnosis: how you figured out where the time or fragility actually came from, rather than guessing and hoping. Third, the actual change, described at a level of technical specificity that shows you did the work yourself (or led it) rather than describing something you read about. Fourth, a real measurement: a before/after number for something concrete (pipeline duration, deployment frequency, failure rate, lead time for changes), plus how you know the change didn't quietly cost you something else (like coverage, in a speed-focused change).

If getting buy-in from a skeptical team was part of the story, the strongest answers describe a concrete objection someone raised and how you addressed it with evidence rather than authority: showing a small pilot's results, running the old and new approach in parallel for a period to build confidence, or directly addressing the specific risk a skeptic named (often exactly the coverage-regression risk described in the CI-speed optimization question) rather than dismissing the concern.

Worked example

A credible shape: 'Our PR pipeline took 35 minutes because it ran the full integration suite on every PR. I profiled the suite and found 60% of test time came from tests that touched services unrelated to a typical PR's changes. I built a change-impact detector using our existing dependency manifest and moved to selective test execution on PRs, with the full suite still running on merge and nightly as a safety net. PR pipeline time dropped from 35 to 9 minutes. Before rolling it out broadly, I ran the selective and full suites in parallel for two weeks and confirmed the selective suite caught the same failures the full suite did, which is what got a skeptical senior engineer, who was worried about missed regressions, on board.' This is credible because it names a specific bottleneck, a specific technique, a specific number, and a specific way of addressing the specific objection raised, not a generic one.

Trade-offs and pitfalls

The most common weak answer is vague on all four dimensions: a generic problem ('CI was slow'), a generic fix ('we added caching'), a suspiciously round or unverifiable number ('we made it 10x faster'), and no mention of how coverage or correctness was protected during the change. The second common weakness is describing only the technical change and skipping the buy-in question entirely when it's explicitly asked; if the interviewer asks about convincing a skeptical team, they want to hear about persuasion and evidence, not another restatement of the technical work.

Direct answer

Hosted (SaaS-provided) CI runners trade cost predictability and low maintenance for less control: you get a managed fleet with no infrastructure to run, but limited access to internal network resources and less customization of hardware. Self-hosted runners flip that trade: more control, network access, and custom hardware (like GPUs), at the cost of you owning the maintenance, security patching, and scaling.

Structured elaboration

Cost predictability. Hosted runners are usually billed per minute of compute used, which is predictable at low-to-moderate volume but can become expensive at high volume, and cost scales linearly with usage with little room to optimize beyond reducing build time itself. Self-hosted runners have a fixed infrastructure cost (owned or reserved hardware) that's more predictable in aggregate but requires capacity planning; you're paying for peak capacity even during quiet periods unless you also build autoscaling.

Security boundaries. Hosted runners are, by design, ephemeral and isolated from your internal network, which is a security feature: a compromised hosted-runner job generally can't pivot into your internal infrastructure. Self-hosted runners, especially if placed inside your internal network for access to private resources (an internal database, an internal artifact registry), need careful isolation, because a compromised job on a self-hosted runner has a much larger potential blast radius.

Performance and custom hardware. Hosted runners typically offer a fixed menu of machine sizes and, on paid tiers, limited GPU options; if your builds need specific hardware (a particular GPU generation, unusually large memory, specialized accelerators), self-hosted is often the only practical option.

Maintenance overhead. Hosted runners require essentially none from you: the platform patches the OS, updates the toolchain images, and handles capacity. Self-hosted runners require you to patch, update, and scale the fleet yourself, which is real ongoing operational work, not a one-time setup cost.

A second, related axis is ephemeral versus long-lived runners, which applies mainly on the self-hosted side (hosted runners are effectively always ephemeral). Ephemeral (single-use, typically container-based) runners are destroyed after each job, which minimizes attack surface (nothing persists between jobs for an attacker to exploit) at the cost of a cold start on every job (no warm dependency or Docker layer cache carried over). Long-lived VM-based runners keep a warm cache between jobs, which is faster, but accumulate state over time (leftover files, drifted configuration) and represent a larger and longer-lived attack surface if compromised.

Worked example

A startup with moderate, spiky CI usage and no need for special hardware is well served by hosted runners: no infrastructure to maintain, and the per-minute cost at their volume is lower than the engineering time it would take to run their own fleet. A company doing GPU-heavy ML training as part of its pipeline, or one whose builds need access to an internal artifact mirror behind a firewall, is pushed toward self-hosted, ideally ephemeral (container-based, torn down after each job) to limit the security exposure of running inside the internal network, with a remote/warm dependency cache layered on top to offset the cold-start cost.

Trade-offs and pitfalls

The most common mistake is choosing self-hosted purely to save money on compute without accounting for the ongoing engineering time to patch, scale, and secure the fleet, which often costs more in practice than the hosted-runner bill it was meant to avoid. The second is running self-hosted runners as long-lived, un-isolated machines for convenience (faster warm builds) without recognizing that a compromised job on a long-lived runner has much more to steal (persisted credentials, cached artifacts from other jobs) than one on an ephemeral runner.

Direct answer

For a monorepo with a 10,000-module dependency DAG (directed acyclic graph, the dependency structure between modules), the incremental build system needs three pieces working together: a mapping from changed files to the targets that directly own them, a reverse-dependency walk that finds every target transitively affected by those direct changes, and content-addressable cache keys so machines that never built a given target before can still get a cache hit.

Structured elaboration

Detecting what changed. Diff the incoming commit against the base (the merge target or the previous build), producing a list of changed file paths. A precomputed file-to-target mapping (which target owns which files, maintained as part of the build configuration) turns that into a set of directly-changed targets.

Computing the minimal impacted set. A target that didn't change directly can still be affected if it depends on something that did. The correct computation is a reverse-dependency graph walk: build an index from each target to the targets that depend on it (the reverse of the normal forward dependency graph), then breadth-first from the directly-changed targets, following reverse edges outward, until no new targets are discovered. Every target visited (directly changed, plus everything downstream of it) is in the impacted set; everything else is provably unaffected and can be safely skipped.

Cache keys for compiled outputs. Each target's cache key should be a hash of everything that affects its output: its own source content, the pinned versions of its direct dependencies' outputs (not just their names, since 'depends on target X' isn't enough information if X's own output changed), and the relevant toolchain version. This is what makes the cache safe: two builds with an identical key are guaranteed to produce identical output, so serving a cached result instead of rebuilding is correct by construction, not just probably fine.

Remote execution and caching at scale. With 10,000 modules, the impacted set for a typical small change should be a small fraction of the total, but building even that fraction serially would still be slow; distributing the impacted targets across many remote workers (each pulling from a shared, content-addressable remote cache) is what makes wall-clock time scale with the size of the impacted set rather than the size of the whole repository.

Correctness and reproducibility under parallelism. The dependency graph itself is what makes safe parallelization possible: two targets can build concurrently only if neither is a (transitive) dependency of the other, so the build scheduler needs to respect the graph's partial order, not just fire off every impacted target at once and hope for the best.

Accuracy versus safety, and the fallback when confidence is low. The reverse-dependency walk is only as trustworthy as the file-to-target mapping and the declared dependency edges it's built from; if either is incomplete (a target reads a config file, or reaches another target's output through a path the build definition never declares), the impacted-set computation can silently under-include a target that actually needed re-testing, and the pipeline stays green while shipping an untested regression. That's the real accuracy-versus-safety trade-off: always rebuilding and retesting everything is maximally safe but throws away the whole speed benefit the incremental system exists to deliver, while trusting the impacted-set computation unconditionally is fast but only as safe as the graph's completeness. The practical answer is a confidence-gated fallback, not an all-or-nothing choice: run the incremental impacted-set build for the common case, but fall back to a full build and test run (or at least a broader, deliberately over-inclusive test suite) whenever confidence in the computation is genuinely low, for example on a merge to a protected branch, on a periodic nightly cadence regardless of what changed that day, whenever the dependency graph or file-ownership mapping itself was recently edited, or when a target's declared dependencies look unusually sparse for its size. This way, a wrong or incomplete impacted-set computation gets caught by the periodic full run within a bounded window, instead of silently understating risk on every single change indefinitely.

Worked example

from collections import deque

def minimal_impacted_set(changed_files, file_to_targets, target_deps):
    # target_deps[target] = set of targets it depends on (edges point TO dependencies)
    reverse_deps = {}
    for target, deps in target_deps.items():
        for dep in deps:
            reverse_deps.setdefault(dep, set()).add(target)

    directly_changed = set()
    for f in changed_files:
        directly_changed |= file_to_targets.get(f, set())

    impacted = set(directly_changed)
    queue = deque(directly_changed)
    while queue:
        t = queue.popleft()
        for consumer in reverse_deps.get(t, set()):
            if consumer not in impacted:
                impacted.add(consumer)
                queue.append(consumer)
    return impacted

On a small representative graph (checkout and inventory depend on a shared common_auth library, payments depends on both common_auth and ledger), changing only common_auth's source correctly returns {common_auth, checkout, inventory, payments} (every direct and transitive consumer), while changing ledger correctly returns only {ledger, payments}, explicitly excluding checkout and inventory, which don't depend on ledger even transitively. A change touching an unrelated leaf target returns just that one target. This is O(V + E) in the size of the dependency graph (a standard BFS), independent of how many of the 10,000 modules are actually unaffected.

Trade-offs and pitfalls

The most common correctness bug is computing only direct impact (which targets own a changed file) and skipping the reverse-dependency walk entirely, which silently under-tests: a change to a widely-depended-on shared library would only rebuild itself, not the dozens of consumers that actually need re-validating. The second common bug is a cache key that hashes a dependency's name instead of its output content, which can serve a stale cached result for a target whose dependency changed, because the key didn't actually change even though the true build inputs did. Both bugs fail silently, which is exactly why they're dangerous: the pipeline goes faster and stays green, right up until a regression that should have been caught ships.

A provenance attestation is fundamentally a signed statement, so the implementation has two halves that both have to be exactly right: building a deterministic payload to sign, and signing it in a way a verifier can independently reproduce and check.

package main

import (
	"crypto"
	"crypto/rand"
	"crypto/rsa"
	"crypto/sha256"
	"crypto/x509"
	"encoding/base64"
	"encoding/json"
	"encoding/pem"
	"fmt"
	"os"
	"time"
)

type FileDigest struct {
	Path   string `json:"path"`
	Sha256 string `json:"sha256"`
}

// Field order is fixed by the struct definition, which is what makes
// json.Marshal deterministic for this payload: the same logical content
// always produces the same bytes to sign and to re-verify against.
type Attestation struct {
	Inputs    []FileDigest `json:"inputs"`
	Outputs   []FileDigest `json:"outputs"`
	Timestamp string       `json:"timestamp"`
	BuilderID string       `json:"builderId"`
	Signature string       `json:"signature"`
}

func signPayload(payload []byte, priv *rsa.PrivateKey) (string, error) {
	digest := sha256.Sum256(payload)
	sig, err := rsa.SignPKCS1v15(rand.Reader, priv, crypto.SHA256, digest[:])
	if err != nil {
		return "", fmt.Errorf("signing attestation payload: %w", err)
	}
	return base64.StdEncoding.EncodeToString(sig), nil
}

func loadPrivateKey(pemPath string) (*rsa.PrivateKey, error) {
	data, err := os.ReadFile(pemPath)
	if err != nil {
		return nil, fmt.Errorf("reading key file: %w", err)
	}
	block, _ := pem.Decode(data)
	if block == nil {
		return nil, fmt.Errorf("no PEM block found in %s", pemPath)
	}
	key, err := x509.ParsePKCS1PrivateKey(block.Bytes)
	if err != nil {
		return nil, fmt.Errorf("parsing private key: %w", err)
	}
	return key, nil
}

func buildAttestation(inputsPath, outputsPath, keyPath string) (*Attestation, error) {
	inputsRaw, err := os.ReadFile(inputsPath)
	if err != nil {
		return nil, fmt.Errorf("reading inputs file: %w", err)
	}
	var inputs []FileDigest
	if err := json.Unmarshal(inputsRaw, &inputs); err != nil {
		return nil, fmt.Errorf("parsing inputs JSON: %w", err)
	}

	outputsRaw, err := os.ReadFile(outputsPath)
	if err != nil {
		return nil, fmt.Errorf("reading outputs file: %w", err)
	}
	var outputs []FileDigest
	if err := json.Unmarshal(outputsRaw, &outputs); err != nil {
		return nil, fmt.Errorf("parsing outputs JSON: %w", err)
	}

	builderID := os.Getenv("BUILDER_ID")
	if builderID == "" {
		return nil, fmt.Errorf("BUILDER_ID environment variable is not set")
	}

	priv, err := loadPrivateKey(keyPath)
	if err != nil {
		return nil, err
	}

	unsigned := Attestation{
		Inputs:    inputs,
		Outputs:   outputs,
		Timestamp: time.Now().UTC().Format(time.RFC3339),
		BuilderID: builderID,
	}

	payload, err := json.Marshal(unsigned)
	if err != nil {
		return nil, fmt.Errorf("canonicalizing payload: %w", err)
	}

	sig, err := signPayload(payload, priv)
	if err != nil {
		return nil, err
	}

	unsigned.Signature = sig
	return &unsigned, nil
}

func main() {
	if len(os.Args) != 4 {
		fmt.Fprintln(os.Stderr, "usage: provenance <inputs.json> <outputs.json> <private_key.pem>")
		os.Exit(2)
	}
	att, err := buildAttestation(os.Args[1], os.Args[2], os.Args[3])
	if err != nil {
		fmt.Fprintf(os.Stderr, "error: %v\n", err)
		os.Exit(1)
	}
	out, _ := json.MarshalIndent(att, "", "  ")
	fmt.Println(string(out))
}

Deterministic JSON serialization

The struct's field order is fixed at compile time by its definition, so json.Marshal always emits fields in the same order for the same logical content; this is why the payload is built as a typed Attestation struct rather than a map[string]interface{}, since Go's JSON encoding of a map sorts keys alphabetically by default but a struct preserves declaration order, either of which is deterministic on its own, but mixing the two within one payload risks a verifier reconstructing a different byte layout than the signer used. The signature field itself is excluded from the signed payload (set to empty string, or omitted, at signing time) since the signature obviously can't be part of what it's signing over.

Verified

Compiled with go build. Generated a real RSA keypair with OpenSSL, ran the program to produce a signed attestation, then wrote an independent Go verifier that re-reads the attestation, strips the signature field, re-marshals the remaining struct exactly as the signer did, and calls rsa.VerifyPKCS1v15 against the public key: verification succeeded. A tamper test, flipping one character in an input's SHA-256 digest and re-running verification, correctly failed with a signature-mismatch error, confirming the scheme actually detects tampering rather than passing regardless of content.

Trade-offs

Using PKCS1v15 padding and a single RSA key here is simple and fully supported by the Go standard library alone (no external dependency), matching the question's constraint; a production system would more likely use Sigstore's keyless signing (short-lived, OIDC (OpenID Connect)-backed certificates) to avoid the operational burden of protecting a long-lived private key file: that file has to be generated once and then protected for its entire lifetime, encrypted at rest, access-restricted, rotated on a schedule, and revoked immediately if it is ever exposed, and if it does leak, whoever holds it can forge valid attestations indefinitely, or at least until the leak is discovered and the key revoked, with no built-in record of who actually produced a given signature. Keyless signing removes that persisted secret entirely: at sign time the CI runner exchanges a short-lived OIDC token (proof that this exact signing request came from this exact workflow run) for a certificate from a public certificate authority (Sigstore's Fulcio), signs with an ephemeral key that is discarded the moment signing completes, and records the signature alongside the certificate and a transparency-log entry (Rekor) that anyone can later audit. There is no long-lived key file to steal, back up, or rotate, at the cost of depending on the OIDC identity provider and Sigstore's infrastructure being reachable at sign time. Container image signing, discussed separately in this topic, makes the identical trade-off: a persisted key you must guard forever versus an ephemeral, OIDC-backed identity with nothing to steal.