DoorDash Site Reliability Engineer (Junior Level) - Comprehensive Interview Preparation Guide

Recommended commands:

1. ps — list all processes including threads, show PID/PPID, full cmd, state, %MEM/%CPU, sort by memory:

ps -eLo pid,ppid,tid,psr,stat,%mem,%cpu,cmd --sort=-%mem

• -e : all processes
• -L : show threads (lightweight processes)
• o ... : custom output columns (pid, ppid, tid=thread id, psr=CPU, stat=state, %mem, %cpu, cmd=full command line)
• --sort=-%mem : sort descending by memory usage

2. top — interactive view, include threads, show full command, sort by memory:

top -H -o %MEM -c

• -H : show threads
• -o %MEM : sort by memory
• -c : show full command line instead of program name

Use top when you need an interactive, real-time view to watch changing CPU/memory usage and drill into PIDs.

3. htop — nicer interactive UI (if installed), toggle threads and full cmd:

htop
# then press F2 -> Display options -> check "Show custom thread names" or press H to toggle threads, F6 to sort by MEM%

• htop provides curses UI, easier filtering, tree view, and mouse support.

When to use ps vs top:

• ps: snapshot, scriptable, reproducible output for audits or logs (use in scripts or to capture current state).
• top/htop: interactive troubleshooting and live monitoring.

Direct answer

Treat this like triaging technical debt: first classify every panel by whether it maps to an actual triage decision, aggressively cut or consolidate everything that doesn't, and validate the result against real on-call engineers before calling it done, not just against your own judgment.

Approach

Step 1: audit and classify

Talk to two or three people who've actually used the dashboard during an incident. For every panel, tag it: critical (maps to a golden signal or SLO), useful for drill-down, rarely used, or dead weight with no clear owner or purpose anyone can name.

Step 2: consolidate around golden signals

Replace many single-entity panels with a small set of golden-signal panels (rate, errors, latency, saturation) plus a "top N by errors or latency" table instead of one panel per service or endpoint.

Step 3: use template variables for the long tail

Instead of one panel per service or region, add a dashboard variable and let the engineer select what to drill into. The front page stays small; context-switching happens on demand rather than by scrolling past 35 panels that don't apply to the current incident.

Step 4: redesign layout around the incident workflow

Row 1: is it broken (SLO burn, error rate, active alerts). Row 2: how broken (latency, saturation, dependency health). Row 3 and below: drill-down detail. This mirrors how someone actually works an incident, overview first, detail on demand, instead of a flat grid of 40 equally-weighted panels.

Step 5: validate, don't just ship

Run the new version past on-call for a rotation or two, or a tabletop exercise against the last few real incidents, and explicitly ask whether this panel set would have gotten them to root cause faster. Cut anything that doesn't survive that test.

Worked example

A concrete, honest instance of the consolidation pattern: suppose 15 of the 40 panels are the same latency chart repeated once per microservice. Replacing those 15 with one templated latency panel (a service selector variable) plus one "top 5 services by p99" ranked table removes 13 panels (15 down to 2) while preserving, and arguably improving, the same information, since the ranked table surfaces the worst offender automatically instead of requiring a scroll through 15 charts to spot it. The same consolidation pattern applied to a repeated error-count-per-service panel family would remove a comparable number. The exact final panel count for the full 40 depends on how much of the original sprawl is this kind of per-entity repetition versus genuinely distinct signals, which is precisely what step 1's audit is for.

Trade-offs and pitfalls

• Cutting aggressively without checking with on-call can remove a panel that's rarely used but critical for one specific failure mode, like a replication-lag panel that's silent 99% of the time but is the first thing needed during that one incident type. Validate against past incidents, not just usage frequency.
• Template variables trade a small amount of always-visible context for a click. That's usually the right trade for an overview dashboard, but a signal that's genuinely page-worthy, like overall SLO burn, should stay pinned rather than sit behind a variable.
• Common wrong turn: fixing the dashboard once without establishing an ownership or review process, which is how it reached 40 panels in the first place. Without governance, the same sprawl recurs within a year.
• Don't conflate "unused" with "unnecessary." Some panels are unused specifically because the failure mode they'd catch hasn't happened yet, not because the panel is dead weight.

Start by defining safety goals: zero user-visible downtime, data integrity, and ability to rollback quickly. For stateful services backed by a DB I prefer a canary-first approach, with blue‑green as an option when rollback must be instantaneous and environment cost is acceptable.

Deployment plan (canary):

• Deploy new service version to a small subset (1–5%) of instances or pods.
• Route a small percentage of traffic (weighted LB or service mesh) to canary; keep same DB.
• Gradually increase weight (5→25→50→100) only after health & metrics are stable.

Schema migration strategy:

• Use backward-compatible, multi-step migrations:
  1. Expand: add new columns/tables/indices (non-destructive) while old code still writes/reads the original fields.
  2. Backfill: populate new fields in background jobs if needed.
  3. Cutover: switch canary/new code to read from new fields once backfill complete.
  4. Contract: once all clients use new schema and stable for N deploys, remove old columns.
• Ensure backwards compatibility: new code should still write data in a shape the old code can tolerate (dual writes if necessary), or use feature flags to gate new reads.
• For incompatible changes use a blue‑green deployment that includes a coordinated DB migration window with transactional guarantees, or use a migration service and versioned APIs.

Traffic shifting details:

• Implement using service mesh (Istio/Linkerd) or LB weights; tie increases to automated checks.
• Warm up caches and connection pools before increasing traffic.

Metrics to monitor during rollout:

• Application errors (5xx, RPC errors)
• Latency P50/P95/P99 per endpoint
• DB metrics: replication lag, transaction rate, lock wait times, slow queries, connection count, CPU/memory
• Business KPIs: request success rate, throughput, user transactions
• Resource metrics: pod restarts, OOMs, GC pauses
• Integrity checks: background job backfill progress, data validation errors

Abort conditions and automated rollback:

• Define SLO-based thresholds (e.g., error rate > 0.5% absolute or +200% relative; P95 latency > threshold; DB replication lag > allowed).
• If thresholds breached, automatically reduce traffic weight to zero and scale down new instances, or switch blue→green instantly.
• Run failure-domain isolation: if canary in one AZ fails, isolate that AZ rather than global abort.
• After abort, capture logs, traces, and perform quick root-cause triage; do not retry full rollout until root cause fixed and replayed in staging.

Other safety practices:

• Use feature flags and dual-write patterns to decouple deploy from migration.
• Run schema migrations in idempotent, logged batches; keep migration rollback scripts and backups.
• Test the whole process in staging with production-sized datasets and chaos tests for DB failures.

This approach balances risk and velocity: slow, observable traffic shifts + backward-compatible migrations minimize downtime and preserve data integrity while preserving fast rollback paths.

Situation: At a previous company a PM wanted to use the remaining error budget to launch a high-profile personalization feature before quarter-end. Engineering and SREs flagged that projected SLO burn would exceed the budget and raise production incident risk; stakeholders called us obstructive.

Task: As the SRE lead, I needed to acknowledge business urgency, protect availability, and find a path forward that balanced risk and delivery.

Action:

• Clarified constraints and goals with stakeholders: asked what business metrics (revenue, retention) this release must hit, acceptable outage window, and rollback tolerance.
• Quantified risk: ran simulations using recent telemetry to show projected SLO burn, expected MTTR impact, and probability of P1 incidents for a full rollout vs. staged rollout.
• Proposed trade-offs and concrete options:
  1. Strict canary + progressive rollout behind a feature flag with automatic rollback if error-rate or latency thresholds breach a safe margin (guardrail).
  2. Temporary, narrowly scoped SLO adjustment for this feature only (timebox 72 hours) with senior-exec approval and stricter alerting/ops on-call augmentation.
  3. Delay full rollout and ship a mitigated subset (reduced traffic or fewer regions) to validate.
• Recommended preferred path: canary + feature flag + enhanced monitoring + playbook + on-call escalation. If business insisted on faster ramp, offer option 2 but require exec sign-off and compensation plan (increased on-call, dedicated runbook, post-mortem commitment).
• Documented decision, measurable KPIs, rollback conditions, and a post-launch review timeline.

Result: Stakeholders accepted the staged rollout with guardrails. We shipped the feature to 5% of traffic, detected a misconfiguration that would have caused a 3x error-rate increase at 100% and rolled back automatically. Business metrics began trending positively; full rollout proceeded after two days of stable metrics. This preserved SLO health and maintained trust.

Why this approach: It balances business impact and reliability by turning an adversarial “no” into collaborative risk management. Quantify risks, offer actionable mitigations, require clear acceptance criteria for any SLO changes, and include compensating controls (feature flags, canaries, automated rollback, on-call support, exec approval).

Approach

Three separate hazards have to be handled together here: the timeout has to actually kill the process (not just stop waiting for it), the streaming has to not deadlock on large output, and cleanup has to happen even if the parent itself is killed.

import subprocess, threading, time

def run_cli(cmd, timeout, log_file):
    with open(log_file, "w") as lf:
        proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
                                 text=True, bufsize=1)
        def pump():
            # read line-by-line in a background thread so stdout is drained
            # continuously -- reading ONLY after the process exits risks the
            # child blocking on a full OS pipe buffer if it writes a lot of output
            for line in proc.stdout:
                lf.write(line)
                lf.flush()
        pumper = threading.Thread(target=pump, daemon=True)
        pumper.start()
        try:
            exit_code = proc.wait(timeout=timeout)
        except subprocess.TimeoutExpired:
            proc.kill()      # SIGKILL: don't trust the child to honor SIGTERM
            proc.wait()       # REAP the process -- this is what prevents a zombie
            exit_code = -9
        pumper.join(timeout=1)
        return exit_code

Verified in a sandbox with two cases: a command that prints two lines and exits 0 returns exit code 0 with both lines correctly captured in the log file; a command that sleeps 5 seconds called with a 0.3s timeout is killed and reaped in ~0.3s (confirmed the process does not linger), returning exit code -9.

Key points

• Streaming to a rotating log file in a background thread, rather than only reading output after proc.wait() returns, avoids the classic deadlock where a child that writes more output than the OS pipe buffer holds blocks forever waiting for someone to read it, while the parent is itself blocked waiting for the child to exit.
• proc.kill() (SIGKILL) rather than proc.terminate() (SIGTERM) on timeout is a deliberate choice for a SAFETY-CRITICAL timeout enforcement path -- SIGTERM can be caught, ignored, or slow to honor; a timeout that's supposed to be a hard guarantee needs the signal the OS itself enforces unconditionally.

How resource cleanup on termination is guaranteed

The critical, easy-to-miss step is proc.wait() immediately AFTER proc.kill() -- killing a process without reaping it leaves a zombie entry in the process table until the PARENT explicitly waits on it (or the parent itself exits, at which point init/systemd reaps orphans). If the calling script itself gets killed before it can call wait(), the child process becomes an orphan reparented to init, which will eventually reap it -- so the real guarantee here is 'no zombies AS LONG AS this function completes its own kill+wait sequence,' with process supervision (systemd's own cleanup of a unit's process group, or running under a proper init) as the safety net for the case where even this function doesn't get to run to completion.

Trade-offs and pitfalls

A rotating log file needs its own size/retention policy independent of this function (this function assumes the log file handle is managed correctly, e.g. via Python's logging.handlers.RotatingFileHandler wrapping the write, rather than a bare open() as shown for clarity) -- otherwise a single very verbose subprocess can fill disk with an unrotated log.

Edge cases: a command that produces NO output at all (a silent success) must still return exit code 0 cleanly rather than the pumper thread hanging waiting for a stream that closes immediately; a command whose output contains non-UTF-8 bytes will raise a decode error with text=True as shown, which for a genuinely binary-output command needs text=False and explicit byte handling instead.

Direct answer

Baking an image means pre-installing everything a server needs (OS packages, hardening, the app itself) into a reusable image with a tool like Packer, instead of configuring the server after it boots. Build the pipeline around one principle: nothing reaches production as an image that has not been baked, tested, and scanned the same way every time, and the fleet gets updated by replacing instances behind health checks and connection draining rather than patching them in place. The design has two paths through the same pipeline: the normal path (bake, test, promote through environments, canary, full rollout) and a fast path for urgent security patches that skips environment promotion but never skips the tests or the scan.

Structured elaboration

Image build and test

• CI triggers a Packer build on a base-image or application change: provision the base OS, apply hardening (CIS-style benchmarks, i.e. standardized security-configuration checklists), install the app artifact, and pull secrets via short-lived tokens rather than baking them in.
• Baked-in automated tests run as part of the same pipeline, not as a separate manual step: unit and config-validation tests during the bake, then a post-bake stage that launches the image in an isolated environment and runs integration and smoke tests against it.
• A vulnerability scan (for example Trivy or Grype against the baked image) runs in that same post-bake stage. This is a hard gate, not advisory: an image with a scan finding above the agreed severity threshold does not get published.
• On pass, the image is registered in the artifact registry tagged with its git SHA, build ID, SBOM, and the CVE baseline it passed against, so any later question of "what is actually running" and "was it scanned against what we knew at the time" has an answer.

Promotion across environments

Promotion is a pipeline gate, not a person clicking approve in a console: dev, then staging with regression tests, then a canary slice of production, each gated on the previous stage's tests and monitoring staying green.

Swapping the fleet over

• The fleet sits behind an ASG (or equivalent instance group) and a load balancer. The launch template points at the new image; an ASG instance refresh (or an equivalent rolling-replace controller) walks the fleet in batches.
• Per instance: deregister from the target group first, which starts connection draining; wait for in-flight requests to finish or the drain timeout to hit; only then terminate it. The replacement instance must pass its health check before the load balancer sends it any traffic.
• A minimum-healthy-percentage setting (for example 90%) caps how much capacity can be replacing at once, so a bad new image degrades a fraction of the fleet rather than all of it while it is still being watched.
• For workloads that carry state (a service with long-lived connections, or one with session affinity), connection draining alone is not enough: the drain window also has to respect existing session affinity, and if any part of the workload is stateful in the sense of holding data (not just connections), that has to coordinate with the data layer's own replication or failover process rather than treating the instance as freely swappable the moment its health check fails.

Fast-tracking an urgent security patch

The fast path changes how far the image travels before real traffic sees it, not whether it is tested:

• Skip the full dev-then-staging promotion chain; go straight from bake to a canary slice of production.
• Keep the bake-time tests and the vulnerability scan as hard gates; an urgent patch that has not been scanned is exactly the failure mode a patch process exists to prevent.
• Shorten, but do not remove, the canary observation window, and have the rollback path pre-verified rather than improvised, since this path is exercised under time pressure.
• Immediately backfill afterward: once the emergency patch has gone through the fast path, run it (or its base) through the normal dev and staging pipeline the following day, so the fast-tracked version does not become a permanent exception living outside the standard promotion history.

Worked example

flowchart TD
  A[Source or base image change] --> B[CI triggers Packer bake]
  B --> C[Bake time tests: hardening, vuln scan, smoke tests]
  C --> D[Publish image with SBOM and CVE tags to registry]
  D --> E[Promote through dev then staging]
  E --> F[Canary: weighted traffic on new AMI]
  F --> G{Health checks and SLOs pass?}
  G -- Yes --> H[Full fleet rollout via ASG instance refresh]
  G -- No --> I[Roll back to prior AMI, tag new image as bad]
  B -.urgent security patch.-> J[Fast path: skip dev and staging, bake plus scan only]
  J --> F

Concretely: a CVE lands in the base OS image. CI triggers a Packer bake immediately (the dashed path above). The bake produces a new image; the same automated tests and the same vulnerability scan run against it as any normal build, just without waiting for a scheduled promotion window. It goes straight to a canary slice of the fleet, monitored against the same health checks and error-rate thresholds as any other rollout, then to the full fleet via instance refresh. The following day, the same image is run through the normal dev and staging environments to confirm nothing outside the emergency scope regressed.

Trade-offs & pitfalls

• Baking images takes longer than patching in place, and that trade-off is deliberate: reproducibility and a clean rollback (revert the launch template to the previous image ID) are worth the extra build minutes.
• The most dangerous version of a "fast path" is one that quietly also skips testing or scanning under time pressure; the fast path should only ever shorten promotion, never verification.
• For stateful workloads, connection draining and health checks are necessary but not sufficient; assuming they are enough to make image replacement safe for anything holding data is a common design mistake.
• Rolling back an in-flight instance refresh needs to be a rehearsed, one-command action (point the launch template back at the previous image ID), not something improvised the first time it is needed.

Direct answer
An automated rollback orchestrator has three jobs done in a strict order: detect that the deployment is failing, pause the rollout so it stops making things worse, and then execute the rollback across every affected service in an order that respects their dependencies, followed by a health check that confirms the system actually landed in a good state and not just a different bad one.

Structured elaboration
Core components:

1. Detector: subscribes to the same metrics/health signals as the canary-analysis gate (error rate, latency, health checks) and raises a rollback INTENT, not the rollback itself.
2. Pause controller: immediately halts further traffic ramp or pod replacement so the blast radius stops growing while the system decides what to do.
3. State store: records, per deployment, the previous good artifact/version for every service touched, the order services were deployed in, and which have already completed rollback. This has to be durable (survive the orchestrator crashing mid-rollback), because a rollback that itself half-completes and then loses track of state is worse than the original failure.
4. Rollback executor: walks the dependency graph in reverse deployment order (or a computed safe order if services were deployed in parallel), issuing the rollback action per service: redeploy the previous artifact, flip the associated feature flag off, or shift traffic weight back to the stable version, depending on what mechanism that service was deployed with.
5. Post-rollback validator: re-runs the same health checks used at deploy time against the rolled-back state; a rollback that "completes" without this step can silently leave the system on a version that's ALSO broken.

Handling a partially completed rollback: the state store's per-service status (not-started / in-progress / done / failed) lets the orchestrator resume from where it left off on restart, and a service stuck in "in-progress" after a timeout should page a human rather than retry forever, since retrying a stuck rollback blind can double-apply a destructive action.

Worked example
Three services deploy together: A -> B -> C, where B depends on A's new API and C depends on B's new schema. If C's rollout fails, the safe rollback order is C, then B, then A, the reverse of the deploy order, because rolling back A first while B still expects A's new behavior would break B on a system that was working seconds earlier.

Trade-offs and pitfalls
Fully automating this across many services is powerful but risky if the dependency graph is wrong or stale; a common failure mode is an orchestrator that trusts a hand-maintained dependency list that's drifted from reality. A pragmatic middle ground many teams adopt is to automate rollback fully for a defined, lower-blast-radius subset of critical services first, and require a human confirmation step for the rest, tightening the automated scope as confidence in the detector and executor grows.

Alert rule (Prometheus):

expr: |
(
histogram_quantile(
0.99,
sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service)
)
)
/
avg_over_time(
(
histogram_quantile(
0.99,
sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service)
)
)[1h]
) > 2
for: 5m
labels:
severity: page
annotations:
summary: "99th pctl latency for {{ $labels.service }} >2x 1h avg"
description: "Current 99th percentile is {{ $value }}s, >2x the 1h moving average for service {{ $labels.service }}."

How it works

• The numerator computes current 99th percentile using histogram_quantile(0.99) on the per-service rate of histogram buckets over a 5m window.
• The denominator computes the 1-hour moving average of that same 99th-percentile time series using avg_over_time(...[1h]).
• The ratio > 2 detects a >2x increase; Prometheus' "for: 5m" ensures the condition is sustained for 5 minutes before alerting.

Potential pitfalls and mitigations

• Histograms at extreme quantiles are sample-hungry: low request volume makes the 99th unstable. Mitigate by requiring a minimum request rate (e.g., add and > threshold: sum(rate(http_requests_total[5m])) by (service) > 5).
• Division by zero if baseline avg is 0: use clamp_min(avg_over_time(...)[1h], 1e-9) or guard with boolean checks.
• Using same rate/window consistently: keep the rate interval (5m) and baseline window (1h) appropriate for traffic characteristics.
• Subqueries and histogram_quantile are expensive at high cardinality; ensure aggregation by service and limit label cardinality.
• Outliers or transient spikes: the for:5m helps; you may also use smoothing (e.g., increase baseline window) or compare percentiles like 95th if 99th too noisy.
• Ensure buckets are correctly configured (sufficient high le values) so 99th can be represented.

Direct answer
Mentoring someone through their first postmortem means pairing them with a real, ideally low-stakes incident, giving specific feedback on structure and tone rather than just 'good job,' and modeling the blameless framing yourself before expecting them to reproduce it independently.

Structured elaboration

• Pick the right first incident. A moderate-severity, reasonably contained incident is a better first assignment than either a trivial one (nothing to learn from) or a highly political, multi-team, high-visibility one (too much pressure for a first attempt).
• Give feedback on structure. Check whether the timeline is objective and evidence-backed rather than reconstructed from memory, whether root cause is separated from contributing factors, and whether action items are specific and owned rather than vague aspirations.
• Give feedback on tone, with concrete examples. Point out any sentence that names a person rather than a system gap, and show, don't just tell, how to rewrite it: 'the engineer forgot to run the checklist' becomes 'the checklist has no automated enforcement, so a required step could be skipped.' Seeing the before-and-after side by side teaches the skill faster than an abstract rule.
• Have them facilitate a real meeting, with you as backup, not the lead. Reading about facilitation and doing it live under mild pressure are different skills; be present to redirect gently if the discussion drifts toward blame, but let them run it.
• Follow up on whether the action items actually happened. Closing the loop on whether their first postmortem's action items got implemented and verified teaches the full lifecycle, not just the writing exercise.

Worked example
A junior engineer is assigned to lead the postmortem for a minor, contained caching bug that caused stale data for about ten minutes. Before the meeting, the mentor reviews their draft timeline and flags one sentence ('the developer pushed an untested change') to rewrite as a system-focused observation about the deploy process lacking a required test gate for cache-invalidation logic specifically. During the meeting, the junior engineer facilitates; the mentor stays quiet unless the discussion drifts, at one point gently redirecting a comment that started to focus on who wrote the original caching code. Afterward, feedback covers three things: the timeline was strong and evidence-based, the action item ('add a test for cache-invalidation edge cases') was specific and well-owned, but the root cause and contributing factors weren't clearly separated in the writeup, which is worth practicing next time. Three weeks later, the mentor checks whether the test was actually added and merged, closing the loop rather than treating the writing exercise as the end of the mentorship.

Trade-offs and pitfalls
The most common mistake is giving only high-level praise or criticism ('good postmortem' or 'needs work') without specific, actionable examples the person can apply next time. A second is the mentor taking over facilitation when things get slightly awkward instead of letting the mentee work through it with light support, which prevents them from actually building the skill.

A provenance attestation is fundamentally a signed statement, so the implementation has two halves that both have to be exactly right: building a deterministic payload to sign, and signing it in a way a verifier can independently reproduce and check.

package main

import (
	"crypto"
	"crypto/rand"
	"crypto/rsa"
	"crypto/sha256"
	"crypto/x509"
	"encoding/base64"
	"encoding/json"
	"encoding/pem"
	"fmt"
	"os"
	"time"
)

type FileDigest struct {
	Path   string `json:"path"`
	Sha256 string `json:"sha256"`
}

// Field order is fixed by the struct definition, which is what makes
// json.Marshal deterministic for this payload: the same logical content
// always produces the same bytes to sign and to re-verify against.
type Attestation struct {
	Inputs    []FileDigest `json:"inputs"`
	Outputs   []FileDigest `json:"outputs"`
	Timestamp string       `json:"timestamp"`
	BuilderID string       `json:"builderId"`
	Signature string       `json:"signature"`
}

func signPayload(payload []byte, priv *rsa.PrivateKey) (string, error) {
	digest := sha256.Sum256(payload)
	sig, err := rsa.SignPKCS1v15(rand.Reader, priv, crypto.SHA256, digest[:])
	if err != nil {
		return "", fmt.Errorf("signing attestation payload: %w", err)
	}
	return base64.StdEncoding.EncodeToString(sig), nil
}

func loadPrivateKey(pemPath string) (*rsa.PrivateKey, error) {
	data, err := os.ReadFile(pemPath)
	if err != nil {
		return nil, fmt.Errorf("reading key file: %w", err)
	}
	block, _ := pem.Decode(data)
	if block == nil {
		return nil, fmt.Errorf("no PEM block found in %s", pemPath)
	}
	key, err := x509.ParsePKCS1PrivateKey(block.Bytes)
	if err != nil {
		return nil, fmt.Errorf("parsing private key: %w", err)
	}
	return key, nil
}

func buildAttestation(inputsPath, outputsPath, keyPath string) (*Attestation, error) {
	inputsRaw, err := os.ReadFile(inputsPath)
	if err != nil {
		return nil, fmt.Errorf("reading inputs file: %w", err)
	}
	var inputs []FileDigest
	if err := json.Unmarshal(inputsRaw, &inputs); err != nil {
		return nil, fmt.Errorf("parsing inputs JSON: %w", err)
	}

	outputsRaw, err := os.ReadFile(outputsPath)
	if err != nil {
		return nil, fmt.Errorf("reading outputs file: %w", err)
	}
	var outputs []FileDigest
	if err := json.Unmarshal(outputsRaw, &outputs); err != nil {
		return nil, fmt.Errorf("parsing outputs JSON: %w", err)
	}

	builderID := os.Getenv("BUILDER_ID")
	if builderID == "" {
		return nil, fmt.Errorf("BUILDER_ID environment variable is not set")
	}

	priv, err := loadPrivateKey(keyPath)
	if err != nil {
		return nil, err
	}

	unsigned := Attestation{
		Inputs:    inputs,
		Outputs:   outputs,
		Timestamp: time.Now().UTC().Format(time.RFC3339),
		BuilderID: builderID,
	}

	payload, err := json.Marshal(unsigned)
	if err != nil {
		return nil, fmt.Errorf("canonicalizing payload: %w", err)
	}

	sig, err := signPayload(payload, priv)
	if err != nil {
		return nil, err
	}

	unsigned.Signature = sig
	return &unsigned, nil
}

func main() {
	if len(os.Args) != 4 {
		fmt.Fprintln(os.Stderr, "usage: provenance <inputs.json> <outputs.json> <private_key.pem>")
		os.Exit(2)
	}
	att, err := buildAttestation(os.Args[1], os.Args[2], os.Args[3])
	if err != nil {
		fmt.Fprintf(os.Stderr, "error: %v\n", err)
		os.Exit(1)
	}
	out, _ := json.MarshalIndent(att, "", "  ")
	fmt.Println(string(out))
}

Deterministic JSON serialization

The struct's field order is fixed at compile time by its definition, so json.Marshal always emits fields in the same order for the same logical content; this is why the payload is built as a typed Attestation struct rather than a map[string]interface{}, since Go's JSON encoding of a map sorts keys alphabetically by default but a struct preserves declaration order, either of which is deterministic on its own, but mixing the two within one payload risks a verifier reconstructing a different byte layout than the signer used. The signature field itself is excluded from the signed payload (set to empty string, or omitted, at signing time) since the signature obviously can't be part of what it's signing over.

Verified

Compiled with go build. Generated a real RSA keypair with OpenSSL, ran the program to produce a signed attestation, then wrote an independent Go verifier that re-reads the attestation, strips the signature field, re-marshals the remaining struct exactly as the signer did, and calls rsa.VerifyPKCS1v15 against the public key: verification succeeded. A tamper test, flipping one character in an input's SHA-256 digest and re-running verification, correctly failed with a signature-mismatch error, confirming the scheme actually detects tampering rather than passing regardless of content.

Trade-offs

Using PKCS1v15 padding and a single RSA key here is simple and fully supported by the Go standard library alone (no external dependency), matching the question's constraint; a production system would more likely use Sigstore's keyless signing (short-lived, OIDC (OpenID Connect)-backed certificates) to avoid the operational burden of protecting a long-lived private key file, which is the same trade-off discussed for container image signing elsewhere in this topic.