Meta Site Reliability Engineer (Junior Level) Interview Preparation Guide

Incident commander

• Primary goal: Coordinate the response, bring incident to steady triage and closure; prioritize actions against customer/SLI impact.
• First 30 min tasks: Declare incident severity, assemble core responders, set incident timeline and objectives (e.g., “restore page uses /api to 90%”), assign roles, run short intake (what’s broken, scope, who’s affected).
• Decision authority: Determine incident priority/severity and whether to escalate to execs or call for a rollback/maintenance window.

Scribe (note-taker)

• Primary goal: Maintain accurate, timestamped record of facts, actions, decisions and communications for coordination and postmortem.
• First 30 min tasks: Create and share incident doc, log alerts, timestamps, commands run, people on call, hypotheses and next steps; capture mitigation steps and owner names.
• Decision authority: Authoritatively record the “single source of truth” timeline (what was recorded is used in post-incident review).

Communications lead

• Primary goal: Keep stakeholders and customers informed with consistent, accurate messages; minimize confusion and rumor.
• First 30 min tasks: Draft and send initial internal and external notifications (status page, Slack channel, pager), agree cadence for updates, coordinate with legal/PR if needed.
• Decision authority: Approve outward-facing incident messages and status-page updates.

Technical lead

• Primary goal: Drive technical diagnosis and remediation; propose and validate fixes.
• First 30 min tasks: Triage telemetry, reproduce symptoms, identify possible root causes, prioritize mitigations (hotfix, config change, rollback), assign engineers to workstreams.
• Decision authority: Approve and execute technical mitigations (e.g., trigger a rollback, switch traffic).

Escalation liaison

• Primary goal: Bridge to other teams, vendors, execs and incident support functions; manage escalation path and resource requests.
• First 30 min tasks: Contact on-call SMEs (DB, network, cloud provider), open vendor tickets, request additional resources or approvals (e.g., cloud quota increase), inform execs of potential business impact.
• Decision authority: Request formal escalations (vendor hotlines, cross-team war room) and authorize use of out-of-band resources.

Example handoffs

• IC → Technical lead: IC delegates “restore traffic” objective; technical lead owns execution.
• Technical lead → IC: When root cause confirmed, tech lead requests IC approval for rollback.
• Scribe ↔ Communications lead: Scribe logs message drafts; communications lead approves and publishes.
• Escalation liaison → IC: Confirms vendor committed to a timeline and hands back ETA for IC to communicate externally.

High-level goal: convert repeatable, low-risk manual incident playbooks into safe, audited automated remediation that reduces MTTR, human toil, and error—over 24 months in phased waves.

24-month roadmap (quarterly milestones)

• Q1: Inventory & baseline — catalog playbooks, classify by frequency, complexity, blast radius; measure current MTTR and human-hours.
• Q2: Pilot automations — pick 3 low-risk, high-frequency playbooks; build automation framework (idempotent actions, authentication, audit logs, feature flags).
• Q3–Q4: Expand to medium-risk — add role-based approvals, canary automation, simulated drills in staging; integrate with alerting/CI.
• Year 2 Q1–Q2: High-risk playbooks — introduce multi-step orchestrations, policy checks, circuit breakers, and safety sandboxes.
• Year 2 Q3–Q4: Harden & scale — central catalog, self-service UI, runbook-as-code, continuous validation, orgwide rollout and training.

Selection criteria for candidate playbooks

• Frequency: >4 occurrences/month (high ROI)
• Determinism: well-defined inputs and expected outcomes
• Blast radius: low to medium (limited services/users)
• Observability: clear signals to detect pre/post state
• Security/Compliance: no sensitive key exposure; approvals possible
• Complexity: single-step or linear multi-step preferred initially

Testing strategy

• Unit test individual actions (mocks for external systems)
• Integration tests in isolated envs with production-like data
• Chaos/simulated incidents: run automation against saved incidents and synthetic faults
• “Dry-run” mode: actions logged but not executed; compare expected to actual
• Progressive rollout: feature flags → canary on non-critical services → small % of incidents → full rollout

Rollback & safety controls

• Idempotency and precondition checks before action
• Transactional/compensating steps where possible
• Circuit breaker: abort if uncertain metrics or >X failures in Y minutes
• Immediate human-in-the-loop escalation if thresholds breached
• Automatic rollback steps and explicit “undo” playbooks
• Audit trail and immutable logs for every invocation

Ownership model

• Playbook owner: service SRE or on-call engineer (maintains correctness)
• Automation owner: automation engineering team (CI/CD, security, infra)
• Review board: cross-functional (SRE, Security, Dev, Compliance) for approvals
• SLAs for maintenance: owners must respond to failures/alerts within defined windows
• Runbook-as-code repo: PRs, automated tests, and changelog governance

KPIs to measure success

• MTTR reduction (per playbook) — target 30–70% decrease
• Number of incidents fully remediated automatically
• Human toil hours saved per month
• False positive/negative rate of automation actions (<1–5% target)
• Failed automation rate and rollback frequency
• Time to remediate when automation fails (ensure not worse than manual)
• Coverage: % of cataloged playbooks automated
• Compliance/audit pass rate and number of security incidents attributable to automation (should be zero)

Example (concrete):

• Pilot: automatic disk-cleanup for noisy log directories (freq: 20/month, blast: low).
• Tests: unit tests for path selection, integration with staging instance, dry-run compared to manual results, canary on a non-critical service for 2 weeks.
• Safety: pre-check free space, throttle deletions, circuit-breaker if deletion > expected bytes, immediate alert to on-call.
• KPI: MTTR for disk-full incidents down from 45m to 5m; human-hours saved ≈ 15 hours/month.

This plan balances speed of automation with rigorous safety, clear ownership, and measurable outcomes so automation increases reliability without increasing risk.

Direct answer

As a security architect you rarely have line authority over another team's backlog, so you get findings prioritized by making them cheap to accept and costly to ignore: translate the finding into the other team's own vocabulary (a defect, a customer risk, a compliance control they must attest to) and attach it to a decision they are already about to make, rather than asking them to open a brand-new work item. You lead with a specific, demonstrated risk instead of a policy citation, offer a menu of remediation options at different costs, and use an existing recurring forum, like a design review or architecture council, so the tradeoff is made visible to the team's own stakeholders, not just to you.

Structured elaboration

• Translate, don't mandate: reframe the threat-modeling finding in terms the team already tracks (a customer-facing incident scenario, a compliance control, a defect class QA can reproduce) instead of a generic "security best practice."
• Time it to their planning cycle: bring a written finding before backlog grooming or sprint planning, not after code is merged, so accepting it is a normal prioritization decision instead of a rework request.
• Offer options, not a mandate: propose two or three remediation paths (a quick mitigating control now, a full fix next sprint, an explicit accepted-risk sign-off) so the team's own product owner makes an informed tradeoff instead of feeling overridden.
• Borrow a forum, don't invent one: attach the ask to a ritual the team already respects, like their design review, so it reads as peer-level influence rather than a unilateral security gate.
• Make patterns visible upward: when a team consistently deprioritizes findings, escalate the pattern, not the individual finding, to a shared forum with both engineering and security leadership present, so someone with authority over both sides makes the call.

Worked example (illustrative, adapt to your own experience)

A security architect threat-models a new payments feature two weeks before the product team's sprint planning. Instead of filing a ticket titled "add input validation" into the team's backlog and hoping it gets picked up, they write a one-page finding: the specific attack path, the customer-facing scenario it enables, and three remediation options ranked by effort. They bring it to the team's existing design review, present it alongside the team's own product owner, and let the team choose between a lightweight mitigating control shippable in the current sprint or a fuller fix in the next one. The team picks the lightweight option and schedules the fuller fix on their own board, because the tradeoff was made visible and owned by them, not imposed from outside.

Trade-offs and pitfalls

• Too formal (a mandatory sign-off gate) breeds resentment and workarounds; too informal (a message in passing) gets lost in someone else's priority queue.
• Offering remediation options is powerful but risks a team always choosing the cheapest option indefinitely, so track accepted-risk decisions somewhere durable so a pattern of chronic deferral becomes visible over time.
• Borrowing an existing ritual only works if that ritual has real teeth; if the design review itself gets skipped or ignored, attaching your ask to it just inherits its weakness.

What the interviewer probes next

They typically follow up on how you handle a team that keeps saying "next sprint" indefinitely, whether you would ever reach for a hard gate like a release-blocking scan instead of persuasion, and how this influence model holds up when you are supporting a dozen teams at once instead of just one.

Direct answer
Five Whys, fishbone (Ishikawa) diagrams, fault-tree analysis, and causal-chain or timeline analysis are all structured root-cause techniques, but they suit different incident shapes. Five Whys is fast and best for a single, mostly-linear chain of causation. Fishbone is best when you suspect several independent categories of cause (people, process, technology, environment) and want to brainstorm broadly before narrowing. Fault-tree analysis is best for complex, multi-path failures where you need to reason about combinations of conditions, not just one chain. Causal-chain or timeline analysis is best when the incident unfolded over a long period with many events, and reconstructing the sequence itself is most of the work.

Structured elaboration

• Five Whys. Strength: fast, requires no special tooling, good for straightforward incidents with a genuinely linear cause. Weakness: it forces a single narrative thread, so on an incident with multiple independent contributing factors it can stop at the first plausible-sounding chain and miss a second, unrelated gap that also mattered. Combining it with a causal-graph or fault-tree check on the resulting hypothesis (does this cause actually explain the full timeline, or just part of it) helps catch that failure mode.
• Fishbone (Ishikawa). Strength: structured brainstorming across categories (commonly people, process, technology, environment) surfaces candidates you might not think of starting from a single chain. Weakness: it's a divergent tool, good for generating hypotheses, but it doesn't by itself tell you which candidate cause is actually correct; you still need evidence to narrow down.
• Fault-tree analysis. Strength: models AND/OR combinations of conditions, so it's the right tool when the incident required several things to go wrong simultaneously (a database failover only failed because BOTH the standby was on an incompatible version AND the health check didn't catch the mismatch). Weakness: more effort and formalism than most incidents justify; overkill for a simple single-cause bug.
• Causal-chain or timeline analysis. Strength: best when the incident unfolded across many events over hours or days, and the real analytical work is establishing what happened when and in what order, which then makes the cause fairly evident once assembled. Weakness: doesn't add much analytical structure beyond reconstruction; you often still need Five Whys or fishbone on top of the assembled timeline to go from 'here's what happened' to 'here's why.'

Worked example
A multi-hour cascading outage across several services: causal-chain or timeline analysis is the right first tool, since the priority is establishing the sequence across services before anything else makes sense. A single service crashing on a specific malformed input: Five Whys is fast and sufficient. A database failover that should have worked but didn't: fault-tree analysis, since it likely required more than one condition (incompatible standby version AND a health check that didn't catch it) to align. A vague, hard-to-pin-down data-quality issue with no obvious single trigger: fishbone, to broadly brainstorm across categories (was it the data source, the pipeline code, a schema change, an environment difference) before narrowing with evidence.

Trade-offs and pitfalls
The most common mistake is defaulting to Five Whys for everything because it's the most familiar technique, even on incidents with multiple independent contributing factors where it will produce a tidy but incomplete story. Pick the technique to fit the shape of the incident, not out of habit, and don't hesitate to combine two (fishbone to generate candidates, then Five Whys or fault-tree to narrow and validate).

Direct answer. Ownership has to be a named role tied to the business function or system the runbook covers, not "whoever wrote it last," with versioning that ties each revision to a specific trigger, not just a date. Catching drift before a real event means testing the runbook during a planned exercise instead of discovering it's wrong while executing it under pressure, paired with a lightweight review trigger whenever something the runbook depends on changes.

1. Ownership

Assign a named owner to every runbook, ideally the person or role accountable for the business function or system it covers, not a shared team inbox and not necessarily the original author, who may move on. Ownership means being responsible for the runbook's accuracy, not personally executing every step during a real event. Make it visible on the document itself (name, role, last-reviewed date) so anyone opening it during an incident can see who to ask if something looks wrong, and so a stale "owner" who's left the organization is an obvious red flag rather than a hidden one. When ownership changes hands, require an explicit handoff with a fresh review as part of the transition, not a silent reassignment.

2. Versioning

Treat runbooks as controlled documents: every change gets a version number, a date, who made it, and why, kept in the document's own change history rather than relying on file metadata or institutional memory. Tie versions to triggers, not just a calendar. A scheduled review is a reasonable baseline, but the more reliable trigger is linking review to the events that actually make a runbook wrong: the system it depends on changes, the org structure around it changes, or a real exercise or incident surfaces a gap. A runbook untouched for eighteen months right after the system it describes was rebuilt is far more suspect than one reviewed on schedule six months ago. Keep prior versions accessible, not just the latest, so a post-event review can tell whether a gap that showed up was already known and fixed but hadn't propagated to whoever was executing, or was genuinely new.

3. Catching drift before it matters

The strongest mechanism is using the runbook for real, on a schedule, inside an exercise (a tabletop discussion or a functional exercise that actually executes some steps), before a live event forces the discovery. A runbook nobody has walked through since it was written is a hypothesis, not a tested procedure. Build a lightweight checklist step into whatever process generates the kind of change that breaks runbooks: when a system change, a supplier change, or an org change is being planned, that process flags which runbooks might be affected, so the review happens close to the change instead of being discovered much later. Track a simple staleness signal across the whole runbook set (time since last review, time since the system or team it covers last changed) and report on it the way you'd report any other risk metric, so an accumulating backlog of unreviewed runbooks is visible to whoever owns the continuity program overall, not discovered runbook by runbook.

Worked example

A payments-processing runbook is owned by the payments platform lead, currently at version 4: v2 added a fallback provider, v3 corrected an escalation contact list after a reorg, v4 followed a functional exercise that found the documented rollback step no longer matched how the system actually rolled back. Six months after v4, the team migrates the payments platform to new infrastructure as part of an unrelated project. Because "infrastructure migration for a system with an owned runbook" sits on that project's change-review checklist, the payments platform lead is flagged automatically and confirms the runbook needs an update before the migration ships, catching the drift as part of the planned change rather than during the next incident. At the next scheduled functional exercise three months later, the team walks through the updated runbook end to end and finds one further step, a monitoring dashboard link, still pointing at the decommissioned infrastructure. That becomes a logged finding and v5.

Trade-offs & pitfalls. A purely calendar-based review cadence catches drift too slowly for a fast-changing system and too often for a stable one; tying review to real change triggers is more work to set up but matches effort to actual risk. Ownership without accountability for staleness (a name on a document nobody ever checks) is barely better than no ownership; the staleness signal has to surface to someone, not just exist. And testing a runbook only during a real event is the worst possible time to discover it's wrong: the exercise programme is what lets a team find the gap on an ordinary afternoon instead of during an actual outage.

Direct answer

Safety, packaging maturity, and contributor familiarity pull in genuinely different directions across these three, so the right answer depends on what the framework is actually FOR, not on picking a single best language in the abstract.

Comparison

Recommendation and reasoning

For a company-wide, contributor-heavy automation framework, Python is the strongest default: the packaging story is more work than Go's single binary, but the contributor-familiarity and library-ecosystem advantages usually outweigh that for a framework meant to be extended by many teams across the org, not just a small platform team. Go is the better call specifically when concurrency and single-binary distribution to environments without a Python runtime matter more than broad contributor familiarity -- for example, a high-throughput worker or a CLI that needs to run on minimal container images. Bash should be reserved for genuinely small, single-purpose glue (a few lines gluing existing CLI tools together) rather than as the implementation language for a 'framework' -- its lack of a type system and weak static analysis make it a poor foundation for something many people will extend over years, even though it's the right tool for a five-line wrapper script.

A concrete task-to-language mapping sharpens this: ad-hoc file operations and small production fixes are legitimately Bash's home turf (fast, universally readable, no build step); a shared deployment CLI that many developer laptops will install points toward Go specifically because a static binary sidesteps 'do you have the right Python version installed' entirely; and a high-throughput worker processing on the order of 10,000 tasks/minute is exactly where Go's goroutine model and lower per-task overhead earn their onboarding cost over Python's asyncio/multiprocessing, which can still hit the target but with more tuning and a higher operational ceiling to manage.

Migration plan from ad-hoc scripts

Don't do a big-bang rewrite. Inventory existing scripts by usage frequency and blast radius, migrate the highest-value/highest-risk ones first (where the framework's retry/logging/secrets primitives pay off fastest), and keep the framework backward-compatible enough that a not-yet-migrated Bash script can still be invoked as a step inside the new framework rather than requiring every single script to be rewritten before the framework has any users.

Trade-offs and pitfalls

The most common mistake is picking the language based on the TEAM's current skill distribution rather than the TASK's actual requirements, then discovering the mismatch only once the framework has grown large enough that switching is expensive. Edge case: a framework whose different SUB-COMPONENTS have genuinely different requirements (a fast CLI entrypoint alongside a slow, throughput-heavy background worker) may legitimately justify two languages rather than forcing one choice across a codebase that doesn't have uniform needs.

A comprehensive security and compliance checklist for production deployments needs to cover the full path from what's in the artifact, to how it was produced, to who's allowed to approve it moving, since a gap at any one of those layers undermines the guarantees the others provide.

The checklist

Secrets management: no plaintext secrets in the artifact or its configuration; all runtime secrets injected via a secrets manager with short-lived, scoped credentials, following the patterns discussed throughout this topic. Image signing and provenance: the artifact is signed, and its provenance (what built it, from what source, when) is attached and verifiable, not merely claimed. Vulnerability scanning: the artifact has passed SCA and container-image scanning with no unaddressed CRITICAL finding, or has an explicit, time-boxed, approved exception on record. RBAC (role-based access control) and least privilege: the deployment identity itself has only the permissions needed to deploy this specific service, and production access more broadly follows a documented, reviewed access model rather than broad, shared credentials. Audit logs: every deployment action (who triggered it, what was deployed, when, and the outcome) is captured in an immutable, queryable log. Immutable infrastructure: the deployment replaces infrastructure with a new, versioned image rather than patching a running instance in place, so the deployed state always traces back to a specific, auditable build. Patching policies: a defined cadence and SLA for applying security patches to base images and dependencies, verified as actually being followed, not just documented as a policy. Evidence collection for audits: the artifacts above (signatures, scan results, approval records, access logs) are retained in a form an auditor can review without requiring a special, one-off data-gathering effort.

Trade-offs between release speed and implementing these controls

Every item on this checklist has a real velocity cost when first implemented (signing and scanning add pipeline time, RBAC design takes real engineering effort, audit-log retention has its own storage and tooling cost), but nearly every one of these, once built, becomes close to free on a per-deployment basis: the marginal cost of THIS deployment being signed, scanned, and logged, given the infrastructure already exists, is small, which is why the honest framing for a team weighing this trade-off is that the real cost is the one-time investment to build the capability, not an ongoing per-release tax. The controls most worth prioritizing first, given limited time, are the ones that are cheapest to build relative to the risk they close (secret scanning and basic signing) versus the ones that take longer to mature (a fully documented, reviewed RBAC model across every production system), which argues for a staged rollout of this checklist itself rather than treating it as an all-or-nothing bar every deployment must clear from day one.

Trade-offs

Treating this as a staged capability build, rather than an immediate, complete gate, means some deployments proceed for a period without every item on the checklist fully in place; that's an honest, deliberate trade for the same reason discussed for shift-left rollouts elsewhere in this topic: demanding full compliance from day one either blocks legitimate work at a team not yet ready for it, or gets quietly bypassed, neither of which actually improves the organization's real security posture.

Direct answer

Run checks in a layered pipeline ordered from cheapest to most expensive: format and lint first (seconds, no cloud access), then static policy/security scanning against the plan or template (still no cloud access), then anything that actually stands up real resources (integration tests, plan review with real credentials), and finally post-deploy verification against the live environment. Each layer is designed to catch a different class of mistake, and putting the cheap checks first means a typo never has to wait for an expensive real-resource test to fail.

The two fundamentally different kinds of check

Before mapping tools to stages, it's worth naming the split explicitly: some checks are fast, syntax- or policy-level, and need no cloud access at all; others genuinely stand up real (usually short-lived) infrastructure to prove it actually works. Confusing the two, or skipping straight to the expensive kind, is the most common mistake in a pipeline like this.

Worked example: a pipeline definition

jobs:
  fmt-and-lint:
    steps:
      - run: terraform fmt -check -recursive
      - run: tflint

  policy-scan:
    needs: fmt-and-lint
    steps:
      - run: checkov -d . --framework terraform

  plan:
    needs: policy-scan
    steps:
      - run: terraform plan -out=plan.tfplan
      - run: terraform show -json plan.tfplan > plan.json
      # a script here fails the job if plan.json contains an unreviewed delete/replace

  integration-test:
    needs: plan
    if: github.event_name == 'push' && github.ref == 'refs/heads/main'
    steps:
      - run: go test ./test/... -run TestModule -timeout 30m
      # this job is the one that actually provisions and tears down real infra

  apply:
    needs: integration-test
    environment: production   # requires manual approval
    steps:
      - run: terraform apply plan.tfplan

The ordering matters: fmt-and-lint and policy-scan run on every commit because they're free and fast; integration-test, which provisions real infrastructure, is scoped to run only on merges to main, not on every push to a feature branch, to bound cost and time.

Trade-offs and pitfalls

• Skipping straight to integration tests (or relying on them to catch what a linter would have caught in seconds) is slow and expensive for no extra safety, put the cheap checks first and let them fail fast.
• Running the real-resource layer on every commit, rather than on merge or nightly, is the single most common way teams accidentally burn cloud spend on a testing pipeline.
• Static policy scanning only catches what it has rules for; it gives false confidence if the rule set isn't kept current with new resource types the team starts using.
• A plan-review gate that only a human reads doesn't scale, encode the "never allow an unreviewed delete on a protected resource" rule as an automated check on the plan JSON, not just a habit.

Framing

Black Friday is a known date with an unpredictable magnitude: you know exactly when the spike hits, but not precisely how large it will be. The plan needs to control cost the rest of the year while accepting that availability risk on this single day is far more expensive than a comparable amount of unused capacity.

Provisioning mix

Keep the normal reserved or committed baseline sized to ordinary daily traffic year-round, that part of the cost model does not change for one event. Ahead of the event, manually pre-scale a known-event buffer rather than trusting reactive autoscaling alone: a 10 to 50 times traffic multiplier can outpace autoscaler reaction time, cold-start latency, and the capacity limits of dependent services (database connection limits, payment gateway throughput) faster than a purely reactive system can respond. Layer on-demand and spot autoscaling above that pre-scaled floor for the unpredictable tail, keeping spot capacity restricted to stateless, safely-retryable tiers only, never the checkout or payment path, where a preemption mid-transaction is unacceptable.

Caching

Static and catalog content (product pages, listings, images) typically dominates request volume during a shopping spike far more than checkout does, so push it aggressively to the CDN edge. Use short-TTL, stale-while-revalidate caching for semi-dynamic content like stock badges, so a slightly stale "in stock" flag is an acceptable trade for not hitting the origin on every page view. Cache expensive read paths (inventory counts, computed pricing) in an application-level cache to keep origin database load roughly flat even as total request volume spikes.

Testing

Load test at 1.5 to 2 times your worst-case forecast, not just the expected case, well before the event, and test the FULL dependent chain, not just your own service: a payment gateway sandbox, the inventory service, database connection limits under sustained concurrency. Run a soak test over a sustained multi-hour window, not just a short burst, to catch memory leaks and connection-pool exhaustion that only appear under prolonged load. Run a deliberate failure exercise, simulating the loss of one availability zone or region during a simulated peak, to confirm failover actually holds up under real load, not just under idle conditions where every test usually happens.

Cost versus risk trade-offs

Pre-provisioning ahead of the event costs real money for capacity that may sit partly unused if the actual spike undershoots the forecast. Under-provisioning risks lost sales during the single highest-revenue day of the year, where lost sales plus reputational damage from an outage on THAT day vastly outweigh the cost of some idle pre-scaled capacity. The risk-adjusted answer is to deliberately over-provision the floor for this one event, even though the same choice would be wasteful as a year-round default.

Operational runbook for a live surge

• Predefined escalation thresholds mapped to predefined actions, for example if p99 latency exceeds twice normal for 5 minutes, manually trigger the pre-approved emergency scale-out template rather than waiting on the reactive autoscaler alone.
• A feature-flag kill switch to shed non-critical functionality (recommendation widgets, reviews, non-essential background jobs) under load, protecting the core browse-to-purchase path first.
• A named on-call and escalation chain for the event window specifically, with clear ownership of the decision to invoke each runbook step, so no one is improvising a decision tree live.
• A deploy freeze for the event window, with a documented, pre-approved emergency-change process as the only exception, so a routine deploy is never the thing that destabilizes the system during the surge.

A provenance attestation is fundamentally a signed statement, so the implementation has two halves that both have to be exactly right: building a deterministic payload to sign, and signing it in a way a verifier can independently reproduce and check.

package main

import (
	"crypto"
	"crypto/rand"
	"crypto/rsa"
	"crypto/sha256"
	"crypto/x509"
	"encoding/base64"
	"encoding/json"
	"encoding/pem"
	"fmt"
	"os"
	"time"
)

type FileDigest struct {
	Path   string `json:"path"`
	Sha256 string `json:"sha256"`
}

// Field order is fixed by the struct definition, which is what makes
// json.Marshal deterministic for this payload: the same logical content
// always produces the same bytes to sign and to re-verify against.
type Attestation struct {
	Inputs    []FileDigest `json:"inputs"`
	Outputs   []FileDigest `json:"outputs"`
	Timestamp string       `json:"timestamp"`
	BuilderID string       `json:"builderId"`
	Signature string       `json:"signature"`
}

func signPayload(payload []byte, priv *rsa.PrivateKey) (string, error) {
	digest := sha256.Sum256(payload)
	sig, err := rsa.SignPKCS1v15(rand.Reader, priv, crypto.SHA256, digest[:])
	if err != nil {
		return "", fmt.Errorf("signing attestation payload: %w", err)
	}
	return base64.StdEncoding.EncodeToString(sig), nil
}

func loadPrivateKey(pemPath string) (*rsa.PrivateKey, error) {
	data, err := os.ReadFile(pemPath)
	if err != nil {
		return nil, fmt.Errorf("reading key file: %w", err)
	}
	block, _ := pem.Decode(data)
	if block == nil {
		return nil, fmt.Errorf("no PEM block found in %s", pemPath)
	}
	key, err := x509.ParsePKCS1PrivateKey(block.Bytes)
	if err != nil {
		return nil, fmt.Errorf("parsing private key: %w", err)
	}
	return key, nil
}

func buildAttestation(inputsPath, outputsPath, keyPath string) (*Attestation, error) {
	inputsRaw, err := os.ReadFile(inputsPath)
	if err != nil {
		return nil, fmt.Errorf("reading inputs file: %w", err)
	}
	var inputs []FileDigest
	if err := json.Unmarshal(inputsRaw, &inputs); err != nil {
		return nil, fmt.Errorf("parsing inputs JSON: %w", err)
	}

	outputsRaw, err := os.ReadFile(outputsPath)
	if err != nil {
		return nil, fmt.Errorf("reading outputs file: %w", err)
	}
	var outputs []FileDigest
	if err := json.Unmarshal(outputsRaw, &outputs); err != nil {
		return nil, fmt.Errorf("parsing outputs JSON: %w", err)
	}

	builderID := os.Getenv("BUILDER_ID")
	if builderID == "" {
		return nil, fmt.Errorf("BUILDER_ID environment variable is not set")
	}

	priv, err := loadPrivateKey(keyPath)
	if err != nil {
		return nil, err
	}

	unsigned := Attestation{
		Inputs:    inputs,
		Outputs:   outputs,
		Timestamp: time.Now().UTC().Format(time.RFC3339),
		BuilderID: builderID,
	}

	payload, err := json.Marshal(unsigned)
	if err != nil {
		return nil, fmt.Errorf("canonicalizing payload: %w", err)
	}

	sig, err := signPayload(payload, priv)
	if err != nil {
		return nil, err
	}

	unsigned.Signature = sig
	return &unsigned, nil
}

func main() {
	if len(os.Args) != 4 {
		fmt.Fprintln(os.Stderr, "usage: provenance <inputs.json> <outputs.json> <private_key.pem>")
		os.Exit(2)
	}
	att, err := buildAttestation(os.Args[1], os.Args[2], os.Args[3])
	if err != nil {
		fmt.Fprintf(os.Stderr, "error: %v\n", err)
		os.Exit(1)
	}
	out, _ := json.MarshalIndent(att, "", "  ")
	fmt.Println(string(out))
}

Deterministic JSON serialization

The struct's field order is fixed at compile time by its definition, so json.Marshal always emits fields in the same order for the same logical content; this is why the payload is built as a typed Attestation struct rather than a map[string]interface{}, since Go's JSON encoding of a map sorts keys alphabetically by default but a struct preserves declaration order, either of which is deterministic on its own, but mixing the two within one payload risks a verifier reconstructing a different byte layout than the signer used. The signature field itself is excluded from the signed payload (set to empty string, or omitted, at signing time) since the signature obviously can't be part of what it's signing over.

Verified

Compiled with go build. Generated a real RSA keypair with OpenSSL, ran the program to produce a signed attestation, then wrote an independent Go verifier that re-reads the attestation, strips the signature field, re-marshals the remaining struct exactly as the signer did, and calls rsa.VerifyPKCS1v15 against the public key: verification succeeded. A tamper test, flipping one character in an input's SHA-256 digest and re-running verification, correctly failed with a signature-mismatch error, confirming the scheme actually detects tampering rather than passing regardless of content.

Trade-offs

Using PKCS1v15 padding and a single RSA key here is simple and fully supported by the Go standard library alone (no external dependency), matching the question's constraint; a production system would more likely use Sigstore's keyless signing (short-lived, OIDC (OpenID Connect)-backed certificates) to avoid the operational burden of protecting a long-lived private key file: that file has to be generated once and then protected for its entire lifetime, encrypted at rest, access-restricted, rotated on a schedule, and revoked immediately if it is ever exposed, and if it does leak, whoever holds it can forge valid attestations indefinitely, or at least until the leak is discovered and the key revoked, with no built-in record of who actually produced a given signature. Keyless signing removes that persisted secret entirely: at sign time the CI runner exchanges a short-lived OIDC token (proof that this exact signing request came from this exact workflow run) for a certificate from a public certificate authority (Sigstore's Fulcio), signs with an ephemeral key that is discarded the moment signing completes, and records the signature alongside the certificate and a transparency-log entry (Rekor) that anyone can later audit. There is no long-lived key file to steal, back up, or rotate, at the cost of depending on the OIDC identity provider and Sigstore's infrastructure being reachable at sign time. Container image signing, discussed separately in this topic, makes the identical trade-off: a persisted key you must guard forever versus an ephemeral, OIDC-backed identity with nothing to steal.