Junior DevOps Engineer Interview Preparation Guide - Microsoft

Direct answer

Waiting stops being the right call once the delay is on your critical path (the chain of work that directly determines your deadline) with no updated ETA, or once the cost of continuing to wait (rework, workarounds, compounding risk) is clearly larger than the cost of escalating. Decide the trigger in advance, not in the moment, and escalate by framing it around the shared deadline and offering to help unblock, not by assigning blame, so the relationship survives the conversation.

Structured elaboration

1. Set the trigger before you need it. At the point you first take on a dependency, agree on what "stalled" means and when you'll escalate if there's no movement, for example, "if there's no updated ETA by [date], I'll raise it." Deciding this ahead of time keeps the eventual call from being an emotionally loaded, in-the-moment judgment.
2. Watch for the signals that waiting has become the wrong call, even without a pre-set trigger: no visible progress or updated estimate, the delay has moved onto your own critical path, you're already absorbing compounding cost (rework, a growing workaround), or the nature of their blocker changed without anyone telling you.
3. Escalate at the right altitude, in order. Start with a direct conversation with the owner (not their manager first, which reads as going around them), then their lead if that doesn't move things, then a cross-functional or executive conversation only if the first two steps don't resolve it. Skipping straight to the top burns trust even when you're right to escalate.
4. Frame the escalation around the shared goal. Bring what you've tried and the concrete impact of the delay, and lead with an offer to help (extra hands, a clearer spec, a joint troubleshooting session) rather than a demand for status. This keeps the conversation collaborative instead of adversarial.
5. When the dependency is an external vendor rather than an internal team, the escalation lever is fundamentally different. There's no peer relationship conversation to have in the same sense: the path runs through contract renegotiation (invoking SLA, or service level agreement, terms, escalating through the vendor's account team) and executive/customer communication about timeline impact, because a vendor delay usually has stakeholders beyond your own working team (customers waiting on the date, your own leadership needing to manage expectations upward). The internal escalation ladder in step 3 assumes a peer relationship you can repair with tone and framing; the vendor case assumes a commercial relationship you manage with contract terms and proactive, honest communication about the schedule impact instead.

Worked example

Two weeks into waiting on an internal platform team's API, with no updated ETA since the first week and the launch date now two weeks out, the trigger from step 1 (no ETA update within a week) has already been crossed. The escalation opens with the owner directly: "This is now going to affect our launch date. What's actually blocking it, and is there anything I can do to help, pair on it, provide test data, take a piece of the work?" Only if that doesn't produce movement within a short, stated window does it go to their lead, framed the same way: shared deadline, concrete impact, an offer to help.

If instead the dependency were owned by an external vendor who'd gone quiet for two weeks on a contracted deliverable, the move isn't a peer conversation with an individual, it's raising the delay through the account relationship against the SLA in the contract, while separately and proactively telling internal leadership (and, if relevant, the customer waiting on the date) what the timeline impact now looks like, rather than continuing to absorb the delay silently and hoping the vendor resolves it before anyone notices.

Trade-offs & pitfalls

• Pitfall: escalating without a pre-agreed trigger, so the decision looks reactive or, worse, personal, when it happens.
• Pitfall: skipping escalation levels internally (going straight to a director) when a direct conversation with the owner hadn't been tried yet, damaging a relationship you'll need again.
• Pitfall: treating a vendor delay like an internal one, i.e., waiting patiently and being "collaborative" with a counterparty who has no equivalent incentive to preserve the relationship the way an internal peer does.
• Senior differentiator: pre-negotiating the escalation threshold when the dependency is first created, not two weeks into silence, and recognizing early which kind of dependency (peer relationship vs. commercial contract) you're actually managing, since that changes which lever you reach for.

A provenance attestation is fundamentally a signed statement, so the implementation has two halves that both have to be exactly right: building a deterministic payload to sign, and signing it in a way a verifier can independently reproduce and check.

package main

import (
	"crypto"
	"crypto/rand"
	"crypto/rsa"
	"crypto/sha256"
	"crypto/x509"
	"encoding/base64"
	"encoding/json"
	"encoding/pem"
	"fmt"
	"os"
	"time"
)

type FileDigest struct {
	Path   string `json:"path"`
	Sha256 string `json:"sha256"`
}

// Field order is fixed by the struct definition, which is what makes
// json.Marshal deterministic for this payload: the same logical content
// always produces the same bytes to sign and to re-verify against.
type Attestation struct {
	Inputs    []FileDigest `json:"inputs"`
	Outputs   []FileDigest `json:"outputs"`
	Timestamp string       `json:"timestamp"`
	BuilderID string       `json:"builderId"`
	Signature string       `json:"signature"`
}

func signPayload(payload []byte, priv *rsa.PrivateKey) (string, error) {
	digest := sha256.Sum256(payload)
	sig, err := rsa.SignPKCS1v15(rand.Reader, priv, crypto.SHA256, digest[:])
	if err != nil {
		return "", fmt.Errorf("signing attestation payload: %w", err)
	}
	return base64.StdEncoding.EncodeToString(sig), nil
}

func loadPrivateKey(pemPath string) (*rsa.PrivateKey, error) {
	data, err := os.ReadFile(pemPath)
	if err != nil {
		return nil, fmt.Errorf("reading key file: %w", err)
	}
	block, _ := pem.Decode(data)
	if block == nil {
		return nil, fmt.Errorf("no PEM block found in %s", pemPath)
	}
	key, err := x509.ParsePKCS1PrivateKey(block.Bytes)
	if err != nil {
		return nil, fmt.Errorf("parsing private key: %w", err)
	}
	return key, nil
}

func buildAttestation(inputsPath, outputsPath, keyPath string) (*Attestation, error) {
	inputsRaw, err := os.ReadFile(inputsPath)
	if err != nil {
		return nil, fmt.Errorf("reading inputs file: %w", err)
	}
	var inputs []FileDigest
	if err := json.Unmarshal(inputsRaw, &inputs); err != nil {
		return nil, fmt.Errorf("parsing inputs JSON: %w", err)
	}

	outputsRaw, err := os.ReadFile(outputsPath)
	if err != nil {
		return nil, fmt.Errorf("reading outputs file: %w", err)
	}
	var outputs []FileDigest
	if err := json.Unmarshal(outputsRaw, &outputs); err != nil {
		return nil, fmt.Errorf("parsing outputs JSON: %w", err)
	}

	builderID := os.Getenv("BUILDER_ID")
	if builderID == "" {
		return nil, fmt.Errorf("BUILDER_ID environment variable is not set")
	}

	priv, err := loadPrivateKey(keyPath)
	if err != nil {
		return nil, err
	}

	unsigned := Attestation{
		Inputs:    inputs,
		Outputs:   outputs,
		Timestamp: time.Now().UTC().Format(time.RFC3339),
		BuilderID: builderID,
	}

	payload, err := json.Marshal(unsigned)
	if err != nil {
		return nil, fmt.Errorf("canonicalizing payload: %w", err)
	}

	sig, err := signPayload(payload, priv)
	if err != nil {
		return nil, err
	}

	unsigned.Signature = sig
	return &unsigned, nil
}

func main() {
	if len(os.Args) != 4 {
		fmt.Fprintln(os.Stderr, "usage: provenance <inputs.json> <outputs.json> <private_key.pem>")
		os.Exit(2)
	}
	att, err := buildAttestation(os.Args[1], os.Args[2], os.Args[3])
	if err != nil {
		fmt.Fprintf(os.Stderr, "error: %v\n", err)
		os.Exit(1)
	}
	out, _ := json.MarshalIndent(att, "", "  ")
	fmt.Println(string(out))
}

Deterministic JSON serialization

The struct's field order is fixed at compile time by its definition, so json.Marshal always emits fields in the same order for the same logical content; this is why the payload is built as a typed Attestation struct rather than a map[string]interface{}, since Go's JSON encoding of a map sorts keys alphabetically by default but a struct preserves declaration order, either of which is deterministic on its own, but mixing the two within one payload risks a verifier reconstructing a different byte layout than the signer used. The signature field itself is excluded from the signed payload (set to empty string, or omitted, at signing time) since the signature obviously can't be part of what it's signing over.

Verified

Compiled with go build. Generated a real RSA keypair with OpenSSL, ran the program to produce a signed attestation, then wrote an independent Go verifier that re-reads the attestation, strips the signature field, re-marshals the remaining struct exactly as the signer did, and calls rsa.VerifyPKCS1v15 against the public key: verification succeeded. A tamper test, flipping one character in an input's SHA-256 digest and re-running verification, correctly failed with a signature-mismatch error, confirming the scheme actually detects tampering rather than passing regardless of content.

Trade-offs

Using PKCS1v15 padding and a single RSA key here is simple and fully supported by the Go standard library alone (no external dependency), matching the question's constraint; a production system would more likely use Sigstore's keyless signing (short-lived, OIDC (OpenID Connect)-backed certificates) to avoid the operational burden of protecting a long-lived private key file: that file has to be generated once and then protected for its entire lifetime, encrypted at rest, access-restricted, rotated on a schedule, and revoked immediately if it is ever exposed, and if it does leak, whoever holds it can forge valid attestations indefinitely, or at least until the leak is discovered and the key revoked, with no built-in record of who actually produced a given signature. Keyless signing removes that persisted secret entirely: at sign time the CI runner exchanges a short-lived OIDC token (proof that this exact signing request came from this exact workflow run) for a certificate from a public certificate authority (Sigstore's Fulcio), signs with an ephemeral key that is discarded the moment signing completes, and records the signature alongside the certificate and a transparency-log entry (Rekor) that anyone can later audit. There is no long-lived key file to steal, back up, or rotate, at the cost of depending on the OIDC identity provider and Sigstore's infrastructure being reachable at sign time. Container image signing, discussed separately in this topic, makes the identical trade-off: a persisted key you must guard forever versus an ephemeral, OIDC-backed identity with nothing to steal.

Direct answer
A zero-downtime rolling update needs three things working together: connection draining so in-flight requests to a pod being removed finish before it's killed, a pre-stop hook that gives the pod time to deregister from load-balancing before the process actually terminates, and a fast-enough detection mechanism to halt the rollout if the new version starts failing partway through, rather than blindly continuing to replace every remaining pod.

Structured elaboration

1. Connection draining: when Kubernetes decides to terminate a pod, it first removes it from the Service's endpoint list (stops NEW traffic from being routed to it) and sends SIGTERM, but requests already in flight to that pod need TIME to complete; the app needs to keep serving in-flight requests during the grace period rather than immediately shutting down on SIGTERM.
2. Pre-stop hook: a preStop lifecycle hook that sleeps for a few seconds BEFORE the app actually starts shutting down handles a real race condition: endpoint removal propagation across the cluster (kube-proxy, any service mesh sidecars, external load balancers) isn't instant, so without a brief delay, some fraction of new requests can still be routed to a pod that's already begun terminating.
3. terminationGracePeriodSeconds: set long enough to cover both the preStop delay AND the time needed for genuinely in-flight requests to complete, or Kubernetes will SIGKILL the process before it's actually done draining.
4. Halting the rollout mid-way: this requires either a readiness-probe-based automatic halt (if enough new pods fail readiness, Kubernetes itself won't proceed past that batch) or an external controller (Argo Rollouts, Flagger) actively watching error-rate/latency metrics and pausing the rollout, since a plain Kubernetes rolling update has no metric-based automatic halt built in beyond the probe mechanism.

Worked example

lifecycle:
  preStop:
    exec:
      command: ["sh", "-c", "sleep 5"]
terminationGracePeriodSeconds: 30

A 5-second preStop sleep gives endpoint-removal propagation time to complete across the cluster before the app starts its own graceful shutdown, and a 30-second total grace period gives genuinely long-running in-flight requests (up to roughly 25 seconds after the preStop sleep) time to finish before Kubernetes force-kills the process. For detecting a regression mid-rollout, pairing a plain rolling update with tight readiness probes (so a genuinely broken new pod never passes readiness and Kubernetes won't proceed to replace MORE old pods with it) provides a basic safety net, though it only catches failures severe enough to fail the readiness probe itself, not a subtler metric-based regression, which needs an external controller watching real traffic metrics to catch.

Trade-offs and pitfalls
A preStop sleep that's too short reintroduces the very race condition it's meant to prevent (a small window of dropped connections); one that's too long slows down every rollout unnecessarily. The most common gap is relying purely on readiness-probe-based halting and assuming that's equivalent to real canary-style metric-based analysis, when it only catches failures severe enough to make a pod fail its OWN health check, missing anything that degrades user-facing behavior without the pod itself appearing unhealthy.

Direct answer
Write a resource block of the correct type declaring only the arguments you intend to manage, run terraform import (or a declarative import block) to attach the real resource's ID to that address, then run terraform plan repeatedly, reconciling your HCL against what it reports, until plan shows no diff. Only then does apply touch anything, and by that point it's a no-op. Nothing about the live VPC or database changes during any of this, because import only writes state and plan never mutates anything, so the "without downtime" requirement is really just discipline about not applying until plan is clean.

Structured elaboration

1. Inspect the real resource before writing HCL
Don't write the resource block from memory. Look up the actual current attributes (via the console, CLI, or the provider's data source) so your first draft doesn't propose spurious changes the moment you import.

2. Write the minimal matching resource block
Keep it to the arguments you actually plan to manage going forward; leave out fields you expect the provider to compute (endpoints, generated ARNs) rather than guessing values for them.

resource "aws_db_instance" "example" {
  identifier = "my-db-identifier"
  # do not add attributes you expect AWS to compute (endpoint, address)
}

3. Import

terraform import aws_db_instance.example my-db-identifier

As of Terraform 1.5+ you can express the same step declaratively with an import block plus terraform plan -generate-config-out=generated.tf, which scaffolds a starting resource block for you instead of hand-writing one, useful when importing many resources at once.

4. Reconcile computed and drifted attributes
Run terraform plan. Anything present in the real resource but missing or different in your HCL shows up as a proposed change. For attributes the provider computes and you don't want Terraform fighting over (an RDS endpoint, a final snapshot identifier), either add the real value explicitly if you want to manage it, or mark it in a lifecycle block so Terraform stops treating drift there as something to fix.

lifecycle {
  ignore_changes = [endpoint, address]
}

5. Verify, then only apply if you mean to change something
terraform state show aws_db_instance.example confirms the mapped ID and attributes. Keep iterating on plan until it's clean. If you genuinely want to change a value going forward, that apply is a deliberate, reviewed change, not an accidental side effect of the import.

Nested/child resources aren't imported automatically
A hand-built VPC or database typically has dependent pieces, a DB subnet group, associated security groups, route tables, that each need their own import call. Importing the parent resource doesn't pull its children into management for you.

Worked example
Importing a hand-built RDS instance end to end:

terraform import aws_db_instance.example my-db-identifier
terraform state show aws_db_instance.example
terraform plan

A first plan on a genuinely hand-built instance will usually show a handful of proposed changes on computed fields like endpoint, address, and tags_all, since those were never in your HCL to begin with. You add ignore_changes for the ones you don't want to manage and explicit values for the ones you do, then re-run plan until it's a clean no-op, which is your confirmation that apply is now safe.

Trade-offs & pitfalls
Skipping the "inspect the real resource first" step is the most common way this goes wrong: you import, run plan, and discover it wants to replace the DB instance's engine or storage_type because you guessed a value instead of reading the real one, right after the whole point was to avoid touching it. Always back up state before starting (terraform state pull > backup.tfstate), and where possible rehearse the import against a non-production copy of the resource type first. Bulk-importing dozens of hand-built resources one CLI call at a time doesn't scale, that's what the 1.5+ import block with -generate-config-out is for.

Direct answer

A CI/CD (continuous integration/continuous delivery) pipeline for Terraform needs five stages in a strict order, because each stage's output is the next stage's required input: lint (fast, cheap syntax/style feedback), plan (compute what would actually change, without changing anything), publish the plan as an artifact (so the EXACT plan that gets reviewed is the EXACT plan that later gets applied, not a freshly recomputed one), manual approval (a human decision gate on that specific, already-computed plan), and apply (execute exactly the approved plan). The apply step's credentials need to be more tightly scoped and more carefully secured than every earlier stage's, since it is the only stage that actually touches live infrastructure.

Structured elaboration

Stage 1, lint. terraform fmt -check and terraform validate, plus any policy-as-code static checks that do not require live provider credentials (a check that a resource type is on an approved allowlist, for instance). Fast (seconds), runs on every push including drafts, gives the fastest possible feedback loop before anything more expensive runs.

Stage 2, plan. terraform plan -out=tfplan, using READ-ONLY credentials scoped to exactly what planning requires (Terraform needs to READ current state and query provider APIs to compute a diff, it does not need WRITE access at this stage). The binary plan file (tfplan) is the stage's real output, not just its console log.

Stage 3, publish the plan artifact. Upload tfplan (and, for reviewability, its human-readable rendering via terraform show tfplan) to the CI system's artifact store, tagged with the exact commit SHA it was computed against. This is the step that makes "what gets approved" and "what gets applied" provably the SAME plan, rather than the apply stage silently recomputing a fresh plan that could differ if anything changed in between (a provider version bump, a concurrent state change) between review and apply. Alongside the binary tfplan and its human-readable text rendering, also generate a machine-readable JSON rendering (terraform show -json tfplan > plan.json) and publish it too: this is the form policy-as-code tooling (Conftest, OPA, or a Sentinel-style policy check) actually consumes to programmatically evaluate the plan against a rule set ("no security group may open 0.0.0.0/0 on port 22", for instance) as an automated gate ahead of the human approval step, something neither the binary state-format plan file nor the human-readable text rendering supports directly, since both are meant for a human or Terraform itself to read, not for a policy engine to parse structurally.

Stage 4, manual approval. A human (or, for lower-risk environments, an automated policy check standing in for a human) reviews the PUBLISHED plan artifact specifically, not a freshly regenerated one, and approves or rejects. This is a genuinely separate gate from the PR's own code-review approval: code review judges the DIFF; this gate judges the ACTUAL PLANNED EFFECT, which can differ from what the diff alone suggests.

Stage 5, apply. terraform apply tfplan (applying the EXACT published artifact, not re-planning), using WRITE-scoped credentials, ideally short-lived and issued specifically for this run (OIDC federation to the cloud provider rather than a long-lived static secret stored in the CI system).

How apply is executed, CI runner vs. internal orchestration. A CI RUNNER executing apply directly is the simpler design, fewer moving parts, but means the CI system itself needs to hold (even briefly, even scoped) write credentials to production infrastructure. An INTERNAL ORCHESTRATION service (a dedicated apply-runner behind its own API, which CI calls rather than applying directly) adds a component but lets that dedicated service be the ONLY thing that ever holds write credentials, with CI only ever able to trigger it via a scoped, auditable API call, a meaningfully smaller credential-exposure surface for organizations at a scale where that trade-off is worth the added complexity.

Securing apply credentials. Short-lived (per-run, OIDC-federated) over long-lived static secrets wherever the cloud provider supports it; scoped to exactly the resources/actions this specific pipeline's applies legitimately touch, not a broad admin role reused across every pipeline; and never exposed in logs (the CI system's own credential-injection mechanism, not an explicit export in a script step that could accidentally get echoed).

Trade-offs and pitfalls

• Common mistake: re-running terraform plan at apply time instead of applying the exact published plan artifact. If ANYTHING changed between the approved plan and apply time (another merged change, a provider update), a freshly recomputed plan at apply time can differ from what was actually reviewed and approved, silently applying something nobody signed off on in that exact form; applying the saved .tfplan file specifically is what closes this gap.
• Read-only credentials at the plan stage are a real, not merely theoretical, blast-radius reduction. A compromised or misconfigured PLAN stage (an untrusted PR from a fork, for example) with only read access cannot modify infrastructure even if it tried; this matters specifically for organizations that run plan automatically on every PR, including from external or less-trusted contributors.
• The internal-orchestration-service design is genuinely higher operational overhead and is not the right default for a small team, it earns its complexity at a scale (many pipelines, many teams, high compliance bar) where minimizing the number of places holding write credentials materially reduces real risk; a small, single-team setup applying directly from CI runners is a reasonable, simpler choice and should not be treated as automatically insufficient.
• Audit records for approvals need to capture WHICH plan artifact (by content hash or artifact ID, not just "the PR's plan") was approved, a generic "approved" log entry with no link to the exact plan content is a weaker audit trail than one an external audit could actually trace end to end.

Direct answer

Size against the throughput a single GPU can sustain while still meeting your latency ceiling, not its theoretical peak throughput, because batching trades latency for throughput and both constraints have to hold at once. Divide the required load by that per-GPU number, round up, and add headroom for failover and variance.

Structured elaboration

• Batching groups concurrent requests so GPU compute is shared across them, raising throughput per GPU but adding queuing and shared-compute wait time to each request.
• Larger batch sizes keep raising throughput per GPU only up to a point; eventually you hit the p95 latency budget or the GPU's memory limit (the KV-cache, per-request memory the model keeps around during generation so it doesn't have to recompute earlier tokens on every step, grows with both batch size and sequence length).
• Fleet size = required QPS divided by per-GPU sustainable QPS at your latency ceiling, plus N+1 redundancy; steady-state size and autoscaling headroom are two different numbers.

Worked example

Assume profiling shows batch size 1 takes 80ms per request (p95 latency, the 95th-percentile response time), well under the 300ms budget:

$\text{unbatched QPS per GPU} = \frac{1000\text{ms}}{80\text{ms}} = 12.5$
$\text{unbatched fleet} = \lceil 200 / 12.5 \rceil = 16 \text{ GPUs}$

Now assume batch size 8 takes 140ms per batch (still under the 300ms budget):

$\text{batched QPS per GPU} = \frac{1000\text{ms}}{140\text{ms}} \times 8 \approx 57.1$
$\text{batched fleet} = \lceil 200 / 57.1 \rceil = 4 \text{ GPUs}$

Batching cuts the fleet from 16 to 4 GPUs, a 4x reduction, while per-request latency rises from 80ms to 140ms, still inside the 300ms p95 budget. Add one GPU for failover: 5 GPUs total versus 17 unbatched.

Trade-offs and pitfalls

Bigger batches help only until you hit a new latency ceiling or run out of GPU memory; past that you are compute-bound, not batching-bound, and no batching buys back headroom. Static batching forces every request to wait for the batch to fill; dynamic or continuous batching captures most of the savings without that wait.

What the interviewer probes next

Expect questions on how you'd handle the request that arrives just after a batch closes, how you'd autoscale for a traffic spike without over-provisioning steady state, and what happens to this math when sequence lengths vary widely.

Approach (brief)
I’ll use stateful Security Groups (SGs) for host-level allowlists and stateless Network ACLs (NACLs) for coarse subnet-level filtering and extra defense-in-depth. SGs permit return traffic automatically; NACLs must include ephemeral port ranges for responses.

Security Groups (stateful)

• Web SG (frontend)
  • Inbound: TCP 80,443 from 0.0.0.0/0 (or CDN IP list) — allow
  • Inbound: TCP 22 from admin CIDR (e.g., 10.0.0.0/24) — allow
  • Outbound: TCP 1024-65535 to App SG — allow (ephemeral for responses)
• App SG (application)
  • Inbound: TCP 8080 (or 443) from Web SG — allow (reference SG)
  • Inbound: TCP 22 from admin CIDR — allow
  • Outbound: TCP 3306 to DB SG — allow
• DB SG (data)
  • Inbound: TCP 3306 (MySQL) from App SG only — allow
  • Inbound: TCP 22 from admin CIDR — allow (or disable in prod)
  • Outbound: restrict to required services (e.g., backups) or deny all

Notes: Use SG references (not CIDRs) for east-west traffic to enforce least privilege. Deny all other inbound by default.

Network ACLs (stateless, per subnet)

• Web subnet NACL
  • Rule 100 Allow TCP 80,443 from 0.0.0.0/0
  • Rule 110 Allow TCP 1024-65535 to App subnet (for ephemeral client ports) — outbound/inbound as needed
  • Rule 120 Deny all (lower priority)
• App subnet NACL
  • Rule 100 Allow TCP 8080 from Web subnet CIDR
  • Rule 110 Allow TCP 1024-65535 to Web subnet CIDR (return)
  • Rule 120 Allow TCP 3306 to DB subnet CIDR (outbound)
  • Rule 130 Deny all
• DB subnet NACL
  • Rule 100 Allow TCP 3306 from App subnet CIDR
  • Rule 110 Allow TCP 1024-65535 to App subnet CIDR (return)
  • Rule 120 Deny all

Notes: Because NACLs are stateless, include both directions for ephemeral ranges (1024-65535) and explicit allow rules for both inbound/outbound pairs. Keep rule numbering consistent and explicit deny at the end.

Why these choices

• SGs stateful + SG references give fine-grained, least-privilege controls between tiers and simplify rule maintenance.
• NACLs add subnet-level protection and an extra block against misconfigured instances; being stateless forces explicit bi-directional rules (safer, explicit).
• Restrict management (SSH) to admin CIDR or use bastion/SSO; avoid opening DB or internal ports to Internet.

Operational recommendations

• Manage via IaC (Terraform/CloudFormation) to enforce reproducibility.
• Rotate admin CIDRs, use bastion + session manager, enable logging (VPC Flow Logs), and monitor with alerts for unexpected allow patterns.

Direct answer: Structure the postmortem around a timeline, not a blame narrative: pull the deployment timestamp, overlay it against the SLI dashboard and the error-budget burn, quantify exactly how much budget the incident consumed, then work backward from the first observed symptom to root cause using traces and logs. The walk-away artifact is a concrete, owned action list, not a general "be more careful" resolution.

Structured elaboration

Data to pull together

• Timeline anchor: the deployment or config-change timestamp, this is almost always where the investigation starts, since most error-budget-burning incidents correlate tightly with a recent change.
• SLO dashboard: the SLI's value before, during, and after the incident, plus the error-budget burn-rate chart so the team can see not just "it broke" but "how fast it was burning."
• Errors by type: which endpoints, which status codes, which exception types spiked, this narrows root cause from "something broke" to "this specific code path broke."
• Traces and logs: a handful of representative failing traces (not all of them, a representative sample is enough to see the pattern) correlated with structured logs via trace ID.
• Resource and dependency metrics: CPU/memory/GC on the affected hosts, and latency/error rate on anything downstream, to distinguish a code bug from a resource exhaustion or dependency failure.

How to explain the impact

• Quantify budget consumed as a fraction of the period's total budget (see worked example), not just "the SLO dropped." A number stakeholders can compare to "how much budget is left this month" is far more actionable than a description of the dashboard dip.
• State the user-facing impact in concrete terms: percentage of requests affected, estimated number of unique users or sessions, and duration of impact.

What you walk away with

• A root cause stated at the mechanism level ("the new code path opened a DB connection per request without pooling, exhausting the connection pool under peak load"), not at the symptom level ("errors increased").
• A small number of owned, dated action items, tightening canary criteria, adding a specific alert that would have caught this faster, a pre-deploy check that would have caught the bug in the first place. Each with a named owner and a due date, reviewed at a fixed checkpoint (commonly one week out) to confirm they actually landed.
• A decision on rollback vs patch-forward for this specific incident, documented with the reasoning, so the next similar incident has a precedent to reference instead of re-litigating the decision from scratch.

Worked example

Assume the service has a monthly correctness error budget of 2,000 failed requests (at 99.9% SLO over 2,000,000 monthly requests, following the same derivation as the SLI/SLO worked example above). A bad deploy is live for 45 minutes before rollback, during which the error rate on this service jumped from a normal 0.05% to 8%, with average traffic of 400 requests/minute during that window.

Failed requests during the incident:

Baseline failures that would have happened anyway in that window (at the normal 0.05% rate) don't count against the incident's attributable impact:

Attributable incident impact:

As a fraction of the monthly budget:

That single number, 72% of the month's budget gone in under an hour, is what belongs at the top of the postmortem's impact section: it makes the severity legible to anyone reading the doc, engineer or not, without them needing to interpret a dashboard.

Trade-offs & pitfalls

• A postmortem that stops at "we rolled back and errors went away" without root-causing the mechanism leaves the door open for the same bug to resurface in a different code path. Push to the mechanism, not just the symptom.
• Action items with no owner or due date are the single most common reason postmortem follow-through fails; the one-week checkpoint above exists specifically to catch that before it becomes a pattern.
• The same structure applies when the trigger is a sustained SLO violation accumulating over several days rather than one bad deploy, the difference is that the timeline anchor isn't a single deploy event, it's a slower-moving cause (a capacity ceiling being approached, a dependency degrading gradually), so the investigation should explicitly check "did this correlate with a change, or was there no single change and the system just outgrew its current capacity."

Direct answer

For a small team, static fixtures, factory patterns, seeded databases, and mocking external services each cover a different, low-overhead niche: fixtures for fixed known-good data, factories for generating varied-but-structured data programmatically, seeded databases for anything needing real persistence behavior, and mocks for anything outside your own codebase.

Structured elaboration

• Static fixtures: a fixed JSON/YAML/code snippet representing known test data, loaded as-is. Typical use: unit tests verifying specific known-input behavior. Downside: doesn't scale to needing many variations, and can drift out of sync with the real schema unless actively maintained.
• Factory patterns: a function or class that programmatically builds a valid object with sensible defaults, letting a test override just the fields it cares about. Typical use: any test needing "a valid X, except this one field is different." Downside: factories can hide what data actually matters for a given test if overused without care for readability.
• Seeded databases: populate a real (test) database with a known baseline before running tests that need actual persistence behavior. Typical use: integration tests verifying query behavior, constraints, or transactions. Downside: adds real setup/teardown time and infrastructure dependency compared to in-memory fixtures.
• Mocking external services: replace a third-party or otherwise external dependency with a controllable double. Typical use: any test where the point is your own logic, not the external service's behavior. Downside: risk of the mock drifting from the real service's actual behavior over time if not periodically validated.

Worked example

A small e-commerce team uses factory functions (make_order(status="pending", **overrides)) for most unit tests, so each test can express exactly what matters ("an order with status=refunded") without repeating boilerplate; a lightly seeded test database for integration tests verifying that order queries and constraints behave correctly; and a mock for the external shipping-rate API, since exercising the real API in every test run would be slow and outside the team's control.

Trade-offs & pitfalls

For a small team specifically, the practical risk is over-investing in an elaborate synthetic-data generation system before it's actually needed; starting with fixtures and factories and adding seeded databases or mocks only where a specific test genuinely needs them keeps the setup proportionate to the team's actual scale.

Approach summary
I’d treat reproducible images as a pipeline problem: eliminate non-determinism (timestamps, ordering, ambient state, base image mutations) and capture build cache/state. Below are concrete techniques and why they matter.

Control timestamps

• Set SOURCE_DATE_EPOCH in CI to a fixed UNIX time so tools embed stable timestamps.
• Normalize file mtimes during build: e.g., in Dockerfile run a step that find . -print0 | xargs -0 touch -h --no-dereference --date="@${SOURCE_DATE_EPOCH}" before packaging layers.

File ordering

• Ensure deterministic archives: sort file lists before adding (git ls-files | sort).
• When copying, copy a tar created with deterministic options (tar --sort=name --mtime="@${SOURCE_DATE_EPOCH}" --owner=0 --group=0 --numeric-owner).

Deterministic build tools

• Use toolchains that support reproducible outputs (go build -trimpath, deterministic jar creation with reproducible-jar plugin).
• Pin compiler flags and strip build IDs or hashes (strip or set linker flags).

Pin base image digests

• Never use floating tags. FROM ubuntu@sha256:<digest> guarantees identical base layer content.

BuildKit / buildx inline cache export/import

• Use BuildKit to reuse and export/import cache to make layer content deterministic across CI agents.
• Example: export inline cache so the built image includes build cache metadata and can be imported:

docker buildx build --cache-to=type=inline,mode=max --cache-from=type=inline --tag repo/image:sha .

• Or export to remote cache (registry or s3) for shared CI runners:

docker buildx build --cache-to=type=registry,ref=repo/cache:buildcache --cache-from=type=registry,ref=repo/cache:buildcache ...

Other practical controls

• Pin dependency versions and lockfiles; vendor deps when possible.
• Strip non-deterministic metadata (build IDs, timestamps) via tool options.
• Verify by rebuilding twice in CI and compare sha256 digests (docker image inspect / content digest).

Trade-offs

• Pinning and strict reproducibility can be heavier operationally (more storage for pinned images, cache management) but dramatically improves traceability and security.