Junior DevOps Engineer Interview Preparation Guide - Microsoft

Direct answer
A zero-downtime rolling update needs three things working together: connection draining so in-flight requests to a pod being removed finish before it's killed, a pre-stop hook that gives the pod time to deregister from load-balancing before the process actually terminates, and a fast-enough detection mechanism to halt the rollout if the new version starts failing partway through, rather than blindly continuing to replace every remaining pod.

Structured elaboration

1. Connection draining: when Kubernetes decides to terminate a pod, it first removes it from the Service's endpoint list (stops NEW traffic from being routed to it) and sends SIGTERM, but requests already in flight to that pod need TIME to complete; the app needs to keep serving in-flight requests during the grace period rather than immediately shutting down on SIGTERM.
2. Pre-stop hook: a preStop lifecycle hook that sleeps for a few seconds BEFORE the app actually starts shutting down handles a real race condition: endpoint removal propagation across the cluster (kube-proxy, any service mesh sidecars, external load balancers) isn't instant, so without a brief delay, some fraction of new requests can still be routed to a pod that's already begun terminating.
3. terminationGracePeriodSeconds: set long enough to cover both the preStop delay AND the time needed for genuinely in-flight requests to complete, or Kubernetes will SIGKILL the process before it's actually done draining.
4. Halting the rollout mid-way: this requires either a readiness-probe-based automatic halt (if enough new pods fail readiness, Kubernetes itself won't proceed past that batch) or an external controller (Argo Rollouts, Flagger) actively watching error-rate/latency metrics and pausing the rollout, since a plain Kubernetes rolling update has no metric-based automatic halt built in beyond the probe mechanism.

Worked example

lifecycle:
  preStop:
    exec:
      command: ["sh", "-c", "sleep 5"]
terminationGracePeriodSeconds: 30

A 5-second preStop sleep gives endpoint-removal propagation time to complete across the cluster before the app starts its own graceful shutdown, and a 30-second total grace period gives genuinely long-running in-flight requests (up to roughly 25 seconds after the preStop sleep) time to finish before Kubernetes force-kills the process. For detecting a regression mid-rollout, pairing a plain rolling update with tight readiness probes (so a genuinely broken new pod never passes readiness and Kubernetes won't proceed to replace MORE old pods with it) provides a basic safety net, though it only catches failures severe enough to fail the readiness probe itself, not a subtler metric-based regression, which needs an external controller watching real traffic metrics to catch.

Trade-offs and pitfalls
A preStop sleep that's too short reintroduces the very race condition it's meant to prevent (a small window of dropped connections); one that's too long slows down every rollout unnecessarily. The most common gap is relying purely on readiness-probe-based halting and assuming that's equivalent to real canary-style metric-based analysis, when it only catches failures severe enough to make a pod fail its OWN health check, missing anything that degrades user-facing behavior without the pod itself appearing unhealthy.

A provenance attestation is fundamentally a signed statement, so the implementation has two halves that both have to be exactly right: building a deterministic payload to sign, and signing it in a way a verifier can independently reproduce and check.

package main

import (
	"crypto"
	"crypto/rand"
	"crypto/rsa"
	"crypto/sha256"
	"crypto/x509"
	"encoding/base64"
	"encoding/json"
	"encoding/pem"
	"fmt"
	"os"
	"time"
)

type FileDigest struct {
	Path   string `json:"path"`
	Sha256 string `json:"sha256"`
}

// Field order is fixed by the struct definition, which is what makes
// json.Marshal deterministic for this payload: the same logical content
// always produces the same bytes to sign and to re-verify against.
type Attestation struct {
	Inputs    []FileDigest `json:"inputs"`
	Outputs   []FileDigest `json:"outputs"`
	Timestamp string       `json:"timestamp"`
	BuilderID string       `json:"builderId"`
	Signature string       `json:"signature"`
}

func signPayload(payload []byte, priv *rsa.PrivateKey) (string, error) {
	digest := sha256.Sum256(payload)
	sig, err := rsa.SignPKCS1v15(rand.Reader, priv, crypto.SHA256, digest[:])
	if err != nil {
		return "", fmt.Errorf("signing attestation payload: %w", err)
	}
	return base64.StdEncoding.EncodeToString(sig), nil
}

func loadPrivateKey(pemPath string) (*rsa.PrivateKey, error) {
	data, err := os.ReadFile(pemPath)
	if err != nil {
		return nil, fmt.Errorf("reading key file: %w", err)
	}
	block, _ := pem.Decode(data)
	if block == nil {
		return nil, fmt.Errorf("no PEM block found in %s", pemPath)
	}
	key, err := x509.ParsePKCS1PrivateKey(block.Bytes)
	if err != nil {
		return nil, fmt.Errorf("parsing private key: %w", err)
	}
	return key, nil
}

func buildAttestation(inputsPath, outputsPath, keyPath string) (*Attestation, error) {
	inputsRaw, err := os.ReadFile(inputsPath)
	if err != nil {
		return nil, fmt.Errorf("reading inputs file: %w", err)
	}
	var inputs []FileDigest
	if err := json.Unmarshal(inputsRaw, &inputs); err != nil {
		return nil, fmt.Errorf("parsing inputs JSON: %w", err)
	}

	outputsRaw, err := os.ReadFile(outputsPath)
	if err != nil {
		return nil, fmt.Errorf("reading outputs file: %w", err)
	}
	var outputs []FileDigest
	if err := json.Unmarshal(outputsRaw, &outputs); err != nil {
		return nil, fmt.Errorf("parsing outputs JSON: %w", err)
	}

	builderID := os.Getenv("BUILDER_ID")
	if builderID == "" {
		return nil, fmt.Errorf("BUILDER_ID environment variable is not set")
	}

	priv, err := loadPrivateKey(keyPath)
	if err != nil {
		return nil, err
	}

	unsigned := Attestation{
		Inputs:    inputs,
		Outputs:   outputs,
		Timestamp: time.Now().UTC().Format(time.RFC3339),
		BuilderID: builderID,
	}

	payload, err := json.Marshal(unsigned)
	if err != nil {
		return nil, fmt.Errorf("canonicalizing payload: %w", err)
	}

	sig, err := signPayload(payload, priv)
	if err != nil {
		return nil, err
	}

	unsigned.Signature = sig
	return &unsigned, nil
}

func main() {
	if len(os.Args) != 4 {
		fmt.Fprintln(os.Stderr, "usage: provenance <inputs.json> <outputs.json> <private_key.pem>")
		os.Exit(2)
	}
	att, err := buildAttestation(os.Args[1], os.Args[2], os.Args[3])
	if err != nil {
		fmt.Fprintf(os.Stderr, "error: %v\n", err)
		os.Exit(1)
	}
	out, _ := json.MarshalIndent(att, "", "  ")
	fmt.Println(string(out))
}

Deterministic JSON serialization

The struct's field order is fixed at compile time by its definition, so json.Marshal always emits fields in the same order for the same logical content; this is why the payload is built as a typed Attestation struct rather than a map[string]interface{}, since Go's JSON encoding of a map sorts keys alphabetically by default but a struct preserves declaration order, either of which is deterministic on its own, but mixing the two within one payload risks a verifier reconstructing a different byte layout than the signer used. The signature field itself is excluded from the signed payload (set to empty string, or omitted, at signing time) since the signature obviously can't be part of what it's signing over.

Verified

Compiled with go build. Generated a real RSA keypair with OpenSSL, ran the program to produce a signed attestation, then wrote an independent Go verifier that re-reads the attestation, strips the signature field, re-marshals the remaining struct exactly as the signer did, and calls rsa.VerifyPKCS1v15 against the public key: verification succeeded. A tamper test, flipping one character in an input's SHA-256 digest and re-running verification, correctly failed with a signature-mismatch error, confirming the scheme actually detects tampering rather than passing regardless of content.

Trade-offs

Using PKCS1v15 padding and a single RSA key here is simple and fully supported by the Go standard library alone (no external dependency), matching the question's constraint; a production system would more likely use Sigstore's keyless signing (short-lived, OIDC (OpenID Connect)-backed certificates) to avoid the operational burden of protecting a long-lived private key file: that file has to be generated once and then protected for its entire lifetime, encrypted at rest, access-restricted, rotated on a schedule, and revoked immediately if it is ever exposed, and if it does leak, whoever holds it can forge valid attestations indefinitely, or at least until the leak is discovered and the key revoked, with no built-in record of who actually produced a given signature. Keyless signing removes that persisted secret entirely: at sign time the CI runner exchanges a short-lived OIDC token (proof that this exact signing request came from this exact workflow run) for a certificate from a public certificate authority (Sigstore's Fulcio), signs with an ephemeral key that is discarded the moment signing completes, and records the signature alongside the certificate and a transparency-log entry (Rekor) that anyone can later audit. There is no long-lived key file to steal, back up, or rotate, at the cost of depending on the OIDC identity provider and Sigstore's infrastructure being reachable at sign time. Container image signing, discussed separately in this topic, makes the identical trade-off: a persisted key you must guard forever versus an ephemeral, OIDC-backed identity with nothing to steal.

Direct answer

Solve this by carving a single, never-touched supernet into fixed-size blocks up front, at two levels (one block per Region, then one block per VPC within that Region), and enforce that allocation through an IP address management (IPAM) tool rather than by convention, so no team can ever hand-pick a CIDR (Classless Inter-Domain Routing) range that happens to collide with another. Manual spreadsheet-based allocation is the thing that fails at this scale; the fix is automated, tool-enforced delegation.

Structured elaboration

CIDR allocation strategy. Start from one enterprise-wide supernet from the private address space, for example 10.0.0.0/8. Split it into per-Region blocks: a /8 divides cleanly into 16 blocks of /12, computed as $2^{(12-8)} = 16$, comfortably covering 10 Regions with 6 blocks held in reserve for future growth. Within each Region's /12, split further into per-VPC blocks: a /12 divides into 256 blocks of /20 (4,096 addresses each), computed as $2^{(20-12)} = 256$, which covers well over 200 VPCs per Region with enormous headroom, and a /20 is large enough for a realistic multi-AZ (Availability Zone), multi-tier subnet layout inside a single VPC without needing to resize it later (VPC CIDR blocks cannot be resized once created; only additional, non-overlapping secondary blocks can be added).

Delegation model or IPAM approach. A centralized IPAM service (AWS IPAM, or an equivalent third-party or homegrown tool) owns the top-level supernet and delegates each Region's /12 as an IPAM pool that only a specific account or automation role can draw from. Within a Region, a second-level pool per environment or business unit can further subdivide the /20 allocations if useful, but the discipline that matters most is that no human ever types a CIDR range by hand when creating a new VPC; the range comes from the pool automatically.

Automated guardrails against collisions. Wire VPC creation through infrastructure as code (a Terraform module or a CloudFormation/Service Catalog product) that calls the IPAM API to request the next available /20 from the correct Region's pool, rather than accepting an arbitrary CIDR input from whoever is provisioning the VPC. Back this with a policy check (an AWS Config rule, or a pre-merge check in the IaC pipeline) that rejects any VPC creation request specifying a CIDR outside the IPAM-managed range entirely, closing the loophole of someone bypassing the module and hand-typing a conflicting block.

Worked example

Region 1 (us-east-1, say) is assigned 10.0.0.0/12 from the pool of 16. Within it, the first three VPCs created via the IPAM-backed module receive 10.0.0.0/20, 10.0.16.0/20, and 10.0.32.0/20, in order, with the IPAM service tracking that these three are consumed and the remaining 253 blocks in that Region's pool are free. A team in a second Region (eu-west-1) independently provisions VPCs from their own 10.16.0.0/12 pool at the same time; because the two Regions draw from disjoint /12 blocks, there is no possibility of collision even though both teams are provisioning concurrently and neither is aware of the other's activity, which is precisely the property a manual spreadsheet cannot guarantee once more than a handful of teams are provisioning independently.

Trade-offs and pitfalls

Fixed-size blocks mean some allocations will be oversized for what a given VPC actually needs (a /20 for a tiny two-instance VPC wastes most of its 4,096 addresses), but the alternative, right-sizing each allocation individually, reintroduces exactly the manual bookkeeping and collision risk this design exists to eliminate; the wasted address space is a deliberate, acceptable cost given how much private IPv4 space a /8 actually provides. A second real trap: retrofitting this scheme onto an existing estate that already has ad hoc, overlapping CIDR ranges is much harder than designing it up front, because existing VPCs can't be renumbered without significant migration work; the guardrails described here are most valuable when established before the estate grows, not after.

Direct answer
Write a resource block of the correct type declaring only the arguments you intend to manage, run terraform import (or a declarative import block) to attach the real resource's ID to that address, then run terraform plan repeatedly, reconciling your HCL against what it reports, until plan shows no diff. Only then does apply touch anything, and by that point it's a no-op. Nothing about the live VPC or database changes during any of this, because import only writes state and plan never mutates anything, so the "without downtime" requirement is really just discipline about not applying until plan is clean.

Structured elaboration

1. Inspect the real resource before writing HCL
Don't write the resource block from memory. Look up the actual current attributes (via the console, CLI, or the provider's data source) so your first draft doesn't propose spurious changes the moment you import.

2. Write the minimal matching resource block
Keep it to the arguments you actually plan to manage going forward; leave out fields you expect the provider to compute (endpoints, generated ARNs) rather than guessing values for them.

resource "aws_db_instance" "example" {
  identifier = "my-db-identifier"
  # do not add attributes you expect AWS to compute (endpoint, address)
}

3. Import

terraform import aws_db_instance.example my-db-identifier

As of Terraform 1.5+ you can express the same step declaratively with an import block plus terraform plan -generate-config-out=generated.tf, which scaffolds a starting resource block for you instead of hand-writing one, useful when importing many resources at once.

4. Reconcile computed and drifted attributes
Run terraform plan. Anything present in the real resource but missing or different in your HCL shows up as a proposed change. For attributes the provider computes and you don't want Terraform fighting over (an RDS endpoint, a final snapshot identifier), either add the real value explicitly if you want to manage it, or mark it in a lifecycle block so Terraform stops treating drift there as something to fix.

lifecycle {
  ignore_changes = [endpoint, address]
}

5. Verify, then only apply if you mean to change something
terraform state show aws_db_instance.example confirms the mapped ID and attributes. Keep iterating on plan until it's clean. If you genuinely want to change a value going forward, that apply is a deliberate, reviewed change, not an accidental side effect of the import.

Nested/child resources aren't imported automatically
A hand-built VPC or database typically has dependent pieces, a DB subnet group, associated security groups, route tables, that each need their own import call. Importing the parent resource doesn't pull its children into management for you.

Worked example
Importing a hand-built RDS instance end to end:

terraform import aws_db_instance.example my-db-identifier
terraform state show aws_db_instance.example
terraform plan

A first plan on a genuinely hand-built instance will usually show a handful of proposed changes on computed fields like endpoint, address, and tags_all, since those were never in your HCL to begin with. You add ignore_changes for the ones you don't want to manage and explicit values for the ones you do, then re-run plan until it's a clean no-op, which is your confirmation that apply is now safe.

Trade-offs & pitfalls
Skipping the "inspect the real resource first" step is the most common way this goes wrong: you import, run plan, and discover it wants to replace the DB instance's engine or storage_type because you guessed a value instead of reading the real one, right after the whole point was to avoid touching it. Always back up state before starting (terraform state pull > backup.tfstate), and where possible rehearse the import against a non-production copy of the resource type first. Bulk-importing dozens of hand-built resources one CLI call at a time doesn't scale, that's what the 1.5+ import block with -generate-config-out is for.

Direct answer

A CI/CD (continuous integration/continuous delivery) pipeline for Terraform needs five stages in a strict order, because each stage's output is the next stage's required input: lint (fast, cheap syntax/style feedback), plan (compute what would actually change, without changing anything), publish the plan as an artifact (so the EXACT plan that gets reviewed is the EXACT plan that later gets applied, not a freshly recomputed one), manual approval (a human decision gate on that specific, already-computed plan), and apply (execute exactly the approved plan). The apply step's credentials need to be more tightly scoped and more carefully secured than every earlier stage's, since it is the only stage that actually touches live infrastructure.

Structured elaboration

Stage 1, lint. terraform fmt -check and terraform validate, plus any policy-as-code static checks that do not require live provider credentials (a check that a resource type is on an approved allowlist, for instance). Fast (seconds), runs on every push including drafts, gives the fastest possible feedback loop before anything more expensive runs.

Stage 2, plan. terraform plan -out=tfplan, using READ-ONLY credentials scoped to exactly what planning requires (Terraform needs to READ current state and query provider APIs to compute a diff, it does not need WRITE access at this stage). The binary plan file (tfplan) is the stage's real output, not just its console log.

Stage 3, publish the plan artifact. Upload tfplan (and, for reviewability, its human-readable rendering via terraform show tfplan) to the CI system's artifact store, tagged with the exact commit SHA it was computed against. This is the step that makes "what gets approved" and "what gets applied" provably the SAME plan, rather than the apply stage silently recomputing a fresh plan that could differ if anything changed in between (a provider version bump, a concurrent state change) between review and apply. Alongside the binary tfplan and its human-readable text rendering, also generate a machine-readable JSON rendering (terraform show -json tfplan > plan.json) and publish it too: this is the form policy-as-code tooling (Conftest, OPA, or a Sentinel-style policy check) actually consumes to programmatically evaluate the plan against a rule set ("no security group may open 0.0.0.0/0 on port 22", for instance) as an automated gate ahead of the human approval step, something neither the binary state-format plan file nor the human-readable text rendering supports directly, since both are meant for a human or Terraform itself to read, not for a policy engine to parse structurally.

Stage 4, manual approval. A human (or, for lower-risk environments, an automated policy check standing in for a human) reviews the PUBLISHED plan artifact specifically, not a freshly regenerated one, and approves or rejects. This is a genuinely separate gate from the PR's own code-review approval: code review judges the DIFF; this gate judges the ACTUAL PLANNED EFFECT, which can differ from what the diff alone suggests.

Stage 5, apply. terraform apply tfplan (applying the EXACT published artifact, not re-planning), using WRITE-scoped credentials, ideally short-lived and issued specifically for this run (OIDC federation to the cloud provider rather than a long-lived static secret stored in the CI system).

How apply is executed, CI runner vs. internal orchestration. A CI RUNNER executing apply directly is the simpler design, fewer moving parts, but means the CI system itself needs to hold (even briefly, even scoped) write credentials to production infrastructure. An INTERNAL ORCHESTRATION service (a dedicated apply-runner behind its own API, which CI calls rather than applying directly) adds a component but lets that dedicated service be the ONLY thing that ever holds write credentials, with CI only ever able to trigger it via a scoped, auditable API call, a meaningfully smaller credential-exposure surface for organizations at a scale where that trade-off is worth the added complexity.

Securing apply credentials. Short-lived (per-run, OIDC-federated) over long-lived static secrets wherever the cloud provider supports it; scoped to exactly the resources/actions this specific pipeline's applies legitimately touch, not a broad admin role reused across every pipeline; and never exposed in logs (the CI system's own credential-injection mechanism, not an explicit export in a script step that could accidentally get echoed).

Trade-offs and pitfalls

• Common mistake: re-running terraform plan at apply time instead of applying the exact published plan artifact. If ANYTHING changed between the approved plan and apply time (another merged change, a provider update), a freshly recomputed plan at apply time can differ from what was actually reviewed and approved, silently applying something nobody signed off on in that exact form; applying the saved .tfplan file specifically is what closes this gap.
• Read-only credentials at the plan stage are a real, not merely theoretical, blast-radius reduction. A compromised or misconfigured PLAN stage (an untrusted PR from a fork, for example) with only read access cannot modify infrastructure even if it tried; this matters specifically for organizations that run plan automatically on every PR, including from external or less-trusted contributors.
• The internal-orchestration-service design is genuinely higher operational overhead and is not the right default for a small team, it earns its complexity at a scale (many pipelines, many teams, high compliance bar) where minimizing the number of places holding write credentials materially reduces real risk; a small, single-team setup applying directly from CI runners is a reasonable, simpler choice and should not be treated as automatically insufficient.
• Audit records for approvals need to capture WHICH plan artifact (by content hash or artifact ID, not just "the PR's plan") was approved, a generic "approved" log entry with no link to the exact plan content is a weaker audit trail than one an external audit could actually trace end to end.

Direct answer

Size against the throughput a single GPU can sustain while still meeting your latency ceiling, not its theoretical peak throughput, because batching trades latency for throughput and both constraints have to hold at once. Divide the required load by that per-GPU number, round up, and add headroom for failover and variance.

Structured elaboration

• Batching groups concurrent requests so GPU compute is shared across them, raising throughput per GPU but adding queuing and shared-compute wait time to each request.
• Larger batch sizes keep raising throughput per GPU only up to a point; eventually you hit the p95 latency budget or the GPU's memory limit (the KV-cache, per-request memory the model keeps around during generation so it doesn't have to recompute earlier tokens on every step, grows with both batch size and sequence length).
• Fleet size = required QPS divided by per-GPU sustainable QPS at your latency ceiling, plus N+1 redundancy; steady-state size and autoscaling headroom are two different numbers.

Worked example

Assume profiling shows batch size 1 takes 80ms per request (p95 latency, the 95th-percentile response time), well under the 300ms budget:

$\text{unbatched QPS per GPU} = \frac{1000\text{ms}}{80\text{ms}} = 12.5$
$\text{unbatched fleet} = \lceil 200 / 12.5 \rceil = 16 \text{ GPUs}$

Now assume batch size 8 takes 140ms per batch (still under the 300ms budget):

$\text{batched QPS per GPU} = \frac{1000\text{ms}}{140\text{ms}} \times 8 \approx 57.1$
$\text{batched fleet} = \lceil 200 / 57.1 \rceil = 4 \text{ GPUs}$

Batching cuts the fleet from 16 to 4 GPUs, a 4x reduction, while per-request latency rises from 80ms to 140ms, still inside the 300ms p95 budget. Add one GPU for failover: 5 GPUs total versus 17 unbatched.

Trade-offs and pitfalls

Bigger batches help only until you hit a new latency ceiling or run out of GPU memory; past that you are compute-bound, not batching-bound, and no batching buys back headroom. Static batching forces every request to wait for the batch to fill; dynamic or continuous batching captures most of the savings without that wait.

What the interviewer probes next

Expect questions on how you'd handle the request that arrives just after a batch closes, how you'd autoscale for a traffic spike without over-provisioning steady state, and what happens to this math when sequence lengths vary widely.

Direct answer

BuildKit is Docker's modern build engine, the component that actually executes a Dockerfile's instructions and assembles image layers, replacing the older "classic" builder Docker shipped with for years. It became the default builder for docker build starting with Docker Engine 23.0 on Linux (Docker Desktop moved to it as the default around the same period), but understanding what it improves still matters, since plenty of environments run older engines or explicitly disable it.

At least four concrete improvements

1. Parallel, dependency-aware execution. The classic builder executes a Dockerfile strictly top to bottom. BuildKit builds a dependency graph of the instructions, executes independent stages of a multi-stage build concurrently, and skips stages nothing in the final image actually depends on.
2. Smarter build context handling. BuildKit transfers only the files a given instruction actually needs from the build context, instead of the classic builder's behavior of shipping the whole context to the daemon up front regardless of what gets used, which matters a lot on a large repository with a thin .dockerignore.
3. Cache mounts and secret mounts. --mount=type=cache persists a directory, such as a package manager's download cache, across builds without it ever becoming part of an image layer, and --mount=type=secret makes a value available to exactly one RUN instruction without ever writing it to a layer. Neither mechanism exists in the classic builder at all.
4. Remote, exportable cache. --cache-to and --cache-from against a registry let build cache be shared across completely different machines, rather than being trapped on whichever single machine happened to build it last. --cache-to=...,mode=min exports cache only for the layers that end up in the final image; mode=max exports cache for every intermediate layer too, including ones from an earlier build stage that never ships, which costs more to push and pull but lets a later build reuse work the final image itself doesn't contain.
5. Multi-platform builds from one invocation. BuildKit, through docker buildx, can build images for multiple CPU architectures, for example amd64 and arm64, in a single command and publish them under one multi-platform manifest, something the classic builder never supported natively.

How to enable it

On Docker Engine 23 and later, it is already the default: docker build transparently runs through Buildx and BuildKit. On an older engine, set the environment variable DOCKER_BUILDKIT=1 before invoking docker build locally, and export that same variable in the CI job's environment, or invoke docker buildx build explicitly, which always uses BuildKit regardless of the daemon's own default. The inverse, DOCKER_BUILDKIT=0, forces the legacy builder even on a modern engine, occasionally still needed to work around a BuildKit-specific edge case.

Worked example of a noticeable improvement

Take the cache-mount behavior on its own: RUN --mount=type=cache,target=/root/.cache/pip pip install -r requirements.txt means that even when requirements.txt genuinely changes, a real cache miss on that layer, not just a reordering fix, the underlying pip download cache from every prior build is still available inside that RUN. Only the packages that actually changed get re-downloaded, instead of the entire dependency set from zero. This was built and confirmed to work exactly as described: the mount attaches cleanly and the build completes normally. The classic builder has no equivalent mechanism; a layer cache miss there means a fully cold reinstall every time, with no way to keep the underlying download cache warm across builds.

Trade-offs and pitfalls

BuildKit's default build output is more compact and less verbose than the classic builder's line-by-line log, which trips people up in CI the first time; --progress=plain restores full, greppable output, worth knowing whenever you are diagnosing why a build step missed the cache. A few classic-builder behaviors changed subtly, some older flag combinations behave differently or are not supported the same way, so a Dockerfile and CI pipeline untouched since before Engine 23 is worth a deliberate verification pass rather than an assumption that it still works identically now that BuildKit is the default.

Direct answer: Structure the postmortem around a timeline, not a blame narrative: pull the deployment timestamp, overlay it against the SLI dashboard and the error-budget burn, quantify exactly how much budget the incident consumed, then work backward from the first observed symptom to root cause using traces and logs. The walk-away artifact is a concrete, owned action list, not a general "be more careful" resolution.

Structured elaboration

Data to pull together

• Timeline anchor: the deployment or config-change timestamp, this is almost always where the investigation starts, since most error-budget-burning incidents correlate tightly with a recent change.
• SLO dashboard: the SLI's value before, during, and after the incident, plus the error-budget burn-rate chart so the team can see not just "it broke" but "how fast it was burning."
• Errors by type: which endpoints, which status codes, which exception types spiked, this narrows root cause from "something broke" to "this specific code path broke."
• Traces and logs: a handful of representative failing traces (not all of them, a representative sample is enough to see the pattern) correlated with structured logs via trace ID.
• Resource and dependency metrics: CPU/memory/GC on the affected hosts, and latency/error rate on anything downstream, to distinguish a code bug from a resource exhaustion or dependency failure.

How to explain the impact

• Quantify budget consumed as a fraction of the period's total budget (see worked example), not just "the SLO dropped." A number stakeholders can compare to "how much budget is left this month" is far more actionable than a description of the dashboard dip.
• State the user-facing impact in concrete terms: percentage of requests affected, estimated number of unique users or sessions, and duration of impact.

What you walk away with

• A root cause stated at the mechanism level ("the new code path opened a DB connection per request without pooling, exhausting the connection pool under peak load"), not at the symptom level ("errors increased").
• A small number of owned, dated action items, tightening canary criteria, adding a specific alert that would have caught this faster, a pre-deploy check that would have caught the bug in the first place. Each with a named owner and a due date, reviewed at a fixed checkpoint (commonly one week out) to confirm they actually landed.
• A decision on rollback vs patch-forward for this specific incident, documented with the reasoning, so the next similar incident has a precedent to reference instead of re-litigating the decision from scratch.

Worked example

Assume the service has a monthly correctness error budget of 2,000 failed requests (at 99.9% SLO over 2,000,000 monthly requests, following the same derivation as the SLI/SLO worked example above). A bad deploy is live for 45 minutes before rollback, during which the error rate on this service jumped from a normal 0.05% to 8%, with average traffic of 400 requests/minute during that window.

Failed requests during the incident:

Baseline failures that would have happened anyway in that window (at the normal 0.05% rate) don't count against the incident's attributable impact:

Attributable incident impact:

As a fraction of the monthly budget:

That single number, 72% of the month's budget gone in under an hour, is what belongs at the top of the postmortem's impact section: it makes the severity legible to anyone reading the doc, engineer or not, without them needing to interpret a dashboard.

Trade-offs & pitfalls

• A postmortem that stops at "we rolled back and errors went away" without root-causing the mechanism leaves the door open for the same bug to resurface in a different code path. Push to the mechanism, not just the symptom.
• Action items with no owner or due date are the single most common reason postmortem follow-through fails; the one-week checkpoint above exists specifically to catch that before it becomes a pattern.
• The same structure applies when the trigger is a sustained SLO violation accumulating over several days rather than one bad deploy, the difference is that the timeline anchor isn't a single deploy event, it's a slower-moving cause (a capacity ceiling being approached, a dependency degrading gradually), so the investigation should explicitly check "did this correlate with a change, or was there no single change and the system just outgrew its current capacity."

I push back by anchoring on the business outcome, not by saying no reflexively.

How I handle it:

1. I first clarify what problem the stakeholder is trying to solve.
2. I compare the request against the current roadmap and explain the trade-off in plain language.
3. I show the impact on timing, quality, or other committed work if we take it now.
4. I offer options: replace something else, phase it into a later release, or test it in a smaller pilot.

Example phrasing:
“Your request is valid, but if we add it this sprint, we’ll delay the launch item we already committed to. We can either swap scope, defer this to the next cycle, or find a thinner version that gets you part of the value sooner.”

How I preserve the relationship:
I stay consistent, transparent, and respectful. I acknowledge the stakeholder’s urgency, follow up with written decisions, and keep them updated so they feel heard even when the answer is no. That usually builds trust, because they see I’m protecting the broader business, not just the team’s convenience.

Direct answer

For a 4-person team with limited operations experience hosting an application around 10,000 daily active users, I would recommend PaaS as the primary model, a managed application platform plus a managed database, reserving IaaS only for any piece that genuinely needs it, and treating SaaS as a source of building blocks to buy, such as authentication or transactional email, rather than as the hosting model for the core app itself.

Comparing the three for a small team

IaaS gives full control and potentially the cheapest per-unit compute, but the team would own OS patching, scaling configuration, and monitoring, which competes directly with the very small number of engineering hours available for the actual product. Wrong default here.

PaaS gives up some control, but a 4-person team gets a production-grade deployment, managed scaling, managed database, automated patching, without hiring or becoming operations specialists. That is exactly the trade a resource-constrained team should make.

SaaS is the right model for well-defined, already-solved problems adjacent to the product, using a SaaS product for email delivery, a SaaS-provided authentication service, or a support-ticketing tool for customer service, rather than for the core application, since the core app is presumably the team's actual differentiated product and isn't something an off-the-shelf SaaS product would be.

The recommendation

A managed application platform for the app tier plus a managed database for storage, with a couple of SaaS building blocks glued in for auth and transactional email rather than built from scratch. This keeps the team's operations surface down to "watch a dashboard and occasionally adjust a scaling policy" instead of "own a fleet."

A hybrid approach, and when it makes sense

A hybrid here would mean keeping the core app on a managed platform while running one specific piece on IaaS, for example a background job that needs a software dependency the managed platform doesn't support, or a steady, cost-sensitive batch workload where a small reserved virtual machine is genuinely cheaper at this scale than the managed equivalent. This makes sense once the team has one clear, isolated piece of the system with a requirement PaaS can't satisfy, or a cost delta large enough to justify the added operational surface for that one piece, not as a blanket decision to control more of the stack.

The analytics variant: a different lock-in calculus

If the workload were analytics or business intelligence (BI) flavored instead of a general customer-facing app, for instance this same 4-person team standing up internal reporting dashboards, the PaaS-first instinct needs an extra check. A fully SaaS BI tool gets a working dashboard fastest, but it usually couples the team's reporting logic and data tightly to that vendor's proprietary query language and connectors, a much larger vendor lock-in exposure than picking a managed application platform for a general web app, where the exit path (redeploy the same container elsewhere) is comparatively cheap. A PaaS-level option, a managed data warehouse queried with standard SQL, costs the team a bit more setup and ongoing responsibility, someone has to model the data and maintain the queries, but keeps the exit path open, since standard SQL and exported data are portable in a way a BI SaaS product's proprietary dashboard definitions typically are not. For a 4-person team specifically, that's worth naming explicitly rather than defaulting to "SaaS is always right for a small team," because the team's limited operations capacity has to be weighed against how expensive it would be for that same small team to unwind a deep SaaS BI dependency two years later if the vendor's pricing or roadmap stops fitting.

Trade-offs and pitfalls

The first pitfall is treating "small team, limited ops" as an automatic vote for SaaS everywhere; SaaS for your core differentiated product usually means you're not actually building a product anymore, just configuring someone else's. The second is over-indexing on cost-per-unit-compute when comparing IaaS to PaaS at this scale: at 10,000 daily active users, the labor cost of even a fraction of one engineer's time spent on operations very likely exceeds the sticker-price premium of PaaS, so the "PaaS costs more per server" argument that matters at large scale mostly doesn't apply yet.