Containerization and Docker Fundamentals Questions

Direct answer

The core principle is build once, promote everywhere: an artifact is built and given an immutable identity exactly once, and every later environment, staging and then production, references that same identity rather than triggering a second build from source. Drift, and the class of bug where something works in staging but not in production, overwhelmingly comes from an environment quietly building its own slightly different copy of "the same" release instead of promoting the one that was already tested.

The design

Build once. CI builds the image a single time per release candidate, tags it immutably and uniquely (a commit hash or a semantic release version), and records its digest (a digest is a cryptographic hash of the image's exact bytes; two images with the same digest are guaranteed to be byte-for-byte identical, unlike a tag, which can later be moved to point at different bytes) at that moment. No later pipeline stage is allowed to rebuild this artifact; staging and production only ever pull or reference it.

Promotion is a metadata operation, not a build. Promoting an artifact from staging to production re-tags the same digest for the next environment, for example adding app:v1.4.2-prod-approved alongside the existing app:v1.4.2, or a promotion tool simply records that this digest is now the production candidate. In a digest-pinned deploy configuration, promotion can be as simple as updating production's deploy manifest to reference the already-tested digest directly.

Registry permissions enforce the guardrail. CI's build identity gets write access to push new images and tags. Production's deploy identity gets read-only access, able to pull, never to push. If the registry supports tag immutability or protected tags, lock release tags so nothing, accidental or malicious, can silently repoint a tag after staging has already tested it. A human or an approval step gets permission to move a "promoted" pointer or write a deployment record, never to trigger a rebuild.

Preventing accidental rebuilds. The biggest practical source of drift is a Dockerfile or CI configuration that is not fully deterministic: a base image referenced by a floating tag like python:3.12-slim instead of a pinned digest, or an install step with no version pins, meaning two builds from "the same" source a day apart can legitimately produce different bytes. Pin base images by digest, pin dependency versions, and treat anything environment-specific, a database URL, a feature flag, as runtime configuration injected into the same image, never as a build-time difference that forces separate, environment-specific images.

flowchart LR
    Commit["source commit"] --> Build["CI build (once)"]
    Build --> Registry["registry: immutable digest"]
    Registry --> Staging["staging deploy (pull-only)"]
    Staging -->|"tests pass, promote"| Prod["production deploy (pull-only, same digest)"]

Worked example

A release pipeline for a checkout service: CI builds on merge to the main branch, tags checkout-service:2026.09.23-abc1234, and records the digest in a release manifest. Staging's deploy configuration is templated to always deploy checkout-service@<digest from the latest release manifest>. After staging's smoke tests pass, a promotion job, not a rebuild, copies that exact digest reference into the production deploy configuration and opens the standard change-approval process. Production's deploy identity has pull-only registry permissions, so even a compromised production deploy credential cannot push a different image under that same tag.

Trade-offs and pitfalls

Locking down write access this tightly adds real friction for a legitimate emergency change: a hotfix now has to go through the same build-once pipeline rather than a quick local build pushed straight to production, a deliberate trade of speed for guaranteed provenance. "Build once" is easy to violate by accident the moment any Dockerfile step reaches for something mutable, an unpinned base image tag, an unpinned package install, or a dependency resolved with no lockfile, so this whole design only holds if the Dockerfile itself is actually reproducible; otherwise the promotion discipline is being built on top of a build that was never deterministic to begin with.

Direct answer

Build the static binary in a golang builder stage with CGO_ENABLED=0 (which disables cgo, Go's mechanism for calling C code, and forces a fully self-contained, statically linked binary with no dynamic C library dependency), then copy just that binary into a scratch (an entirely empty base image) or distroless (a minimal image with no shell, package manager, or other OS tooling beyond what the language runtime needs) final stage, running as a non-root user. CGO_ENABLED=0 is what makes scratch or distroless usable at all: with cgo enabled, the binary dynamically links against the host's C library, and a scratch image has no C library for it to find at runtime. Build metadata (version, build time) gets embedded at compile time via -ldflags -X, which sets the value of specific Go package-level string variables directly in the compiled binary, without any config file or environment variable needed at runtime.

Structured elaboration

The Dockerfile

# syntax=docker/dockerfile:1
FROM golang:1.23-alpine AS builder
WORKDIR /src
COPY go.mod ./
COPY main.go main_test.go ./
ARG VERSION=dev
ARG BUILD_TIME=unknown
RUN CGO_ENABLED=0 GOOS=linux go build \
      -ldflags "-s -w -X main.version=${VERSION} -X main.buildTime=${BUILD_TIME}" \
      -o /out/svc .

# A discrete test stage: CI runs `docker build --target test .` as a gate
# before it ever builds the `final` target below. It never appears in the
# final image's own dependency chain, so a plain `docker build .` (which
# resolves to the last stage, `final`) does not pay for the Go toolchain
# or re-run tests as part of shipping the artifact.
FROM builder AS test
RUN CGO_ENABLED=0 go test ./... -v

FROM gcr.io/distroless/static-debian12:nonroot AS final
COPY --from=builder /out/svc /svc
USER nonroot:nonroot
EXPOSE 8080
HEALTHCHECK --interval=10s --timeout=3s --start-period=5s --retries=3 \
  CMD ["/svc", "-healthcheck"]
ENTRYPOINT ["/svc"]

-ldflags "-s -w -X main.version=${VERSION} -X main.buildTime=${BUILD_TIME}" does two things: -s -w strip debug symbols to shrink the binary, and -X main.version=... -X main.buildTime=... overwrite the value of the named package-level string variables (declared in the Go source as var version = "dev") at link time, which is how the running binary can report its own version and build time without reading any file.

Why CGO_ENABLED decides your final base image

• CGO_ENABLED=0: the Go toolchain avoids cgo entirely and produces a statically linked binary with no external C library dependency at all. That binary runs correctly on scratch (nothing else in the image) or a distroless static base, because it needs nothing from the base image except its own bytes.
• CGO_ENABLED=1 (the default when cgo is used, for example by some database drivers or net package DNS resolution modes): the binary dynamically links against the host's C library at build time and needs a compatible one present at run time. Shipping that binary on scratch fails outright (no C library exists there to load); it needs at minimum a distroless base variant that includes glibc, or a small distribution image, not the fully empty scratch image or the static distroless variant.

A discrete test stage, and why it is separate from final
Defining FROM builder AS test as its own stage lets CI run docker build --target test . as an explicit quality gate: if go test fails, that build step fails, and the pipeline stops before ever producing a shippable artifact. Because test is not in final's FROM chain, a plain build that targets final (the default target, being the last stage) never executes the tests as part of producing the production image; the test run is a separate, addressable step in CI, not baked into every build.

Non-root and why it matters here specifically
scratch and distroless images have no /etc/passwd, so USER nonroot:nonroot on the gcr.io/distroless/static-debian12:nonroot base relies on that image already defining a numeric non-root user (uid 65532, conventionally). Running as non-root means that even if the application were compromised through some other flaw, the compromised process would not hold root privileges inside its own container, narrowing what an attacker who gained code execution could do next (for example, it cannot bind privileged ports below 1024 or, on a properly configured host, escalate as easily through kernel-level capabilities that root retains by default).

A HEALTHCHECK with no shell available
Distroless and scratch images have no shell and no curl or wget to run as a health-check command. The pattern that still works is to have the binary itself perform the check: a small -healthcheck flag on the same binary that makes an HTTP request to its own /healthz endpoint and exits 0 or 1 accordingly, invoked directly (CMD ["/svc", "-healthcheck"], exec form, no shell involved).

Why multi-stage matters here beyond image size
Multi-stage builds improve security by keeping the Go compiler, source code, and build-time tooling entirely out of the shipped image (there is nothing in a distroless final stage for an attacker to use to compile or run arbitrary code, and no build-time secrets or cache files can leak into it if the COPY --from=builder only ever names the compiled binary). They improve deployability because the resulting image is a single small, self-contained binary with no OS package manager or shell to patch, version, or scan for unrelated vulnerabilities, and it starts as fast as the binary itself starts, with no runtime dependency resolution.

Worked example

Building the Dockerfile above with --build-arg VERSION=1.3.0 --build-arg BUILD_TIME=2026-09-24T00:12:54Z produces a final-stage image measuring 14.6 MB. Running docker build --target test . executes go test ./... -v, which prints --- PASS: TestVersionString (0.00s) and exits 0; a broken versionString() implementation would instead fail this build step, before final is ever built. Running the resulting container and calling its /version endpoint returns version=1.3.0 build_time=2026-09-24T00:12:54Z, exactly the values passed as build arguments, confirming the -ldflags -X substitution took effect. docker inspect on the running container shows Health.Status: healthy, produced by the binary's own -healthcheck flag rather than any shell command, since the final image has no shell to run one.

Trade-offs & pitfalls

• Leaving CGO_ENABLED at its default and only discovering the dynamic-linking requirement when a scratch-based image fails at container start (a missing shared library error) is a common and entirely avoidable mistake; decide the flag deliberately based on whether anything in the dependency tree actually needs cgo.
• A fully empty scratch final stage has no CA certificate bundle either, which breaks outbound TLS connections silently; either copy /etc/ssl/certs/ca-certificates.crt from the builder stage or use a distroless base (not static) variant that already includes it, depending on whether the service needs to make outbound HTTPS calls.
• Debugging a distroless container in production is harder by design (no shell to docker exec into); that is the intended trade-off for a smaller attack surface, and it means logging and the -healthcheck-style self-diagnostic pattern have to carry more of the debugging burden than they would in a full-featured base image.

Direct answer

Standardizing container practices across a team with inconsistent base images and ad hoc language versions works best as a small set of shared building blocks people opt into because they are genuinely easier, not a mandate enforced all at once. Define one or two approved base images per language, publish them somewhere central, migrate the highest-friction or most-broken services first as visible proof it works, and measure success by concrete, checkable signals (fewer "works on my machine" incidents, a shrinking count of services still on the old pattern), not by a one-time announcement that a policy now exists.

Structured elaboration

Defining the strategy

• Start from what is actually causing pain today, not from an abstract ideal: pull specific recent incidents (an image that broke because of an unpinned base image update, a new hire who lost half a day to a subtly wrong local toolchain version) and use them as the concrete justification for standardizing, rather than "best practice" as an argument on its own.
• Pick a small number of golden, versioned base images per language (one for Python, one for Node, and so on), each pinned to a specific, deliberately chosen version, published to an internal registry, and owned by a specific person or small group responsible for updating them on a known cadence.
• Write down the standard as a short, concrete checklist (which base image, how dependencies get pinned, what the multi-stage pattern should look like) rather than a long policy document; a checklist gets followed, a policy document gets skimmed once and forgotten.

Rolling it out without blocking delivery

• Never require every existing service to migrate before the standard is considered "live." Make the new base images available immediately, require them only for new services going forward, and migrate existing ones opportunistically, attached to work already happening on them (a routine dependency bump, a feature that touches the Dockerfile anyway), rather than as a dedicated, disruptive migration project competing with everything else the team is shipping.
• Pick one or two of the most painful existing services to migrate first, deliberately, as an early, visible example, ideally ones whose current state already causes recognizable pain, so the migration itself demonstrably fixes something people already wanted fixed, not just something that satisfies a new policy.
• Provide a clear, low-friction path for the exception case: a service with a genuine reason it cannot use the standard base image yet should have an explicit, time-bound way to say so, rather than silently ignoring the standard or being blocked outright; a standardization effort that has no accommodation for real exceptions tends to get quietly worked around instead of followed.

Measuring whether it is actually improving reproducibility

• Track a concrete, falling count: the number of services still on a non-standard base image or an unpinned dependency set, visible on a dashboard everyone can see, not just reported occasionally in a meeting.
• Track "works on my machine" incidents specifically (however the team already reports bugs or friction) before and after, since this is the exact symptom the standardization is meant to fix; a real reduction here is a much stronger signal than adoption count alone.
• Track new-hire or new-service onboarding time (how long it takes someone unfamiliar with a given service to get it running locally), since a genuinely standardized set of practices should make this measurably faster and more consistent across services, not just theoretically cleaner.

Worked example

A team of six services has four different Python base images in active use, two of which are no longer receiving security updates upstream, and a recurring pattern of new hires losing time to subtly different local dependency versions. The first move is not a team-wide announcement; it is publishing one pinned, actively maintained internal Python base image and migrating the single service that has caused the two most recent "works on my machine" incidents, as a visible fix to a problem people already recognized. Over the following quarter, three more services adopt the new base image as part of routine work already scheduled on them, and new services are required to start from it from day one. A dashboard tracking "services on the old pattern" drops from five to one over that period, and reported onboarding friction for a new service drops in the team's own retrospective notes, which together are the actual evidence the standardization worked, rather than the initial announcement alone.

Trade-offs & pitfalls

• Mandating an immediate, blanket migration of every existing service tends to either stall (competing with every other priority the team has) or get rushed and done poorly; opportunistic migration attached to work already happening is slower in wall-clock time but far more likely to actually finish.
• Measuring success only by "did we publish the standard" or "how many services adopted it" misses whether it is actually solving the underlying problem; track the specific pain (onboarding friction, "works on my machine" incidents) the standardization was meant to fix, not just adoption as an end in itself.
• No accommodation for genuine exceptions turns a well-intentioned standard into something people quietly route around, which produces worse long-term consistency than a standard with an explicit, visible exception process would have.

Direct answer

For a typical Spring Boot service, I would default to Spring Boot's own layered-jar (Java ARchive, the packaged format a Java application ships as) extraction pattern inside a multi-stage Dockerfile, because it directly targets the two things the question asks to optimize, build cache reuse and startup time, with the least operational complexity. I would reach for a GraalVM native image only once startup latency and baseline memory become a hard requirement the JVM (Java Virtual Machine) cannot meet, since native image trades a much heavier build for a much lighter runtime, and reach for Cloud Native Buildpacks mainly when I want to avoid maintaining a Dockerfile at all rather than for a technical advantage over layered jars.

The three approaches

• Spring Boot layered jars. A Spring Boot fat jar (one file containing the application classes and all its dependencies) is not itself cache-friendly to COPY as a single unit, since any code change invalidates the whole layer. Spring Boot's own extraction tooling splits it into separate directories, dependencies (which change rarely), the Spring Boot loader itself (changes only on a Spring Boot version bump), snapshot dependencies, and the application's own classes (which change on every commit), so that a Docker layer built from the rarely-changing directories is reused across builds, and only the layer built from the application's own classes needs to be re-pulled after a typical code change.
• jlink, a JDK (Java Development Kit) tool that builds a custom, minimal Java runtime containing only the modules an application actually uses, instead of shipping a full general-purpose JRE (Java Runtime Environment) base image. This can meaningfully shrink the runtime image's baseline size and attack surface, but Spring Boot's heavy use of reflection and dynamic class loading (core to how its dependency injection and auto-configuration work) makes jlink's module-dependency analysis (via jdeps) less reliable than for a simple, mostly-static application: jdeps can miss modules only reached through reflection, and the fix is to either whitelist extra modules manually or accept a runtime failure only surfaces at whatever code path exercises the missed reflection, not at build time.
• Cloud Native Buildpacks (a standard, defined by the Cloud Native Buildpacks project, for turning source code directly into a runnable OCI (Open Container Initiative, the industry standard defining the container image and runtime format that Docker and other tools implement) image without writing a Dockerfile), invoked from Spring Boot via ./mvnw spring-boot:build-image or ./gradlew bootBuildImage. This produces a layered image using largely the same layering principle as the manual approach above, but the build logic, base image selection, and layer boundaries are owned by the buildpack rather than a Dockerfile you maintain yourself, trading control and transparency for less Dockerfile maintenance burden across many services that all want the same pattern.

Worked example: how dependencies get cached under the layered-jar approach

This is Spring Boot's own current documented pattern (verified directly against the live Spring Boot reference documentation), using the tools jarmode (the current name; older Spring Boot versions and older tutorials use the now-superseded layertools jarmode, worth recognizing if you encounter it in existing pipelines):

FROM bellsoft/liberica-openjre-debian:25-cds AS builder
WORKDIR /builder
ARG JAR_FILE=target/*.jar
COPY ${JAR_FILE} application.jar
RUN java -Djarmode=tools -jar application.jar extract --layers --destination extracted

FROM bellsoft/liberica-openjre-debian:25-cds
WORKDIR /application
COPY --from=builder /builder/extracted/dependencies/ ./
COPY --from=builder /builder/extracted/spring-boot-loader/ ./
COPY --from=builder /builder/extracted/snapshot-dependencies/ ./
COPY --from=builder /builder/extracted/application/ ./
ENTRYPOINT ["java", "-jar", "application.jar"]

Each COPY --from=builder line becomes its own Docker layer. On a typical code change (only the application directory's contents differ from the previous build), Docker reuses the cached dependencies, spring-boot-loader, and snapshot-dependencies layers unchanged, and only rebuilds and re-pushes the small application layer, rather than the whole fat jar as one monolithic unit. The base image here also matters for startup time specifically: bellsoft/liberica-openjre-debian:25-cds ships with CDS (Class Data Sharing, a JVM feature that pre-parses and caches class metadata to speed up JVM startup) enabled, which Spring Boot's current layered layout is explicitly designed to be compatible with.

Trade-offs against GraalVM native image

A GraalVM native image ahead-of-time (AOT) compiles the application into a standalone native executable with no JVM startup cost at all, giving near-instant startup and substantially lower baseline memory, which matters for workloads that scale to zero or spin up many short-lived instances. The cost is a much heavier and slower build (AOT compilation analyzes the whole reachable call graph, which takes meaningfully longer and more build memory than packaging a jar), and Spring Boot's reflection-heavy dynamic behavior needs explicit AOT processing support (Spring Boot's own native-image integration, introduced to generate the reachability metadata GraalVM needs) to work at all; a library or a piece of application code that does something dynamic GraalVM's static analysis cannot see can fail only at native-image build time or, worse, only at runtime if the reachability metadata is incomplete.

Trade-offs and pitfalls

The layered-jar approach is a strict improvement over shipping the fat jar as one COPY for both stated goals (cache reuse and startup, via CDS-friendly layout) with essentially no downside, which is why I would default to it. jlink's benefit is real but harder to fully realize on a framework as reflection-heavy as Spring Boot without careful validation of what jdeps actually captured. GraalVM native image is the right call only once the JVM's startup and memory profile is an actual, measured problem for the deployment shape (very short-lived instances, extremely tight memory budgets), since committing to it means committing to a slower, more fragile build pipeline and Spring's native-specific AOT support surface, not just flipping a build flag.

Direct answer

I design retention as tiers keyed to what a tag actually represents, not a single blanket rule: release and production-facing tags are protected and effectively kept forever, branch and experiment tags decay automatically on a short clock, and garbage collection runs as a scheduled, low-traffic maintenance job rather than an ad hoc cleanup, so nobody has to manually decide what is safe to delete.

The design

Garbage collection (GC, the process of reclaiming storage for image layers no longer referenced by any remaining tag or manifest) is scheduled during low-traffic hours rather than triggered manually, because a naive GC run that walks the whole blob store while pushes are actively happening can race with an in-flight push and, on registries that require it, needs the registry put into a read-only or maintenance mode for the duration. This differs sharply by registry implementation and matters when choosing one: a self-hosted registry (the open-source Docker Registry, or Harbor, a registry that adds retention policies and tag immutability rules on top of it) needs an explicit, scheduled GC job and, on the open-source registry specifically, historically needs the registry to be read-only while GC runs to avoid corrupting in-flight uploads. A managed registry like Amazon ECR (Elastic Container Registry, AWS's managed container registry) has no manual GC step at all: you configure lifecycle policies (rules like "expire untagged images after 14 days" or "keep only the last 10 images matching feature/*") and the service reclaims storage automatically and continuously.

Worked example

For a registry that has accumulated hundreds of tags from months of unmanaged experimentation, the rollout is staged rather than a one-shot deletion: first, tag every existing image against the classification above (a one-time audit script that reads push metadata and branch-name patterns), then apply the new lifecycle rules going FORWARD only (new pushes obey the tiers immediately), then run a single manual cleanup pass on the pre-existing backlog using the same rules with a dry-run flag first, so the retention design is validated against real data before anything is actually deleted. Concretely, an ECR lifecycle policy rule expressing the feature-branch tier looks like: match tag prefix feature/, keep the most recently pushed 5 images matching that prefix, expire the rest. A second rule expresses the experiment tier: match tag prefix exp/, expire any image pushed more than 7 days ago, with no count-based exception.

Trade-offs and pitfalls

The main cost-versus-convenience trade-off is the idle-window length on feature and experiment tags: a shorter window saves more storage but increases the chance someone needs an image that already expired (usually recoverable by rebuilding from source, since the tag was never a durable artifact, but still a real interruption). The most common design mistake is applying one retention rule uniformly across all tag classes, which either keeps release tags too loosely (a rollback target could theoretically be garbage-collected) or expires experimental tags too slowly (storage cost creeps up for images nobody was ever going to reuse). Tag immutability on release tags specifically closes a separate but related risk: without it, someone can accidentally push new content to an existing release tag, which silently breaks reproducibility for anyone who redeploys that "same" version later, independent of whatever the retention window is set to.