Containerization and Docker Fundamentals Questions

Direct answer

Build the static binary in a golang builder stage with CGO_ENABLED=0 (which disables cgo, Go's mechanism for calling C code, and forces a fully self-contained, statically linked binary with no dynamic C library dependency), then copy just that binary into a scratch (an entirely empty base image) or distroless (a minimal image with no shell, package manager, or other OS tooling beyond what the language runtime needs) final stage, running as a non-root user. CGO_ENABLED=0 is what makes scratch or distroless usable at all: with cgo enabled, the binary dynamically links against the host's C library, and a scratch image has no C library for it to find at runtime. Build metadata (version, build time) gets embedded at compile time via -ldflags -X, which sets the value of specific Go package-level string variables directly in the compiled binary, without any config file or environment variable needed at runtime.

Structured elaboration

The Dockerfile

# syntax=docker/dockerfile:1
FROM golang:1.23-alpine AS builder
WORKDIR /src
COPY go.mod ./
COPY main.go main_test.go ./
ARG VERSION=dev
ARG BUILD_TIME=unknown
RUN CGO_ENABLED=0 GOOS=linux go build \
      -ldflags "-s -w -X main.version=${VERSION} -X main.buildTime=${BUILD_TIME}" \
      -o /out/svc .

# A discrete test stage: CI runs `docker build --target test .` as a gate
# before it ever builds the `final` target below. It never appears in the
# final image's own dependency chain, so a plain `docker build .` (which
# resolves to the last stage, `final`) does not pay for the Go toolchain
# or re-run tests as part of shipping the artifact.
FROM builder AS test
RUN CGO_ENABLED=0 go test ./... -v

FROM gcr.io/distroless/static-debian12:nonroot AS final
COPY --from=builder /out/svc /svc
USER nonroot:nonroot
EXPOSE 8080
HEALTHCHECK --interval=10s --timeout=3s --start-period=5s --retries=3 \
  CMD ["/svc", "-healthcheck"]
ENTRYPOINT ["/svc"]

-ldflags "-s -w -X main.version=${VERSION} -X main.buildTime=${BUILD_TIME}" does two things: -s -w strip debug symbols to shrink the binary, and -X main.version=... -X main.buildTime=... overwrite the value of the named package-level string variables (declared in the Go source as var version = "dev") at link time, which is how the running binary can report its own version and build time without reading any file.

Why CGO_ENABLED decides your final base image

• CGO_ENABLED=0: the Go toolchain avoids cgo entirely and produces a statically linked binary with no external C library dependency at all. That binary runs correctly on scratch (nothing else in the image) or a distroless static base, because it needs nothing from the base image except its own bytes.
• CGO_ENABLED=1 (the default when cgo is used, for example by some database drivers or net package DNS resolution modes): the binary dynamically links against the host's C library at build time and needs a compatible one present at run time. Shipping that binary on scratch fails outright (no C library exists there to load); it needs at minimum a distroless base variant that includes glibc, or a small distribution image, not the fully empty scratch image or the static distroless variant.

A discrete test stage, and why it is separate from final
Defining FROM builder AS test as its own stage lets CI run docker build --target test . as an explicit quality gate: if go test fails, that build step fails, and the pipeline stops before ever producing a shippable artifact. Because test is not in final's FROM chain, a plain build that targets final (the default target, being the last stage) never executes the tests as part of producing the production image; the test run is a separate, addressable step in CI, not baked into every build.

Non-root and why it matters here specifically
scratch and distroless images have no /etc/passwd, so USER nonroot:nonroot on the gcr.io/distroless/static-debian12:nonroot base relies on that image already defining a numeric non-root user (uid 65532, conventionally). Running as non-root means that even if the application were compromised through some other flaw, the compromised process would not hold root privileges inside its own container, narrowing what an attacker who gained code execution could do next (for example, it cannot bind privileged ports below 1024 or, on a properly configured host, escalate as easily through kernel-level capabilities that root retains by default).

A HEALTHCHECK with no shell available
Distroless and scratch images have no shell and no curl or wget to run as a health-check command. The pattern that still works is to have the binary itself perform the check: a small -healthcheck flag on the same binary that makes an HTTP request to its own /healthz endpoint and exits 0 or 1 accordingly, invoked directly (CMD ["/svc", "-healthcheck"], exec form, no shell involved).

Why multi-stage matters here beyond image size
Multi-stage builds improve security by keeping the Go compiler, source code, and build-time tooling entirely out of the shipped image (there is nothing in a distroless final stage for an attacker to use to compile or run arbitrary code, and no build-time secrets or cache files can leak into it if the COPY --from=builder only ever names the compiled binary). They improve deployability because the resulting image is a single small, self-contained binary with no OS package manager or shell to patch, version, or scan for unrelated vulnerabilities, and it starts as fast as the binary itself starts, with no runtime dependency resolution.

Worked example

Building the Dockerfile above with --build-arg VERSION=1.3.0 --build-arg BUILD_TIME=2026-09-24T00:12:54Z produces a final-stage image measuring 14.6 MB. Running docker build --target test . executes go test ./... -v, which prints --- PASS: TestVersionString (0.00s) and exits 0; a broken versionString() implementation would instead fail this build step, before final is ever built. Running the resulting container and calling its /version endpoint returns version=1.3.0 build_time=2026-09-24T00:12:54Z, exactly the values passed as build arguments, confirming the -ldflags -X substitution took effect. docker inspect on the running container shows Health.Status: healthy, produced by the binary's own -healthcheck flag rather than any shell command, since the final image has no shell to run one.

Trade-offs & pitfalls

• Leaving CGO_ENABLED at its default and only discovering the dynamic-linking requirement when a scratch-based image fails at container start (a missing shared library error) is a common and entirely avoidable mistake; decide the flag deliberately based on whether anything in the dependency tree actually needs cgo.
• A fully empty scratch final stage has no CA certificate bundle either, which breaks outbound TLS connections silently; either copy /etc/ssl/certs/ca-certificates.crt from the builder stage or use a distroless base (not static) variant that already includes it, depending on whether the service needs to make outbound HTTPS calls.
• Debugging a distroless container in production is harder by design (no shell to docker exec into); that is the intended trade-off for a smaller attack surface, and it means logging and the -healthcheck-style self-diagnostic pattern have to carry more of the debugging burden than they would in a full-featured base image.

Direct answer

I treat intermittent package-download failures as a network triage problem before a build problem: first isolate whether the failure sits on the CI (continuous integration, the automated system that builds and tests every change) runner's network path or on the far side at the registry, then add resilience (retries, mirrors, caching) so a transient blip does not fail the whole pipeline, and only after that look at whether the underlying rate or capacity limit needs a structural fix.

Systematic troubleshooting steps

1. Capture the exact failure, not just "it failed." A timeout (the client gave up waiting) and a 502 (the server or an intermediate proxy returned a bad gateway response) point in different directions: a timeout suggests a slow or congested path, a 502 suggests the registry or a caching proxy in front of it is unhealthy or overloaded, not the runner's network.
2. Rule out DNS on the runner itself. Docker builds can resolve names using a different resolver configuration than the host (the daemon's --dns setting, or a resolv.conf baked into the build container), so test resolution from inside the same execution context the build uses, not just from the host shell: docker run --rm alpine nslookup registry.npmjs.org (or the relevant registry hostname) using the same network mode the CI job uses.
3. Rule out a proxy misconfiguration. If the runner sits behind an HTTP/HTTPS proxy, confirm the proxy environment variables are actually propagated into the build (--build-arg HTTP_PROXY=... --build-arg HTTPS_PROXY=... --build-arg NO_PROXY=... for BuildKit (Docker's build engine, the default backend behind docker build), since build-time environment is not automatically inherited from the host shell), and check whether internal package mirrors were accidentally routed through the external proxy via a missing NO_PROXY entry.
4. Check for shared-egress rate limiting. Self-hosted runners commonly share one NAT (Network Address Translation, the mechanism that lets many internal machines share one public IP address) gateway IP. If several concurrent jobs pull from the same public registry, they can collectively exhaust that registry's per-IP rate limit even though no single job is doing anything wrong. Docker Hub, for example, currently documents a pull-rate limit of 100 pulls per 6-hour window per IP address for unauthenticated pulls and 200 pulls per 6-hour window for an authenticated free-tier account; a fleet of shared runners hitting Docker Hub anonymously is a textbook source of intermittent 5xx-and-timeout style failures that look random but are actually load-dependent.
5. Add retries with backoff at the package-manager level, not just at the CI-step level: most package managers support this natively (npm config set fetch-retries 5 fetch-retry-mintimeout 5000, pip install --retries 5, apt-get with Acquire::Retries=5), so a single dropped connection does not fail the whole job. CI-step-level "retry the whole job" retries are a much blunter and slower fallback for the same problem.
6. Add a mirror or pull-through cache for the packages and images the build depends on (an internal npm/PyPI proxy such as Artifactory or Verdaccio, or an authenticated pull-through mirror for base images) so most requests never leave the internal network, and the ones that do are authenticated rather than anonymous.
7. Add dependency caching (a package-manager cache directory persisted between CI runs, or a BuildKit cache mount, which keeps a package-manager cache across builds without baking it into an image layer) so a transient failure on a package you already fetched last run does not even trigger a new network request.
8. Confirm the fix by measuring failure rate, not a single green run. Re-run the pipeline enough times (or track the failure rate over the following days) to see the intermittent-failure rate actually drop, since a single passing run after a change proves nothing about a problem that was intermittent to begin with.

Trade-offs and pitfalls

Retries alone can mask a real, worsening capacity problem: if a shared-IP rate limit is the root cause, retries just spread the same load over more time and eventually stop working as traffic grows, so the actual fix is authenticating pulls or standing up a mirror, not retrying harder. Mirrors and caches add an operational dependency of their own (someone now owns keeping that internal proxy up and its cache warm), and a mirror that silently falls behind upstream can serve a stale or vulnerable package version without anyone noticing until it matters.

Direct answer

In development, a bind mount (a direct mapping of a path on the host's own file system into the container) is usually the better fit for a stateful database, because it puts the data somewhere a developer can see, back up, and delete with ordinary host tools, at the cost of portability and occasional permission friction. In production, a named Docker volume (storage that Docker itself creates and manages, identified by a name rather than a host path) is the better fit, because it is portable across hosts, does not depend on a specific host directory existing with the right permissions, and integrates with Docker's own backup and volume-driver tooling rather than whatever ad hoc scripts a bind mount setup would need.

Structured elaboration

Portability
A bind mount hardcodes a host path into the container's configuration; moving the workload to a different host, or a different developer's machine with a different directory layout, means updating that path everywhere it is referenced. A named volume is referenced only by name; Docker manages where its data actually lives on disk, and that mapping does not need to change when you move the container definition to a new host, or attach the same volume to a replacement container after an upgrade.

Performance
On Linux hosts, both a bind mount and a named volume are ordinary file system access with negligible overhead. The gap shows up specifically on Docker Desktop for macOS or Windows, where a bind mount crosses into a virtualized Linux environment and back, and can be noticeably slower for I/O-heavy workloads (a database doing many small reads and writes is exactly this case) than a named volume, which lives natively inside that virtualized environment and avoids the cross-boundary file-sharing layer entirely.

Backup strategy
A bind mount's data is just files at a known host path, so host-level backup tools (a cron job running rsync or a snapshotting file system) work without any Docker-specific knowledge, which is convenient for a single developer's machine. A named volume's data lives inside Docker's own storage area, so backing it up properly means either running a helper container that mounts the volume and streams a tar archive out (docker run --rm -v myvolume:/data -v $(pwd):/backup alpine tar czf /backup/myvolume.tar.gz -C /data .), or using a volume driver that has its own backup integration. This is a small amount of extra ceremony that is worth it in production for the portability and permission benefits above.

Permission issues, including SELinux
A bind mount exposes the container to the host's actual file ownership and permission model directly: a container process running as a different user ID (UID) than whatever owns the host directory can get permission-denied errors immediately, since the container is not exempt from ordinary Unix file permissions just because it is a container. On a host with SELinux (Security-Enhanced Linux, a mandatory access control system that restricts what a process may do to specific files beyond ordinary Unix permissions, common on Red Hat family distributions) enforcing, a bind-mounted directory additionally needs the right SELinux label before a container process can access it at all, even if the Unix permission bits look correct; Docker's :z and :Z mount flags request that label automatically (:z shares the label across multiple containers that need to read the same host directory; :Z marks it private to just this one container). A named volume mostly avoids this class of problem, since Docker creates and manages its storage directly and typically already applies a compatible label, rather than depending on whatever an existing host directory happened to be labeled for some other purpose.

Worked example

A developer runs Postgres locally with -v ./pgdata:/var/lib/postgresql/data (a bind mount) so they can browse the raw data files, delete the directory to reset state instantly, and back it up by just copying the folder; on a Fedora workstation with SELinux enforcing, the container fails to start with a permission error until the mount is changed to -v ./pgdata:/var/lib/postgresql/data:Z, which applies the correct private SELinux label to that directory for this container. In production, the same service instead uses -v pgdata:/var/lib/postgresql/data (a named volume): the same container definition now works unchanged on any host in the fleet, regardless of that host's specific directory layout or SELinux configuration, and backups run through a scheduled helper container rather than a host-specific script tied to one machine's file paths.

Trade-offs & pitfalls

• Using a bind mount in production because "it worked in development" reintroduces exactly the portability and permission problems a named volume exists to avoid, the first time the container needs to run on a different host.
• Using a named volume in development removes the convenience of browsing and editing the data directly with ordinary host tools, which is often exactly what a developer wants while debugging.
• Forgetting the SELinux label on a bind mount on an enforcing host produces a permission error that looks identical to a plain Unix permissions problem; check getenforce on the host before assuming the fix is a UID or chmod change.

Direct answer

A rollback under a faulty prod tag has one job: get back to the exact previous image bytes as fast as possible, without relying on a rebuild, since even a rebuild from "the same" source tag is not guaranteed to reproduce identical bytes if a base image or a dependency resolves differently since then. With Docker Compose, that means pointing the affected service at the previous image's content digest directly and recreating just that one service.

The immediate rollback

1. Find the last known-good digest. If digests were already being recorded at deploy time, this is a lookup in the deploy log, not a hunt. Without that record, docker images --digests or the registry's own tag history is the fallback, which is exactly the gap worth closing before an incident, not during one.
2. Point the compose file at that digest, not the floating tag:

services:
  app:
    image: registry.example.com/app@sha256:d6cd6f0583314b6592c3e0b0ae2f6778a6b2373e9cf283c3d979c65c7ebf47d7

3. Recreate only that service: docker compose up -d --no-deps app (adding --force-recreate if Compose believes nothing changed). This pulls the pinned digest, guaranteed to be the same bytes as before since digests are content-addressed, and swaps only that one container, leaving sibling services and the network untouched.
4. Verify before declaring it done. Confirm the running container's actual image digest matches what was intended (docker inspect --format='{{.Image}}' cross-checked against the pulled digest), and that health checks or smoke tests pass. A rollback that silently landed on the wrong digest is worse than a slower, correct one.

Making this safer and faster with automation

• Record last-known-good automatically. Every successful, health-checked deploy writes its digest to a small, durable record, a file, a row in a deploy-tracking table, or a tag like app:last-good that only the automated pipeline is ever allowed to move. This turns step one above from a stressful hunt through logs into a lookup that already exists before the incident happens.
• A one-command rollback script. Wraps the steps above into a single invocation: it reads the last-known-good record, updates the compose file's image reference for that one service, runs the targeted docker compose up -d --no-deps, then runs the same smoke test the normal deploy pipeline runs before declaring success. The goal is that an on-call engineer under pressure runs one command instead of reconstructing this whole sequence from memory.
• Guardrails. The rollback script should refuse to "roll back" to the digest that is already running, protecting against accidentally rolling back twice, and should log its own action to the same deploy-tracking record a normal deploy would, so the audit trail stays accurate even during an incident.

Trade-offs and pitfalls

Rolling back the image alone does not roll back anything that shipped alongside it: a database migration that ran with the bad release does not reverse itself, so a rollback plan needs a documented answer for what happens when the bad release also changed schema, not just code. --no-deps matters specifically because a full docker compose up -d can recreate dependent services too if their configuration appears to have changed, a much bigger blast radius (a bigger set of things a mistake could break) than the incident called for. Automating "last known good" purely as "the previous deploy" is subtly wrong if that previous deploy was also bad; the record should track the last deploy that actually passed a real health check, not merely the one immediately before this one chronologically.

Direct answer

For a container that runs as root, ships general shell utilities, is built from a large general-purpose base, and is directly exposed to untrusted input, the goal is defense in depth: assume the application itself will eventually be exploited, since it is the one thing an attacker directly controls, and make sure a successful exploit lands in a process with almost nothing useful available to it. No root privileges, no unnecessary Linux capabilities, no writable filesystem, and no shell or general-purpose tooling to pivot with.

The policy, layer by layer

• Run as a non-root user. Add a dedicated user in the Dockerfile and set USER, so a process compromise does not automatically hand the attacker root inside the container. This alone does not stop a container-escape-class exploit, but it removes the most common privilege-escalation shortcut and is a prerequisite for the other controls below being fully effective: a read-only root filesystem matters far less if the compromised process is still root and able to remount things.
• Drop Linux capabilities. Linux capabilities are the finer-grained privileges historically bundled together into "root": binding low network ports, changing file ownership, loading kernel modules, and dozens more. Docker grants a small default set to every container, and a typical API service needs almost none of it. --cap-drop=ALL, then adding back only what is strictly required (commonly nothing at all for a plain HTTP service; CAP_NET_BIND_SERVICE only if it must bind a port below 1024), shrinks what a compromised process can even ask the kernel to do, regardless of its user ID.
• Read-only root filesystem. --read-only at runtime makes the entire root filesystem immutable, combined with a tmpfs mount for the one or two paths the application genuinely needs to write to, a cache directory, /tmp. Even a successfully injected payload cannot persist itself to disk inside the container or tamper with the application's own binaries.
• Rootless containers, a different control from a non-root USER. The USER instruction changes which user ID the process runs as inside the container. Rootless mode goes a layer deeper: it runs the container engine itself without root privileges on the host, using Linux user namespaces so that even a process that is user ID 0 inside the container maps to an unprivileged, ordinary account on the host. A container-escape exploit that would normally hand an attacker host root instead hands them an unprivileged host account, a meaningfully smaller blast radius if the layers above somehow fail.
• Seccomp. A seccomp (secure computing mode) profile is a kernel-enforced allowlist or denylist of which system calls a process may make at all. The engine's default profile already blocks a range of dangerous, rarely needed syscalls, including loading kernel modules; a custom profile generated from tracing what a specific workload actually uses, denying everything else, closes off entire exploitation techniques, since many container-breakout and privilege-escalation chains rely on a syscall the application itself never legitimately needs.

Enforcing and auditing at deploy time

None of the above helps if it is only a suggestion in a document. Enforce it at the deploy gate, a policy engine or admission controller that rejects a deployment specification missing a non-root user requirement, a dropped-capabilities list, or a read-only filesystem setting, rather than trusting every team to remember. Audit continuously by periodically diffing running container configurations against the required policy and alerting on drift, since a policy checked only at initial deploy time misses a manual run or an emergency change that bypassed the pipeline.

Worked example, measured

This exact hardening stack was built and run against a small HTTP service: a dedicated non-root user baked into the image, started with --read-only --cap-drop=ALL --security-opt no-new-privileges --tmpfs /tmp. Inside the running container, whoami returned the unprivileged application user, not root, and id showed no supplementary groups beyond that user's own group. The service still answered a real request over its published port with the expected response body, confirming the hardening did not break operability. Attempting to write a file outside the tmpfs mount (touch /app/newfile) failed with "Read-only file system," confirming the control was actually enforced rather than merely configured.

Trade-offs and pitfalls

A read-only root filesystem breaks any application that writes logs, temp files, or cache data to an unexpected path by default; audit what the application actually touches, or test it once with --read-only in staging, before enforcing it in production, and provide tmpfs or volume mounts for the legitimate write paths rather than disabling the control entirely. Dropping every capability can break something subtle that quietly relied on one, binding a low port, or a health check needing a specific network privilege, so validate functionally, not just that the image built successfully. A custom seccomp profile generated from one narrow test run can be too restrictive if it never exercised every code path, an error-handling branch that needs a syscall the happy path never calls, so generate it from realistic, broad testing, or accept a slightly looser default profile over an untested custom one that breaks in production on the first edge case.