Containerization and Docker Fundamentals Questions

Direct answer

On distributed CI (continuous integration) runners, the local build cache that normally makes instruction-ordering discipline work does not help, because a given job can land on a different, cold runner every time with no prior build history at all. The fix is to stop relying on any one runner's local cache and make the cache itself a portable artifact, using the remote cache export and import built into BuildKit (Docker's modern build engine, driven from the command line through the docker buildx plugin) against a registry, so a fresh runner can pull down another runner's cache before it builds anything.

The design, piece by piece

Shared base images. Build and publish one pinned internal base image, with the OS, language runtime, and common system packages already installed, that every service's Dockerfile starts FROM, instead of forty services each independently installing the same packages. This collapses the expensive, rarely-changing part of every build into one artifact that only needs rebuilding when the base itself changes.

BuildKit remote cache export and import. docker buildx build --cache-to=type=registry,ref=<registry>/<service>-cache,mode=max ... pushes the build's layer cache to a registry as its own artifact after every build. --cache-from=type=registry,ref=<registry>/<service>-cache on the next build, on any runner, cold or warm, pulls that cache down before building starts, so a fresh runner behaves as if it had just built this service minutes ago. mode=max, versus the default mode=min, exports cache for every intermediate layer, including ones that never end up in the final image, which matters for multi-stage Dockerfiles where a builder stage's cache would otherwise be discarded.

Cache namespacing per service. One shared cache reference for all forty services in the monorepo causes constant cross-service cache thrashing, where one service's build evicts or overwrites cache entries another service needed. Namespace the cache reference per service, and typically per branch too (<registry>/cache/<service>:<branch>, with a fallback import from :main when a feature branch has no cache of its own yet), so each service's cache import only ever competes with its own history, never the whole monorepo's.

Securing and storing the remote cache. The cache registry needs the same access control as the image registry, because a poisoned cache blob is later reused as literal layer content in a real build, making it a genuine supply-chain risk, not a throwaway artifact. Use a private repository with the CI system's own short-lived, narrowly scoped credentials rather than a long-lived shared token. Plan for retention: cache blobs accumulate fast under mode=max across many services and branches, so a time-to-live or a periodic prune job on the cache repository is needed, since unlike the production image registry, cache artifacts have no long-term retention requirement.

flowchart LR
    RunnerA["cold CI runner A"] -->|"pull"| Base["shared base image"]
    RunnerA -->|"--cache-from"| CacheReg["cache registry (per-service namespace)"]
    RunnerA -->|"build"| Image["service image"]
    Image -->|"--cache-to"| CacheReg
    RunnerB["cold CI runner B"] -->|"--cache-from"| CacheReg

Worked example

A Node.js service in the monorepo starts FROM internal/node-base:20.3-slim (the shared base), then COPY package*.json . followed by RUN npm ci (the dependency layer, ordered so the most rarely changing inputs come first), then COPY . .. The CI job runs docker buildx build --cache-from=type=registry,ref=registry.internal/cache/checkout-service:main --cache-to=type=registry,ref=registry.internal/cache/checkout-service:main,mode=max -t checkout-service:$SHA .. The first build on a brand-new runner still pulls the npm ci layer from the registry cache blob instead of recomputing it, because the cache import happens before the build starts, entirely independent of anything that specific runner has ever built before.

Trade-offs and pitfalls

mode=max cache exports are noticeably larger to push and pull than mode=min, a bandwidth-versus-hit-rate trade-off that gets worse the more services share one CI cluster's egress (its outbound network capacity to the registry). Per-branch namespacing needs a sane fallback chain, branch cache miss falling back to the main branch's cache, or every new feature branch pays a full cold build. A shared base image becomes a coordination point in its own right: bumping it now needs a rollout plan across every dependent service, rather than each service upgrading on its own schedule, a real cost worth naming rather than treating this design as a pure win.

Direct answer

A multi-stage build failing with a missing shared library at runtime almost always means a runtime-only dependency (a shared library that a compiled extension needs, but that was only ever installed as a side effect of the builder stage's toolchain) never made it across the COPY --from=builder boundary; the fix is to find exactly which file or package is missing and add that one thing to the final stage, not to widen what gets copied or fall back to a single fat stage.

Step-by-step debugging plan

1. Reproduce interactively against the final image, not the builder. docker run --rm -it --entrypoint sh <final-image> and attempt the failing import or command by hand, to get the exact error and confirm it happens in the shipped image, not just in some other environment.
2. Identify precisely what is missing. ldd (a standard Linux utility that lists the shared libraries a binary depends on) run against the specific binary or shared object (ldd $(which python3), or ldd path/to/the_compiled_extension.so) prints => not found next to any dependency the runtime image cannot resolve. This turns a vague ImportError into a concrete missing filename.
3. Compare against the discarded builder stage. docker history only shows the layers of the image you run it against, which for a multi-stage build is the FINAL stage; the intermediate builder stage's layers are discarded once the build finishes and are not visible this way. To inspect what the builder stage actually had, build up to it explicitly and tag it: docker build --target <builder-stage-name> -t debug-builder ., then run docker history debug-builder and docker run --rm -it debug-builder sh to confirm the missing file genuinely exists there.
4. Use dive (a terminal tool for browsing a Docker image's layers and file contents) on the tagged debug-builder image to search for the specific missing path or package by name, confirming both that it exists in the builder and which layer introduced it, then check the COPY --from=builder line in the final stage to see whether that path was actually included in what got copied over.
5. Fix with the smallest addition that closes the gap, either a targeted COPY --from=builder /exact/path/to/lib.so /usr/lib/ for a single file, or installing the specific runtime package via the distribution's package manager (for example apt-get install -y --no-install-recommends libpq5), never the corresponding -dev/full build-toolchain package, which drags headers, compilers, and static libraries back into the image the multi-stage build was written to keep out.
6. Verify without reintroducing bloat. Re-run the same reproduction from step 1 to confirm the import now succeeds, and check docker images for the size delta: a targeted runtime library addition should move the image size by kilobytes, not the hundreds of megabytes a reintroduced build toolchain would add.

Worked example

I verified this exact failure shape directly: python:3.12-slim, a common runtime base for a multi-stage Python build, does not have the PostgreSQL client library installed by default (ldconfig -p | grep -i libpq inside a fresh python:3.12-slim container returns nothing). A realistic version of this bug: a python:3.12 builder stage compiles psycopg2 from source, which links against libpq because the builder stage has libpq-dev installed (it needs the development headers to compile against); the final stage COPY --from=builder /usr/local/lib/python3.12/site-packages /usr/local/lib/python3.12/site-packagess the compiled package across, but never installs the runtime libpq5 package in the slim final stage. The result at runtime is ImportError: libpq.so.5: cannot open shared object file: No such file or directory, exactly the symptom ldd against the compiled psycopg2 .so file would surface directly. The fix is a single added line in the runtime stage: RUN apt-get update && apt-get install -y --no-install-recommends libpq5 && rm -rf /var/lib/apt/lists/*, which adds the shared library alone, not the development headers or compiler that libpq-dev would bring back.

Trade-offs and pitfalls

The tempting shortcut, when a missing dependency shows up, is to just install the same -dev package in the final stage that the builder stage used, since it is guaranteed to fix the immediate error; this reliably reintroduces the exact bloat multi-stage builds exist to avoid, because dev packages typically pull in a compiler toolchain and headers that a running application never needs. A second pitfall is fixing the symptom for one missing library and stopping, without checking whether the same class of gap exists for other compiled extensions in the same image; ldd against every compiled .so file the application actually loads, not just the first one that errored, catches the rest before they surface one at a time in later incidents.

Direct answer

A Dockerfile is a plain-text set of instructions that Docker's build engine executes in order to produce an image. Some instructions run at build time and create a new filesystem layer (FROM, RUN, COPY, ADD); others are metadata that gets recorded on the image and only takes effect when a container actually starts (CMD, ENTRYPOINT, EXPOSE, ENV at runtime, HEALTHCHECK); a couple sit in between.

The instructions, using a simple web service as the example

FROM python:3.12-slim
ARG APP_VERSION=1.0
ENV PORT=8000
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
EXPOSE 8000
VOLUME /app/data
HEALTHCHECK CMD python -c "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8000/')" || exit 1
CMD ["python", "app.py"]

Layer-creating versus metadata-only, and build-time versus runtime

The pattern worth internalizing: instructions that touch the filesystem during the build (FROM, RUN, COPY, ADD) create layers, because that is literally what a layer is, a filesystem diff. Instructions that configure behavior (ENV, ARG, WORKDIR, EXPOSE, VOLUME, CMD, ENTRYPOINT, HEALTHCHECK) are recorded as image metadata and add negligible size, but they split further into when they take effect: ARG only exists during the build and is gone by the time a container runs, ENV is set at build time but its value is then baked into the image and visible to every container started from it, and CMD/ENTRYPOINT/HEALTHCHECK do nothing at all until a container is actually started.

Common mix-ups

Confusing EXPOSE with actually publishing a port (it is documentation for humans and other tooling; -p on docker run is what actually maps a port); assuming ENV values set during the build can be overridden by an ARG after the fact (it is the reverse: ARG values can be used to set an ENV's default, not changed by one); and forgetting that VOLUME declared in a Dockerfile creates an anonymous volume automatically if you do not mount something specific there yourself, which can silently leave stray data outside your intended storage.

Direct answer

For a service with a genuinely slow startup and flaky dependencies, size HEALTHCHECK's start_period to comfortably exceed the worst realistic startup time (including retries against a slow dependency), keep interval and retries tight enough that a true failure is caught within your deployment tool's own timeout, and keep the restart policy focused on crashes, not on health status, since plain Docker does not restart a container just because it reports unhealthy. On top of the health check itself, add startup-phase logging and a small number of metrics (time to first healthy, dependency-connect latency, restart count) so you can tell "slow but fine" apart from "actually stuck" without staring at docker inspect.

Structured elaboration

Sizing the HEALTHCHECK knobs

• start_period: failures during this window do not count toward retries and do not flip the container to unhealthy; the status shows starting instead. Set it to comfortably exceed your measured p99 startup time (dependency waits included), not your typical case. If startup can occasionally take three minutes when a dependency is slow, a sixty-second start_period will produce exactly the flapping this design is meant to avoid.
• interval and timeout: once past start_period, these control how quickly you detect a real regression. A shorter interval catches problems sooner but adds load; a timeout shorter than the check's own worst-case latency will produce false failures under normal load, not just real outages.
• retries: the number of consecutive failures required to flip to unhealthy. Combined with interval, this is your total detection latency after start_period ends (interval x retries, roughly). Tune it against whatever consumes the status: a deployment tool waiting for containers to become healthy before proceeding, or an operator dashboard.

Restart policy, and the boundary it does not cross

• A restart policy (on-failure[:max-retries], always, unless-stopped) reacts to the container's process exiting, not to the health status. A container can sit at unhealthy indefinitely without ever being restarted by Docker itself. This matters here specifically: if a dependency is temporarily unavailable, you want the health check to say unhealthy (so the status is honest and load balancers or an orchestrator can act on it) without the container being torn down and restarted, which would only add restart churn on top of an already-degraded dependency and could reset any warm state (caches, pooled connections) for no benefit.
• Use on-failure:N for genuine crash protection (the process actually exits), and let the health status, not the restart policy, communicate "temporarily degraded, dependency down."

What still belongs in logs and metrics, beyond the health check

• A single structured log line per startup phase (config loaded, dependency connected, ready to serve) so a human can see exactly where a slow startup is spending its time, rather than staring at a binary unhealthy flag.
• A startup-duration metric (time from process start to first successful health check) so a slow trend is visible before it becomes an outright failure.
• A dependency-connect-attempt counter or histogram, so "dependency flaky" is a graph, not a guess reconstructed from timestamps in raw logs after the fact.
• The count of consecutive health-check failures and the current status, exported as a metric, so dashboards and alerting do not have to poll docker inspect to know what Docker already knows internally.

Worked example

A service depends on a message broker that occasionally takes up to 90 seconds to accept connections during a rolling broker upgrade, and the service itself otherwise starts in about 5 seconds. Setting start_period=120s, interval=5s, timeout=3s, retries=3 gives the connection up to two minutes to succeed before any failure counts at all, and once past that window, a real failure is confirmed within about 15 seconds (3 retries x 5s). The restart policy is on-failure:5: if the process actually crashes (a bug, not a slow dependency), it gets a bounded number of automatic restarts, but a container that is merely waiting on a slow broker is never killed by Docker itself; it stays starting, then unhealthy if the broker genuinely does not come back, and an operator or orchestrator layered on top decides what to do about that, informed by the startup-phase logs showing exactly which dependency it was still waiting on.

Trade-offs & pitfalls

• A start_period set too long delays real detection: a container that is actually broken (not just slow) will look identical to a healthy one for that entire window.
• A start_period set too short reintroduces the original problem: legitimate slow starts get flagged as failures, and if something downstream (a deployment tool, an orchestrator) does act on that status, you get exactly the premature restart churn this design is trying to prevent.
• Relying on the restart policy to "fix" a container stuck at unhealthy is a common but wrong instinct on plain Docker: it will not restart on health status alone, so the actual remediation (alerting a human, or letting an orchestrator's own readiness/liveness model act) has to live somewhere else.

Direct answer

Start with docker history, which prints every layer in the image along with the size it added and the instruction that created it; sort mentally (or pipe through sort -h on the size column) to find the one or two layers responsible for most of the growth, then go fix whatever Dockerfile instruction produced them. Growth from 900 MB to 4+ GB is almost always one of a handful of well-known mistakes, not death by a thousand small cuts.

A layer-by-layer investigation method

1. Run docker history --no-trunc <image> and look at the SIZE column for each layer; the --no-trunc flag matters because it shows the full command instead of a truncated one, which you need to tell two similar RUN lines apart.
2. For any suspiciously large layer, look at what CreatedBy command produced it; a RUN apt-get install or RUN pip install line that jumps by hundreds of megabytes is your first suspect.
3. Check the base image itself: docker images <base-image> for the tag you are using; a full python:3.12 or node:20 image can be over a gigabyte on its own, versus low hundreds of megabytes for the -slim variant.
4. Check whether the build context has a .dockerignore; without one, a COPY . . can silently pull in .git, local virtual environments, or node_modules from the host, none of which belong in the image.

Worked example

Building a deliberately unoptimized image and inspecting it:

FROM python:3.12
RUN apt-get update && apt-get install -y build-essential
RUN pip install --no-cache-dir pandas==2.2.3 scikit-learn==1.5.2
COPY . .

$ docker history --no-trunc --format "{{.Size}}\t{{.CreatedBy}}" bloated-image
393MB   RUN /bin/sh -c pip install --no-cache-dir pandas==2.2.3 scikit-learn==1.5.2
22.6MB  RUN /bin/sh -c apt-get update && apt-get install -y build-essential
...
73.5MB  RUN ... (compiling Python from source inside the base image)

$ docker images bloated-image --format '{{.Size}}'
2.14GB

Two things jump out immediately from that history: the pip install layer alone is 393 MB, and the base image is python:3.12 (the full, non-slim variant), which compiles CPython from source and measures roughly 1.6 GB to 2 GB on its own before a single application dependency is added (the exact figure moves a little with CPU architecture and the current Debian point release). For comparison, python:3.12-slim pulled fresh is 204 MB, roughly an eighth of that (measured directly: 204 MB against a 1.62 GB python:3.12 pull), not the mere rounding difference a smaller gap would suggest. In a real "900 MB became 4 GB" case, docker history on the new image would show the same signature: either a newly-added large RUN layer, or a base image tag that quietly changed from a slim variant to a full one (for example, someone bumped FROM python:3.12-slim to FROM python:3.12 to "fix" a missing build tool, and never reverted it).

The usual suspects

Using a full, non-slim base image instead of a slim or distroless one; installing build tools (compilers, -dev headers) in the same layer as the final application instead of a multi-stage build, so the toolchain ships in production; not cleaning package-manager caches in the same RUN layer they were populated in (rm -rf /var/lib/apt/lists/* has to happen in the same RUN as the apt-get install, or the cache is already baked into an earlier layer and cannot be removed by a later one); and a missing or incomplete .dockerignore letting COPY . . pull in .git, log files, or dependency directories from the host.