Containerization and Docker Fundamentals Questions

Direct answer

If an API cannot reach its database by service name inside the same Compose stack, the near-universal cause is that the two containers are not actually on a network with Docker's embedded DNS server enabled, most commonly because they ended up on the default bridge network instead of a user-defined one: Docker's default bridge network intentionally does not do name-based service discovery, while any user-defined network (which is what Compose creates for you automatically, unless you have overridden it) runs an embedded DNS resolver that resolves container and service names to their current IP addresses automatically.

Structured elaboration

Debugging DNS resolution and network attachment, in order

1. Confirm both containers are actually attached to the same network: docker network inspect <network-name> and look for both container names in its Containers section. A Compose stack with a mis-scoped networks: block, or a container started with docker run outside the Compose-managed network entirely, is invisible to the other side no matter what DNS is doing.
2. From inside the calling container, attempt resolution directly: docker exec <api-container> getent hosts <db-service-name> (or nslookup, if available in that image). A clean IP address back means DNS resolution itself works and the problem is elsewhere (the port, the database not actually listening yet, a firewall rule); a resolution failure means the DNS layer is the actual problem.
3. Check which network the containers landed on: docker network ls and docker inspect <container> --format='{{json .NetworkSettings.Networks}}'. If it shows the network named literally bridge, that is Docker's default network, not a Compose-created user-defined one, and name resolution by container name is not expected to work there at all.
4. If both containers are confirmed on the same user-defined network and resolution still fails, check for a typo or mismatch between the service name used in code and the actual Compose service name (Compose registers the service name, and, for the container's own hostname, the container name, as DNS entries on the network it creates; a hardcoded old container name or a renamed service is a common, easy-to-miss mismatch).

Default bridge network versus a user-defined network, for service discovery specifically

• The default bridge network (what you get from plain docker run with no --network flag) does not run an embedded DNS server for container name resolution at all; containers on it can only reach each other by IP address (historically, the deprecated --link flag patched hostnames into /etc/hosts as a workaround, which is not how modern setups should discover services).
• Any user-defined network, including the ones Compose creates automatically for a project, runs Docker's embedded DNS resolver (at the well-known address 127.0.0.11 inside each container on that network), which resolves both container names and Compose service names to the correct current IP address, and updates automatically if a container restarts and gets a new IP.
• This is precisely why Compose "just works" for service-to-service calls by name out of the box: it always creates a user-defined network for a project unless you explicitly override that, so the embedded DNS behavior is present by default, not something you have to opt into separately.

Worked example

Two containers, api and db, are both started, and api's logs show connection refused or "unknown host" errors when it tries to reach db by name. Running docker exec api getent hosts db returns nothing and exits non-zero, confirming DNS resolution itself is failing, not just the connection. docker network ls shows both containers only ever attached to the default bridge network (the Compose file had accidentally used network_mode: bridge on the api service, which pins it to Docker's default bridge network instead of letting Compose create and use its own project network). Removing that override lets both services join the Compose-created user-defined network instead; getent hosts db then resolves to the container's current IP address, and the application connects successfully on the very next attempt, with no code change at all.

Trade-offs & pitfalls

• It is easy to "fix" this the wrong way by hardcoding the database's current IP address once resolution is confirmed working; container IPs on a bridge network are not stable across restarts, so this reintroduces the same failure the first time the database container restarts and is assigned a different address.
• A resolution failure and a "resolves fine but the connection is refused anyway" failure look similar from the application's point of view (both surface as some flavor of connection error) but have entirely different causes; always separate the DNS step (getent hosts) from the actual connection step before concluding which layer is broken.
• Legacy guidance mentioning the --link flag as a way to make one container resolve another by name is outdated; a user-defined network's embedded DNS server does that automatically for every container on the network, with no per-pair linking needed.

Direct answer

A tag like v1.2.3 or latest is a mutable, human-friendly pointer that can be reassigned to a different image at any time. A digest like sha256:... is a content-addressable hash of the image's manifest: it can only ever refer to one exact set of bytes, and if the image content changes, the digest changes with it. You deploy by digest in production so that "deploy version 1.2.3" cannot silently mean a different image today than it meant yesterday.

Why tags are not enough for production

Nothing stops someone (or an automated pipeline) from pushing a new image and re-pointing an existing tag at it; latest is the most obvious case, but even a version-looking tag like v1.2.3 can be overwritten by a re-push. If your deployment manifests reference myapp:v1.2.3, a rollback, an incident replay, or a second cluster pulling the "same" tag at a different time can end up running genuinely different bytes without any change to your deployment configuration. A digest closes that gap: myapp@sha256:abc123... is the same image forever, because the hash is computed from the image's actual manifest content.

Worked example

Getting an image's digest after pushing it, verified directly against a local image:

$ docker inspect --format='{{index .RepoDigests 0}}' alpine:3.20
alpine@sha256:d9e853e87e55526f6b2917df91a2115c36dd7c696a35be12163d44e6e2a4b6bc

(docker push also prints the digest directly in its final output line when you push to a registry.) To reference that exact digest afterward, whether pulling manually or writing a deployment manifest, you swap the tag for the digest in the same repository[:tag] position:

$ docker pull alpine@sha256:d9e853e87e55526f6b2917df91a2115c36dd7c696a35be12163d44e6e2a4b6bc

or, in a Kubernetes (a system for orchestrating containers across many machines) pod spec or a docker run/Compose file, image: myregistry/myapp@sha256:abc123... instead of image: myregistry/myapp:v1.2.3. Whoever runs that reference, on whatever machine, on whatever day, gets the identical image content, because the reference is the content's hash rather than a label someone assigned to it.

Trade-offs

Digests are unreadable to a human glancing at a deployment manifest (nobody can tell sha256:abc123... is "the release from last Tuesday" without cross-referencing a build log), so most teams keep both: a semantic or commit-SHA tag for humans and CI/CD (continuous integration and continuous delivery) tooling to reason about, and pin the actual deployed reference by digest so the running artifact is provably immutable regardless of what the tag does later.

Direct answer

PID 1, the process the kernel assigns process ID 1 to, either at boot on a normal host, or as the first process started inside a container's own PID namespace (an isolated view of process IDs separate from the host's), has two duties an ordinary process never has to think about: it inherits any process whose original parent has already died (an orphan) and must call wait()/waitpid() on it to collect its exit status, and the kernel gives it special signal-handling behavior, a signal being the kernel's way of asynchronously telling a process about an event such as a shutdown request, where most signals simply aren't delivered to PID 1 unless it explicitly installs a handler for them (only SIGKILL and SIGSTOP, which force-terminate or pause a process, always work regardless). An ordinary application binary run directly as PID 1 usually does neither: it never calls wait() on children it didn't fork itself, so those orphans exit but are never reaped and pile up as <defunct> zombies, and it never registered a SIGTERM handler, so it may not even shut down cleanly on docker stop. The concrete fix is to put a tiny, purpose-built init process in front of the application: run docker run --init (which uses tini, bundled into the Docker Engine since version 1.13) or add tini/dumb-init as the image's ENTRYPOINT, so the real application runs as PID 2 and the init process handles reaping and signal forwarding on its behalf.

Structured elaboration

Why zombies accumulate specifically because of PID 1, not just "processes exiting"

A zombie is a process that has already exited but whose exit status hasn't been collected yet; the kernel keeps a small process-table entry for it until something calls wait(). On a normal host, the system init, the first process the kernel starts at boot and the one responsible for starting and supervising every other service on the machine (systemd is the most common example on modern Linux distributions), reaps any orphan automatically, all the time, as a routine part of its job. Inside a container, whatever process the container engine starts is that namespace's PID 1, whether or not it was written with that responsibility in mind. Most application binaries (a Python script, a Node server, a compiled Go binary) were never written to reap other processes; they assume some init layer is already doing that for them, true on a host, false inside a bare container. The practical trigger is usually an application that shells out to short-lived helper processes, each one that exits without being reaped adds one more zombie, and because a single zombie barely consumes resources, this can run for a long time before it becomes visible as process-table pressure or confusing monitoring output.

Comparing the three fixes

sequenceDiagram
    participant App as PID 1 process
    participant Child as Worker process
    App->>Child: fork
    Child-->>Child: exits
    Child->>App: SIGCHLD delivered
    alt PID 1 calls waitpid
        App->>Child: reap exit status
    else PID 1 never calls waitpid
        Child->>Child: stays a zombie, shown as defunct
    end

The right default for almost every case is the lightweight init: a one-line ENTRYPOINT or flag change that doesn't meaningfully alter the container's resource footprint, and it fixes both halves of the problem, reaping and signal forwarding, without touching the container's isolation model. systemd-in-a-container earns its cost only when deliberately running something that behaves like a small VM with several cooperating services, a rare, specific need, not a general containerization pattern. Sharing the host PID namespace is a debugging tool, letting you inspect host processes from inside a container, not a production fix, because the isolation it gives up is exactly the isolation a container exists to provide.

Worked example

An engineer notices ps aux inside a long-running container shows dozens of <defunct> entries, and the container's memory climbs slowly over days despite the application's own metrics looking flat. The application is a Python service that shells out to a short-lived helper process per request, and one of those helper processes occasionally forks a further child that gets orphaned before it exits, a grandchild the application's own subprocess-handling code never sees and so never reaps. The Dockerfile's ENTRYPOINT currently runs the Python app directly as PID 1: ENTRYPOINT ["python", "app.py"]. Adding docker run --init (or, equivalently, ENTRYPOINT ["tini", "--", "python", "app.py"]) puts tini at PID 1; the Python app becomes PID 2, and any orphaned grandchild now reparents to tini, which reaps it immediately. No application code changes, no measurable image size increase, and docker stop now delivers SIGTERM to tini, which forwards it to the app, instead of the kernel silently declining to deliver it to an application that never asked to handle it.

Trade-offs & pitfalls

• A common wrong turn is treating this as "the app should just add a SIGTERM handler and move on," which fixes shutdown but not reaping; an app can handle its own termination signal correctly and still accumulate zombies from subprocesses it spawns and never wait()s on.
• docker run --init only takes effect at docker run time; if the container is launched by an orchestrator that constructs its own run configuration, the equivalent has to be configured there, or baked into the image's ENTRYPOINT instead, which works regardless of how the container is launched and is usually the more portable choice.
• --pid=host looks like it "just works" in a quick test because reaping does start happening; that's exactly the trap, the fix appears to work while quietly removing process isolation, which won't surface as a problem until it's exploited or audited.
• Running systemd in a container without the specific cgroup and mount configuration it expects tends to fail in ways that look unrelated to init at all (units failing to start, logging silently going nowhere), which sends people debugging the wrong layer if they don't already know systemd needs that container-specific setup.

Direct answer

I split this into two separate questions that are easy to conflate: is the application's own code genuinely doing more work (an application-level profiling question), or is the container's CPU allocation causing it to be paused and resumed in a way that looks like erratic high usage from the outside (a container-level scheduling question)? Intermittent high CPU is very often the second one, specifically CPU throttling, which needs to be ruled out before spending time profiling code that may not actually be the problem.

Application-level profiling

Tools depend on the language, but the goal is the same: find which functions are actually consuming CPU time, as opposed to time spent waiting on I/O that gets misattributed as "CPU usage" by a coarse metric. perf (a Linux profiling tool that samples what the CPU is executing) works for native processes and needs the container to have the SYS_PTRACE capability (Linux capabilities are fine-grained permissions the kernel can grant separately from full root access; SYS_PTRACE specifically is the one that lets one process inspect and attach to another) or run with elevated privileges to attach; py-spy profiles a running Python process from outside it without requiring code changes or a restart; Node exposes --prof (or attaching Chrome DevTools via --inspect) for CPU profiles and flame graphs; the Java Virtual Machine has JFR (Java Flight Recorder, a built-in low-overhead profiler) and async-profiler as a widely used external option. A flame graph from any of these should show whether CPU time concentrates in a small number of hot functions (a real optimization target) or is spread thinly across the whole call graph (more often a sign the bottleneck is elsewhere, such as garbage collection or lock contention).

Container-level: cgroups, quotas, and throttling

cgroups (Linux control groups, the kernel mechanism that limits and accounts for resource usage per group of processes) enforce a --cpus limit through the CFS (Completely Fair Scheduler, the Linux kernel's default CPU scheduler) bandwidth controller: a quota of CPU time allowed per fixed time period. When a container consumes its full quota before the period ends, the kernel throttles it, meaning it pauses scheduling that container's processes until the next period begins. This shows up as latency spikes and stalls, not as a smooth, sustained high-CPU line, which is exactly the "intermittent" pattern the question describes, and is easy to misread as "the code is doing something expensive sometimes" when the actual cause is quota exhaustion.

I verified this mechanism directly: starting a container with --cpus=0.1 running a tight busy loop, and after roughly three seconds, reading /sys/fs/cgroup/cpu.stat from inside the container showed:

usage_usec 388796
nr_periods 39
nr_throttled 39
throttled_usec 4148462

Every single one of the 39 scheduling periods observed was throttled (nr_periods equals nr_throttled), and the container spent about 4.15 seconds of that window paused by the scheduler (throttled_usec, in microseconds). /sys/fs/cgroup/cpu.max read back 10000 100000, meaning a 10,000 microsecond quota per 100,000 microsecond period, exactly the 0.1 CPU allocation requested. nr_throttled climbing alongside real request-latency spikes is the direct, unambiguous signal that the container's own quota, not its code, is the bottleneck.

Scheduling: shares versus quota, and pinning

--cpu-shares sets a relative weight (default 1024) that only matters when the host's CPUs are actually contended; a container with low shares on an otherwise idle host runs completely unrestricted, which is the most common misunderstanding of this flag. --cpus (a hard quota) applies its ceiling continuously regardless of host contention. --cpuset-cpus pins a container to specific physical cores, which avoids cross-core migration and cache-thrashing overhead when several containers compete for the same cores on a shared host (the classic noisy-neighbor pattern, where one busy container degrades its neighbours).

Trade-offs and pitfalls

Profiling application code first, without checking cpu.stat for throttling, wastes time chasing a code-level explanation for a scheduling-level problem, and any optimization found that way tends to be marginal, since the actual ceiling is the configured quota, not the algorithm. The opposite mistake, assuming every CPU spike is throttling without ever profiling, misses genuine application regressions (an accidentally quadratic code path, a newly introduced busy-wait) that raising the quota would only temporarily paper over before usage grows to fill the new ceiling too.

Direct answer

A container process should run as non-root because the container's process namespace boundary is not a security boundary by itself: if an attacker escapes the container (through a kernel vulnerability, a misconfigured mount, or a vulnerable dependency processing untrusted input), running as root inside the container often means they land as root on the host's view of that process too, and root inside a container frequently has more host-visible capability than people expect. For a service that parses untrusted input or handles sensitive data, that escape path is exactly the one you are trying to make as narrow as possible.

Dockerfile technique

Create a dedicated, low-privilege system user and group, then switch to it before the final CMD/ENTRYPOINT:

FROM python:3.12-slim
RUN addgroup --system app && adduser --system --ingroup app app
WORKDIR /app
COPY --chown=app:app . .
USER app
CMD ["python", "app.py"]

Everything before USER app (installing packages, writing files as part of the build) still runs as root, which is fine since that happens at build time, not against untrusted input; USER app switches the actual runtime process. --chown=app:app on the COPY avoids a separate chown layer and ensures the application files are actually readable by the user that will run them.

Binding a privileged port without running as root

Ports below 1024 are privileged on Linux by default, and a non-root process cannot bind one directly. The clean fix that does not require root at all is a Linux capability: grant the specific CAP_NET_BIND_SERVICE capability to the binary itself with setcap, rather than granting the whole process root:

RUN setcap 'cap_net_bind_service=+ep' /app
USER app

Verified directly: a small Go HTTP server built with this exact technique, running as the non-root app user, still bound to port 80 successfully and served a request:

$ docker exec nonroot-demo whoami
app
$ curl http://127.0.0.1:18080/
ok

(port 18080 on the host was mapped to port 80 in the container). The far more common real-world alternative, when you do not strictly need the container's internal process to bind a low port, is simply to have the application listen on an unprivileged port (like 8080) and let Docker's own port publishing do the remapping (-p 80:8080), since the privileged-port restriction only applies inside the container's own network namespace, not to the host-side port a container is published on.

Trade-offs and gotchas

setcap requires the capability tool (libcap2-bin on Debian-based images) to be present at build time, and the capability has to be reapplied if the binary is rebuilt or replaced later in the Dockerfile, since it is a filesystem attribute on that specific binary, not a property of the user. If you only need the container to appear on port 80 externally and are not otherwise constrained (no host networking, no orchestrator requirement to bind low ports internally), prefer the simpler port-remapping approach; reach for setcap when the application specifically must bind a privileged port inside its own namespace.

A container orchestrator (the system that decides where and how containers run in production, such as Kubernetes) usually layers its own version of this same control on top of the Dockerfile: a Kubernetes pod's securityContext.runAsNonRoot: true tells the orchestrator to refuse to start the container at all if the image's own configuration would otherwise run it as root, enforcing the requirement from outside the image in addition to, not instead of, the USER instruction inside it.