Containerization and Docker Fundamentals Questions

To build a minimal, reproducible runtime image for a Java 17 Spring Boot microservice using Maven, use a multi-stage Dockerfile: first stage builds the artifact using a Maven image; second stage copies only the JAR into a slim OpenJDK runtime and sets JVM tuning environment variables suitable for memory-limited containers.

# build stage: use Maven with JDK 17 to compile and package
FROM maven:3.9.4-eclipse-temurin-17 AS build
WORKDIR /app
# cache Maven dependencies by copying only pom first
COPY pom.xml .
RUN mvn -B -f pom.xml dependency:go-offline
# copy sources and build
COPY src ./src
RUN mvn -B -DskipTests package

# runtime stage: use a minimal JRE image (distroless or slim)
FROM eclipse-temurin:17-jre-alpine
# create non-root user
RUN addgroup -S app && adduser -S -G app app
WORKDIR /app
# copy the fat/boot jar (adjust path if using spring-boot:repackage)
COPY --from=build /app/target/*.jar app.jar
RUN chown app:app app.jar
USER app

# JVM tuning for container limits
ENV JAVA_TOOL_OPTIONS="-XX:+UseContainerSupport -XX:MaxRAMPercentage=75.0 -XX:MinRAMPercentage=25.0 -XX:MaxMetaspaceSize=256m -XX:+UseG1GC -Djava.security.egd=file:/dev/./urandom"

EXPOSE 8080
ENTRYPOINT ["sh","-c","exec java $JAVA_TOOL_OPTIONS -jar /app/app.jar"]

Why multi-stage builds help:

• Size: build tools and caches stay in the build stage and are not copied, producing a much smaller runtime image.
• Reproducibility: fixed base images and clear build steps produce deterministic artifacts; caching dependency step improves repeatable builds.
• Security: reduced attack surface by excluding compilers and package managers from final image.

JVM tuning notes:

• UseContainerSupport enables container-aware heap sizing (Java 10+).
• MaxRAMPercentage controls heap relative to container memory; set to 75% to leave headroom for metaspace/native.
• MinRAMPercentage ensures some heap even with very small limits.
• G1GC is a good default for latency and throughput balance.
  Adjust percentages and metaspace for your pipeline memory constraints and monitor with metrics (jvm memory/GC) in production.

Approach: build a small, secure image that optimizes layer caching for Python deps, runs as non-root, uses environment variables for config, and exposes a HEALTHCHECK endpoint the ETL app provides (e.g., /health or a lightweight script).

Dockerfile:

# syntax=docker/dockerfile:1.4
FROM python:3.11-slim AS base
ENV PYTHONDONTWRITEBYTECODE=1 \
    PYTHONUNBUFFERED=1

# Create non-root user
RUN useradd --create-home --shell /bin/bash appuser

WORKDIR /app

# Install build deps only if needed (for some wheels)
RUN apt-get update && apt-get install -y --no-install-recommends build-essential gcc \
 && rm -rf /var/lib/apt/lists/*

# Cache dependencies: copy only requirements first
COPY requirements.txt .

# Use pip cache mount when supported for faster CI builds
# Install deps as appuser into system site-packages (or consider venv)
RUN --mount=type=cache,target=/root/.cache/pip \
    pip install --no-cache-dir -r requirements.txt

# Copy application code
COPY . .

# Ensure app files are owned by non-root user
RUN chown -R appuser:appuser /app

USER appuser

# Entrypoint and default command (app should read env vars)
ENV PORT=8080
EXPOSE ${PORT}

HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
  CMD curl -f http://127.0.0.1:${PORT}/health || exit 1

CMD ["python", "etl_service.py"]

Key points / reasoning:

• python:3.11-slim keeps image small. Creating appuser reduces risk if container compromised. Copying requirements.txt before app sources leverages layer caching so rebuilds don't reinstall deps if only code changes. --mount=type=cache improves CI build speed. HEALTHCHECK validates runtime readiness. Use env vars for DB/S3 credentials and endpoints; never bake secrets into image (use secrets manager/CI secrets).

Testing locally:

• Unit tests: run pytest on host and in a test container:
  docker build -t etl:test . && docker run --rm etl:test pytest
• Integration: use docker-compose with Postgres and MinIO (S3-compatible) services and environment variables; run the container and simulate an ETL run, verify rows in Postgres and objects in MinIO.
• Example docker-compose: include postgres, minio, and the etl service with env vars pointing to services; use healthcheck dependencies to wait.

CI pipeline (e.g., GitHub Actions):

• Steps: checkout, lint (flake8), build image with cache, run unit tests in build container, run integration tests using services (postgres/minio) via services: postgres, minio; run security scans (trivy/hadolint). Use secrets for DB/S3 creds; push image only on main or tag.

Edge considerations:

• Use IAM roles or ephemeral credentials for production S3 access; rotate secrets. Limit installed packages to reduce CVEs. For high-scale ETL consider splitting worker and orchestration, and use multi-stage builds to remove build tools from final image.