Container Image Management and Registry Questions

Focuses on the lifecycle of container images from build to runtime. Includes building reproducible images, multi stage builds, tagging strategies and semantic versioning, minimizing image size and attack surface, running as non root, and use of minimal base images. Covers pushing to and pulling from registries including public and private registries, authentication and access control, registry lifecycle policies, image caching and garbage collection, image signing and provenance, vulnerability scanning and remediation processes, integration with CI CD pipelines, and operational considerations such as storage costs, replication and geo distribution, and registry high availability.

HardTechnical

32 practiced

During a deployment window your private registry unexpectedly becomes read-only, causing new image pushes and some deployments to fail. As the SRE on-call, outline immediate triage steps, mitigations to resume deployments, communication plan for stakeholders, and long-term remediation and post-incident actions.

Sample Answer

Immediate triage (first 0–15 minutes)- Acknowledge alert, take ownership and set incident channel (e.g., #inc-registries-YYYY).- Quickly confirm scope: which registries (private, replicas), which services/deployments failing, and whether reads are impacted. Commands: - curl -I https://registry.example.com/v2/ (check HTTP status) - ssh to registry host(s) and check disk, mounts: df -h, mount, lsblk - docker login/push/pull tests from a bastion to reproduce - check orchestration errors: kubectl -n X get pods, kubectl describe deployment for ImagePullBackOff- Check recent changes: deploys, storage maintenance, ACL changes, certificate renewals, or quota events in cloud provider logs.

Short-term mitigations to resume deployments (15–60 minutes)- If registry is read-only due to filesystem remount or storage snapshot: attempt safe remount/read-write if storage healthy (sudo mount -o remount,rw /data) — only with owner approval.- If caused by storage quota/lifecycle policy: increase quota or free space (delete old GC-eligible images using a safe retention script).- If write ops blocked by a process/lock: restart registry service gracefully (systemctl restart registry / restart container) after ensuring no in-progress critical writes.- If registry cannot be restored quickly: failover to an alternate registry: - Promote a read-only mirror to writable (if supported). - Push new images to an emergency registry (e.g., staging-registry.example.com or cloud ECR/GCR) and update deployment imagePullSecrets and image references via a rapid, validated patch or ImagePolicy. - Use Kubernetes imagePullSecrets to allow pulls from temporary registry and perform rolling updates.- If only pushes fail but pulls work, prioritize redeploying using already published images; hold back any new-image-dependent features.

Communication plan (who, what, cadence)- Initial broadcast (within 15 min) to: Engineering, Product, Release Engineering, Security, Executives as needed. Include: incident summary, impacted services, immediate mitigation in progress, ETA for next update.- Status updates every 30 minutes until stable; switch to hourly once mitigations in place.- Provide actionable instructions for devs: stop pushing new images, use emergency registry if available, avoid triggering rollouts that build new images.- Post-resolution notification: confirm root cause, mitigations applied, and next steps for permanent fix and postmortem timing (within 72 hours).

Long-term remediation and post-incident- Root-cause analysis: collect logs, timeline, storage metrics, config changes, and runbooks followed.- Fixes: - Remove single points of failure: enable replicated registry with automatic failover, or multi-AZ storage. - Implement automated health checks and alerts for registry read-only states (e.g., write probe that attempts a small push). - Expand monitoring: disk, inode usage, storage backend errors, registry metrics (uploads failed, requests 4xx/5xx). - Harden CI/CD: automated fallback to mirrored registries and a documented emergency push workflow. - Improve image lifecycle management: automated GC, retention rules, and staging retention to prevent unexpected deletions or quota. - Access control/process fixes if human error caused it (change control, approvals).- Post-incident actions: - Run a blameless postmortem within 3–5 days with timeline, root cause, impact, decisions, and owners for action items. - Create/update runbook: step-by-step recovery for read-only registry, safe remount, failover to backup registry, and clear rollback criteria. - Track action items to completion with owners and deadlines; verify fixes with a game-day exercise.

Key considerations and safety- Avoid risky write operations (e.g., force remount) without understanding storage backend.- Preserve forensic data (logs, metrics, configs) before disruptive actions.- Prioritize restoring production reads and deployments before attempting risky writes.

This plan balances quick restoration with safety, clear communications, and durable fixes to prevent recurrence.

MediumTechnical

35 practiced

Explain the OCI image manifest and the image index (manifest list) structure, including config, layers, and annotations. Describe how you would use annotations to attach provenance or SBOM links and how you would programmatically query those fields from a registry.

Sample Answer

An OCI image manifest is a JSON document describing a single image variant: it references a config object (JSON with runtime metadata like entrypoint, env, diff IDs) and an ordered list of layer descriptors (blobs with mediaType, size, digest). An image index (manifest list) is a higher-level OCI index that points to multiple platform-specific manifests (each a descriptor), with optional annotations and platform fields (os/arch).

Key fields:- manifest: { schemaVersion, mediaType, config: {digest, size, mediaType}, layers: [ {digest,size,mediaType, annotations?} ], annotations? }- index: { schemaVersion, mediaType: application/vnd.oci.image.index.v1+json, manifests: [ {digest, size, mediaType, platform:{os,arch}, annotations?} ], annotations? }

Using annotations:- Attach SBOM/provenance by adding annotations either on the index (global to the multi-arch image) or on an individual manifest (per-platform): - e.g. "org.opencontainers.image.source", "io.cncf.notary.signature", or custom keys like "io.example.sbom": "https://sbom.example.com/artifact.cdx.json"- For signed provenance, point to a checksumed SBOM blob digest rather than an external URL.

Programmatic queries:- Use the OCI Distribution API: GET /v2/<name>/manifests/<reference> with Accept header for manifest or index media types. Response body is the manifest/index JSON; annotations are in the JSON.- Example with curl + jq:

bash

curl -s -H "Accept: application/vnd.oci.image.index.v1+json, application/vnd.oci.image.manifest.v1+json" \
  -u user:pass \
  "https://registry.example.com/v2/myrepo/myimage/manifests/latest" | jq '.annotations, .manifests[]?.annotations'

- Tools: skopeo inspect, crane manifest, oras/oci packages (Go) expose descriptors and annotations programmatically.- In Go (github.com/opencontainers/image-spec/specs-go/v1) you can Unmarshal into v1.Index/v1.Manifest and read Descriptor.Annotations.

Best practices:- Use stable annotation keys (reverse-domain), include digests for integrity, prefer storing SBOMs as registry blobs (referenced by digest) rather than fragile external URLs, and place annotations at the appropriate level (index vs manifest) depending on whether SBOM is global or platform-specific.

HardTechnical

31 practiced

Design an audit logging and compliance pipeline for a container registry that captures push/pull events, vulnerability scan results, signature verification, and admin actions. Include retention policies, tamper-resistance, searchable indexes, and integration points with SIEM and compliance reports.

Sample Answer

Requirements & constraints:- Capture events: image push/pull, vulnerability scan results, signature verification, admin actions (create/delete/permission changes)- Tamper-resistant, searchable, long-term retention for compliance (e.g., 7+ years), SIEM integration, report generation, low-latency for investigations

High-level architecture:- Registry emits structured audit events (JSON schema with timestamps, user/ID, IP, artifact digest, repo, action, result, trace_id) to a durable event bus (Kafka or Kinesis).- Stream processors (Flink/Beam or Kafka Streams) enrich events with metadata (vulnerability IDs, CVSS, signature status from Sigstore/cosign, user LDAP groups) and normalize schema.- Two sinks: 1. Fast index: OpenSearch (hot/warm nodes) for fast search/forensics, retention 90–365 days. 2. Immutable archive: Object store (S3) with Object Lock in Compliance mode + server-side encryption, events batched into daily/hourly gzipped ndjson and stored in partitioned prefixes (year=/month=/day=). Maintain manifest and checksums.- Append-only audit ledger: periodically hash batches and write Merkle-tree checkpoints to a signed, externally-sealed ledger (can be a blockchain-backed service or write checkpoint to an HSM/Vault-signed record + secondary storage) to detect tampering.

Tamper-resistance & integrity:- Transport: TLS + mutual auth for producers.- Signing: Each event signed at source (registry/service) using keys managed in KMS/HSM; stream processor verifies signature.- Immutable storage: S3 Object Lock WORM + strict IAM; restricted deletion via governance workflows.- Ledger: Merkle root checkpoints stored with timestamp in Vault/HSM and replicated to an independent read-only store for audits.- Audit trail of access: log all reads of archive and index queries into a separate access log stream.

Searchable indexes & retention:- OpenSearch schema with ingest pipelines for mappings (keyword fields for user, repo, digest; full-text for error messages; numeric for CVSS).- Hot nodes (0–30d) for incident response; warm (30–90d) for investigations; cold long-term snapshots taken and exported to archive.- Retention policy: hot/warm/cold lifecycle automated; archive retention per policy (e.g., 7 years for PCI/industrial clients), with legal-hold support.

SIEM & Alerting:- Real-time alerts via stream rules (e.g., large number of pulls from unusual IP, failed signature verifications, high-severity CVEs) pushed to SIEM (Splunk/QRadar/Elastic SIEM) via connectors or syslog forwarding.- Enrich SIEM events with context (owner, SLO impact).- Use detection rules, threat intel feeds, and automated playbooks (runbook automation) for common remediations (quarantine image, revoke keys).

Compliance reporting & audits:- Scheduled reports generated from archive + index: tamper-proof manifests, digest timelines, signer attestations, vulnerability history per artifact, admin action timelines.- Provide export formats: CSV, PDF with signed manifest and cryptographic proof (Merkle inclusion proof).- Support on-demand auditors access via time-limited read-only S3 presigned URLs and audit-viewer UI (reads are logged).

Operational considerations:- SLOs: event delivery to index within X seconds (e.g., 30s), archive durability 11 9s (S3), search latency targets.- Scaling: partition key on repo/name for Kafka; OpenSearch index per time window; archive throughput via multipart uploads.- Security: RBAC for access to indexes/archives, KMS/HSM for signing keys, periodic key rotation and key compromise playbook.- Testing & validation: periodic replay tests, integrity checks (recompute hashes, verify signatures), red-team tamper attempts.- Cost & trade-offs: keep recent data in OpenSearch for rapid response, archive large volumes cheaply; ledger/ledger-signing introduces extra complexity but required for strong non-repudiation.

Example event (JSON):{ "ts":"2025-01-02T12:34:56Z","actor":"alice@org","action":"push","repo":"org/app","digest":"sha256:...","ip":"1.2.3.4","vulns":[{"id":"CVE-...","cvss":9.1}],"signature":{"status":"verified","signer":"cosign:alice"},"event_id":"uuid","signature_proof":"base64..."}

This design balances fast investigability, tamper-evidence, long-term retention, and SIEM/compliance integration while remaining operationally scalable for an SRE-run platform.

MediumSystem Design

29 practiced

Design a registry replication strategy to replicate images across three continents to minimize pull latency. Discuss push vs pull replication models, tag conflict resolution, eventual consistency implications, and network considerations for large blob transfers.

Sample Answer

Requirements and constraints:- Replicate container images across three continents (low read latency for pulls), tolerate network partitions, reasonable storage cost, support CI/CD pushes, and handle large blobs efficiently.High-level architecture:- Central origin registry (primary) + three regional read-replicas (one per continent).- Control plane service for metadata (manifests/tags) replication and a blob-store layer (object storage) with CDN-like caching for layers.

Push vs Pull replication:- Push (active): when image pushed to origin, asynchronously push manifests and blobs to regional replicas. Pros: reduces first-pull latency; cons: spikes outbound bandwidth and storage duplication.- Pull (lazy): replicas fetch blobs/manifests on first regional pull. Pros: saves bandwidth/storage for unpopular images; cons: first-pull latency spike.- Hybrid: push manifests and small/common base layers proactively; lazy-pull large/rare blobs. Use popularity metrics to promote lazy->push.

Tag conflict resolution:- Use immutable digests for image content; tags are pointers. For same tag updated concurrently, implement causal ordering with per-repo monotonic sequence (vector clocks or origin-timestamp + origin-ID). Prefer last-writer-wins with audit logs, but expose conflicts to CI (fail-fast) and allow manual reconciliation or automated rollback to previous digest.

Consistency and eventual consistency:- Metadata (tags/manifests) replicate asynchronously — eventual consistency acceptable for most pulls but SLOs must define staleness window. Provide read-after-write guarantee within origin region via synchronous tag update there; cross-region reads may return older tag until replication completes. Offer optional synchronous cross-region tag propagation for critical releases (feature flag).

Network considerations for large blobs:- Use object storage with cross-region replication capability and CDN edge caches for blobs. Deduplicate by digest across regions to avoid duplicate uploads. Implement chunked uploads with resumable transfers, parallel ranged downloads, and checksum verification. Throttle/concurrency limits and backpressure to protect WAN links. Schedule bulk pre-warm transfers off-peak and use compression/delta/OCI layer reuse to minimize transfer size.

Operational and SRE concerns:- Monitor replication lag (manifests & blob availability), bandwidth utilization, failed transfers, and cache hit rates. Alert on elevated first-pull latency or staleness beyond SLO. Automate re-syncs, exponential backoff for failures, and provide tools for emergency global invalidation/rollbacks.

EasyTechnical

31 practiced

List practical techniques to minimize container image size and attack surface: multi-stage builds, removing package caches, using --no-install-recommends, distroless/scratch, stripping binaries, and minimizing layers. For each technique give a rough estimate of the potential size savings and trade-offs.

Unlock Full Question Bank

Get access to hundreds of Container Image Management and Registry interview questions and detailed answers.

Join thousands of developers preparing for their dream job.