DevOps Engineer (Entry Level) Interview Preparation Guide - Apple

Direct answer

Use flock (from util-linux) on a lock file: open it on a file descriptor, then take an exclusive lock on that descriptor. The kernel drops the lock when the process dies, however it dies (the lock is tied to the open file, which closes with the process), so there is no stale lock to clean up. For a job that runs every minute, use the non-blocking form flock -n and skip the run if the lock is held, instead of queueing runs behind a slow one. Use a timeout (flock -w SECONDS) when a short wait is acceptable. On a host without flock, mkdir is an atomic lock, paired with an EXIT trap and a PID file so a stale lock from a kill -9 can usually be detected (best effort, see the pitfalls).

How the lock works

• A file descriptor (fd) is a small number a process uses to refer to a file it has open. 0, 1 and 2 are standard input, output and error; 9 is just an unused number picked for the lock.
• exec 9>"$lock" opens the lock file for writing (creating it) and attaches it to fd 9 for the rest of the script. flock -n 9 then asks the kernel (the core of the operating system) for an exclusive lock on the open file behind fd 9, and fails at once if another process already holds one. flock comes from util-linux, the standard package of basic Linux command-line tools.
• The lock belongs to the open file, not to the script. It is released when every descriptor pointing at that open file is closed, and the kernel closes a process's descriptors when the process dies, however it dies. That is why a crash leaves no stale lock.
• The caveat shown later is the same rule seen from the other side: a child process inherits a copy of fd 9, so if the parent is killed while the child lives on, one descriptor is still open and the lock is still held. 9>&- closes fd 9 for that one command, so the child holds no copy.
• Other terms: a process group is the set of processes started together that can be signalled as one; a watermark is a saved marker (a timestamp or id) of how far processing has got, so the next run resumes from it.

Blocking, non-blocking, and timeout

• Non-blocking (flock -n): fails immediately with exit status 1 if the lock is held. Right for a frequent cron job: if run N is still going, run N+1 has nothing to add, and queuing would build a backlog.
• Blocking (default): waits forever. Rarely right for cron, since waiters pile up until the host is out of processes or memory.
• Blocking with timeout (flock -w 30): waits up to 30 seconds, then exits 1. Useful when the holder is expected to finish soon and the new run should not be lost, for example an hourly job that may overlap with a quick one.
• Whatever you pick, log the skip. A job that skips silently every minute for a day looks like a job that is working.

Worked example, with flock

#!/usr/bin/env bash
set -euo pipefail
lock=${LOCKFILE:-/tmp/report.lock}
exec 9>"$lock"                 # open (create) the lock file on fd 9; closing the fd releases the lock
if ! flock -n 9; then          # -n: do not wait; another run holds it
    echo "[$$] previous run still going, skipping" >&2
    exit 0
fi
echo "[$$] got lock, working"
sleep "${WORK:-2}" 9>&-    # children must not inherit the lock fd
echo "[$$] done"

Reading order of the driver: (1) five copies start together, so one should win and four skip; (2) the lock is free again afterwards; (3) a waiter with a 1 second timeout gives up while a 3 second run holds the lock; (4) a child that inherits fd 9 keeps the lock after kill -9; (5) the same scenario with 9>&- releases it; (6) the mkdir version repeats the five-way race and then shows cleanup on SIGTERM and stale-lock detection after kill -9. The driver below starts five copies at once, tests the timeout form, demonstrates the descriptor trap described afterwards, and then exercises the mkdir version from the next section. Its full output follows the driver.

echo '=== flock: five starts at once'
rm -f /tmp/report.lock
for i in 1 2 3 4 5; do bash job-flock.sh 2>&1 & done | cat; wait
echo '=== flock: lock free again after the run'
WORK=0 bash job-flock.sh
echo '=== flock: blocking acquire with a 1 second timeout while a 3 second run holds the lock'
(WORK=3 bash job-flock.sh &) ; sleep 0.5
exec 8>/tmp/report.lock; flock -w 1 8; echo "waiter exit=$? (1 means timed out)"; exec 8>&-
sleep 3
echo '=== a child that inherits fd 9 keeps the lock after kill -9'
cat > inherits.sh <<'EOF'
exec 9>/tmp/o.lock; flock -n 9 || { echo skipped; exit 0; }; sleep 30
EOF
bash inherits.sh & p=$!; sleep 0.5; kill -9 $p; wait $p 2>/dev/null; bash inherits.sh
echo '=== job-flock.sh closes fd 9 for its child, so kill -9 releases the lock'
WORK=30 bash job-flock.sh & p=$!; sleep 0.5; kill -9 $p; wait $p 2>/dev/null
WORK=0 bash job-flock.sh
echo '=== mkdir: five starts at once'
rm -rf /tmp/report.lockdir
for i in 1 2 3 4 5; do sh job-mkdir.sh 2>&1 & done | cat; wait
ls -d /tmp/report.lockdir 2>&1 | sed 's/^/after: /'
echo '=== mkdir: SIGTERM cleans up'
WORK=30 sh job-mkdir.sh & p=$!; sleep 0.5; kill -TERM $p; wait $p 2>/dev/null; ls -d /tmp/report.lockdir 2>&1 | sed 's/^/after TERM: /'
echo '=== mkdir: kill -9 leaves a stale lock that the next run detects'
WORK=30 sh job-mkdir.sh & p=$!; sleep 0.5; kill -9 $p; wait $p 2>/dev/null
ls -d /tmp/report.lockdir | sed 's/^/after KILL: /'
WORK=0 sh job-mkdir.sh

=== flock: five starts at once
[11] got lock, working
[10] previous run still going, skipping
[12] previous run still going, skipping
[13] previous run still going, skipping
[14] previous run still going, skipping
[11] done
=== flock: lock free again after the run
[21] got lock, working
[21] done
=== flock: blocking acquire with a 1 second timeout while a 3 second run holds the lock
[25] got lock, working
waiter exit=1 (1 means timed out)
[25] done
=== a child that inherits fd 9 keeps the lock after kill -9
skipped
=== job-flock.sh closes fd 9 for its child, so kill -9 releases the lock
[38] got lock, working
[42] got lock, working
[42] done
=== mkdir: five starts at once
[50] got lock, working
[48] previous run still going, skipping
[51] previous run still going, skipping
[52] previous run still going, skipping
[49] previous run still going, skipping
[50] done
after: ls: cannot access '/tmp/report.lockdir': No such file or directory
=== mkdir: SIGTERM cleans up
[70] got lock, working
after TERM: ls: cannot access '/tmp/report.lockdir': No such file or directory
=== mkdir: kill -9 leaves a stale lock that the next run detects
[77] got lock, working
after KILL: /tmp/report.lockdir
[83] removing stale lock of dead pid 77
[83] got lock, working
[83] done

The pids differ on every run, and so does the order in which the four skipped runs print. The shape does not: one run gets the lock and four skip, and the lock is free again after the run. The waiter with -w 1 gave up with exit 1 while a 3 second run held the lock. After kill -9 on the holder, the next run got the lock.

One trap found while testing: sleep is a stand-in for the real work, and a child process inherits the lock's file descriptor. If the holder is killed but a child survives, the child still holds the lock and the next run skips silently (the skipped line under "a child that inherits fd 9"). The fix is to close the descriptor for children with 9>&- (as job-flock.sh does), or to run the whole job under flock -n lockfile command ... and kill the process group.

In crontab you can also write it on one line, with the lock file in a directory the job's user owns: * * * * * flock -n /home/report/report.lock /usr/local/bin/report.sh.

Without flock: the mkdir lock

The essential idea is the first sentence of the explanation after the script: mkdir is an atomic, one-winner gate. The stale-lock recovery is a best-effort extra for the kill -9 case and has the races listed under the pitfalls.

#!/bin/sh
# Lock without flock: mkdir is atomic, so exactly one caller can create the directory.
lockdir=${LOCKDIR:-/tmp/report.lockdir}

acquire() {
    if mkdir "$lockdir" 2>/dev/null; then
        echo $$ > "$lockdir/pid"
        return 0
    fi
    # Lock exists. If its owner is gone (kill -9, power loss), the lock is stale.
    owner=$(cat "$lockdir/pid" 2>/dev/null || true)
    if [ -n "$owner" ] && ! kill -0 "$owner" 2>/dev/null; then
        echo "[$$] removing stale lock of dead pid $owner" >&2
        rm -rf "$lockdir"
        mkdir "$lockdir" 2>/dev/null && { echo $$ > "$lockdir/pid"; return 0; }
    fi
    return 1
}

if ! acquire; then
    echo "[$$] previous run still going, skipping" >&2
    exit 0
fi
trap 'rm -rf "$lockdir"' EXIT
trap 'exit 129' HUP
trap 'exit 130' INT
trap 'exit 143' TERM

echo "[$$] got lock, working"
sleep "${WORK:-2}"
echo "[$$] done"

mkdir either creates the directory or fails, in a single step, so two processes cannot both succeed. The EXIT trap removes the directory on normal exit, and the signal traps make it also run for HUP, INT and TERM. A kill -9 leaves the directory behind, so the script records its PID and a later run asks kill -0 PID (signal 0 sends nothing but reports whether the process exists) to see if the owner is gone. In the output above, the SIGTERM case left no directory behind, the kill -9 case left /tmp/report.lockdir, and the next run printed removing stale lock of dead pid followed by the dead PID, before taking the lock.

Trade-offs and pitfalls

• The stale-lock recovery in the mkdir version has a small race: two runs may both judge the lock stale, and one can delete the lock the other just took. Use flock where it exists and keep mkdir for minimal hosts. A reused PID can also make a dead owner look alive, so a stale lock then waits for a manual fix.
• Put the lock file in a directory that only the job's user can write to, such as the user's home directory or a systemd RuntimeDirectory. /tmp is world-writable, and so is /run/lock on Debian and Ubuntu (ls -ld /run/lock shows drwxrwxrwt in the Debian and Ubuntu images tested; the alpine image shows drwxr-xr-x). In a world-writable directory another user can pre-create the lock file or directory and make every run skip.
• flock locks on network file systems (NFS, Network File System) depend on server and client support, so keep the lock file on local disk. Do not use it to coordinate several hosts: use a lock service or a database lock for that.
• A lock directory with no pid file is never judged stale: if the process dies between mkdir and writing the PID (tested by creating an empty lock directory), every later run skips until someone removes it by hand.
• Also bound the job's own runtime (timeout 55 job.sh), so a hung run does not hold the lock for hours.
• A lock prevents overlap; it does not make missed runs up. If the job must process every minute's data, track a watermark instead of relying on the schedule.
• macOS has no flock by default. That does not affect Linux servers, but a script developed on a laptop may need the mkdir fallback there.

Direct answer
Write a resource block of the correct type declaring only the arguments you intend to manage, run terraform import (or a declarative import block) to attach the real resource's ID to that address, then run terraform plan repeatedly, reconciling your HCL against what it reports, until plan shows no diff. Only then does apply touch anything, and by that point it's a no-op. Nothing about the live VPC or database changes during any of this, because import only writes state and plan never mutates anything, so the "without downtime" requirement is really just discipline about not applying until plan is clean.

Structured elaboration

1. Inspect the real resource before writing HCL
Don't write the resource block from memory. Look up the actual current attributes (via the console, CLI, or the provider's data source) so your first draft doesn't propose spurious changes the moment you import.

2. Write the minimal matching resource block
Keep it to the arguments you actually plan to manage going forward; leave out fields you expect the provider to compute (endpoints, generated ARNs) rather than guessing values for them.

resource "aws_db_instance" "example" {
  identifier = "my-db-identifier"
  # do not add attributes you expect AWS to compute (endpoint, address)
}

3. Import

terraform import aws_db_instance.example my-db-identifier

As of Terraform 1.5+ you can express the same step declaratively with an import block plus terraform plan -generate-config-out=generated.tf, which scaffolds a starting resource block for you instead of hand-writing one, useful when importing many resources at once.

4. Reconcile computed and drifted attributes
Run terraform plan. Anything present in the real resource but missing or different in your HCL shows up as a proposed change. For attributes the provider computes and you don't want Terraform fighting over (an RDS endpoint, a final snapshot identifier), either add the real value explicitly if you want to manage it, or mark it in a lifecycle block so Terraform stops treating drift there as something to fix.

lifecycle {
  ignore_changes = [endpoint, address]
}

5. Verify, then only apply if you mean to change something
terraform state show aws_db_instance.example confirms the mapped ID and attributes. Keep iterating on plan until it's clean. If you genuinely want to change a value going forward, that apply is a deliberate, reviewed change, not an accidental side effect of the import.

Nested/child resources aren't imported automatically
A hand-built VPC or database typically has dependent pieces, a DB subnet group, associated security groups, route tables, that each need their own import call. Importing the parent resource doesn't pull its children into management for you.

Worked example
Importing a hand-built RDS instance end to end:

terraform import aws_db_instance.example my-db-identifier
terraform state show aws_db_instance.example
terraform plan

A first plan on a genuinely hand-built instance will usually show a handful of proposed changes on computed fields like endpoint, address, and tags_all, since those were never in your HCL to begin with. You add ignore_changes for the ones you don't want to manage and explicit values for the ones you do, then re-run plan until it's a clean no-op, which is your confirmation that apply is now safe.

Trade-offs & pitfalls
Skipping the "inspect the real resource first" step is the most common way this goes wrong: you import, run plan, and discover it wants to replace the DB instance's engine or storage_type because you guessed a value instead of reading the real one, right after the whole point was to avoid touching it. Always back up state before starting (terraform state pull > backup.tfstate), and where possible rehearse the import against a non-production copy of the resource type first. Bulk-importing dozens of hand-built resources one CLI call at a time doesn't scale, that's what the 1.5+ import block with -generate-config-out is for.

The core risk in this migration is not containerization itself, it is state: a monolith on VMs with local disk and in-process sessions has state living in three places, local disk, in-memory sessions, and the database, that Kubernetes' pod model does not preserve across restarts, rescheduling, or rolling updates. The plan externalizes each of those in turn, in order of risk, before extracting pieces of the monolith into separate services with the strangler pattern, so the system never depends on a single big-bang cutover.

1. Containerize the monolith as-is first

Build an image for the existing Java monolith with externalized configuration (environment variables, ConfigMaps and Secrets instead of files baked into the VM), add liveness and readiness probes so Kubernetes can tell a hung JVM (the Java Virtual Machine, the runtime the monolith executes in) from a healthy one, and handle SIGTERM correctly (the termination signal Kubernetes sends a container before killing it): drain in-flight requests before exit, matching the JVM's shutdown hook to Kubernetes' terminationGracePeriodSeconds so a rolling update does not drop in-flight traffic. Get this version running and stable on Kubernetes before touching architecture.

2. Externalize local disk state

Kubernetes pods are ephemeral and can be rescheduled to a different node at any time, so anything the monolith wrote to local disk and expected to still be there tomorrow has to move:

• Transient, pod-lifetime-only files: an EmptyDir volume, deleted with the pod, is fine.
• Durable files the application actually needs to persist: a networked or object store. Blob-shaped data (uploads, reports) moves to an object store; anything requiring POSIX file semantics moves to a ReadWriteMany-capable networked filesystem.
• Migrate existing files with a one-time copy job, verify with checksums, and cut reads and writes over behind a feature flag so a bad migration can be reverted without a second migration.

3. Externalize session state

The same ephemeral-pod problem applies to in-process HTTP sessions: a session held only in the JVM's memory on one pod disappears the moment that pod is rescheduled, evicted, or replaced during a rolling update, which shows up to a user as an unexplained logout. The fix at the workload-design level is to externalize sessions to an external store, Redis is the common choice, so any pod can serve any request. Routing a user's requests back to the same pod is a different, ingress-layer session-affinity technique with its own trade-offs; it does not solve the underlying problem that a rescheduled pod still loses in-process state, so it is not a substitute for externalizing it.

4. Database migration strategy

Move the database carefully, since it is both the hardest state to externalize and the one most likely to be gotten wrong:

• Add logical replication (change-data-capture tooling such as Debezium, or the database's native replication) from the source to the target so the target stays current while the application still writes to the source.
• Use the expand-contract pattern for schema changes so old and new code can both run correctly during the transition: expand (add new columns or tables without touching old ones), migrate and backfill data, dual-write or dual-read as needed, then only after the new code path is validated in production, contract (stop writing the old shape, then drop it).
• Cut writes over in a single, short, controlled window once replication lag is at or near zero and read paths have already been validated against the target; keep the source writable-but-idle for a defined rollback window before decommissioning it.

5. Incremental rollout: the strangler pattern

Rather than extracting everything at once, put a facade, typically at the ingress layer, in front of the monolith that can route specific paths to a newly extracted service while everything else still goes to the monolith unchanged. Extract modules with low fan-in first, not the highest-traffic module first, because fan-in (how many other parts of the system call into a module) is what determines how much of the system breaks if the extraction has a bug, independent of how much traffic the module carries.

Worked example: choosing extraction order

Say three candidate modules have the following measured properties:

Using a simple priority heuristic of request share divided by fan-in, favoring modules that matter but are not deeply entangled:

$\text{Auth: } \frac{15}{2} = 7.5$

$\text{Reporting: } \frac{5}{1} = 5$

$\text{Checkout: } \frac{40}{10} = 4$

Auth extracts first (highest score), Reporting second, and Checkout last, despite carrying the most traffic, because its fan-in of 10 means a bug in the extraction has ten times the blast radius of Reporting's. This is the general shape of the reasoning a strangler-pattern plan should show, not a fixed formula: traffic volume alone is the wrong axis to sequence on.

6. Rollback strategy

Rollback needs to work at three independent layers, because a single "revert" button does not exist across all of them:

• Code: Kubernetes Deployments keep revision history, so kubectl rollout undo reverts to the previous ReplicaSet; tune maxSurge and maxUnavailable so a bad rollout is caught by readiness probes before it receives significant traffic.
• Data: because schema changes were expand-contract, the previous code version still works against the current schema right up until the contract step, so a code rollback does not also force an emergency schema rollback.
• Extracted services: keep the facade able to route a given path back to the monolith if the newly extracted service misbehaves, for as long as the monolith still contains that functionality; do not delete monolith code paths the moment a service is extracted, retire them on a delay after the extraction is proven.

Primary risks and mitigations

Direct answer

When legal or compliance flags a possible regulatory problem on something about to ship, that flag is new information, not an attack on the project. The first move is to separate the specific risk from the whole feature: find out exactly what triggers the concern, then look for a way to ship everything outside that blast radius (the specific data, users, or markets the flagged concern actually touches) while the risky piece gets handled properly. Treating the flag as either a full block to fight or a formality to route around are both weak answers; the senior move is to make the freeze as small as the actual risk.

Structured elaboration

1. Turn the flag into a scoped, written finding

Ask for the specific clause or regulation, the specific data flow or behavior it applies to, and which markets or user segments are affected. A flag that sounds like 'this violates a regulation' often narrows down to 'this one data field, in these two markets.' Until that scoping happens, nobody can reason about mitigation, they can only argue about the abstract freeze.

2. Sort what's actually blocked from what's just slow

Once scoped, most flags fall into three buckets: genuinely unsafe to ship anywhere (rare, but real, treat it as a hard stop); unsafe in specific markets or for specific data (the common case, often scoped out with a flag or market-level rule); or unsafe as currently designed but fixable with a smaller change than a full freeze (needs a scoped rework, not a blanket delay).

3. Bring a mitigation, not just a constraint

Offer a concrete option: disable the flagged behavior for the affected markets, gate it behind a feature flag (a toggle that turns a piece of functionality on or off without a new deployment), or ship a version that omits the specific data flow while the rest proceeds. This turns the conversation from 'can we go or not' into 'does this mitigation satisfy the concern,' which moves much faster.

4. Get joint, written sign-off before proceeding

Both the business owner and compliance need to agree in writing on what shipped, what did not, the remaining risk, and who owns closing it. This protects everyone if the interpretation is questioned later and prevents the same argument from recurring next release.

5. If a real freeze can't be avoided, negotiate the timeline explicitly

Sometimes there is no safe scoped path and the freeze has to hold for the affected piece. Here the negotiation shifts to: what's the minimum change needed to clear the concern, who is assigned to it, and can the review be fast-tracked with a dedicated reviewer instead of sitting in a general queue. A freeze with a committed, shrinking timeline is a very different conversation from an open-ended one.

Worked example

A team is about to ship a feature that logs a new field for product analytics, and legal flags that collecting that field may violate a data-protection rule in one region. Scoping the flag shows the issue is narrow: one field, one region. Instead of freezing the whole release, the team ships everywhere else immediately, and for the flagged region ships the same feature with that one field's collection disabled behind a config switch. Legal signs off on the scoped version in writing. The team opens a follow-up item, with an owner and a target date, to redesign how that field is collected (for example, aggregating it instead of storing it per user), so the region isn't stuck without the feature indefinitely.

Trade-offs and pitfalls

• Treating every compliance flag as either a full block or a nuisance to route around is the most common mistake here; both extremes erode trust with the compliance function over time.
• Scoped mitigations (flags, market gating, field exclusions) are good short-term tools but can quietly become permanent if nobody owns the follow-up fix. The sign-off should name an owner and a date, not just describe a workaround.
• Escalating past compliance to force a ship date, without addressing the underlying concern, tends to resurface later as a bigger problem: a real violation or a regulator inquiry. Speed gained by skipping the process rarely survives contact with the risk it was protecting against.
• The strongest signal of seniority isn't how fast the team got to yes, it's whether the final decision is something both sides would still defend the same way months later.

Direct answer

/etc/resolv.conf tells the system's name resolution machinery which DNS servers to query and in what order, via nameserver lines, plus optional search domains to try appending to unqualified hostnames. On a modern Linux host that file is rarely hand-maintained directly: it is usually generated and kept in sync by whichever layer is actually managing DNS underneath, most commonly systemd-resolved (the DNS-resolving component of systemd, the init system and service manager most modern Linux distributions use; it runs its own local stub resolver, typically listening on 127.0.0.53, and often makes /etc/resolv.conf a symlink pointing at a file it generates) or NetworkManager, sometimes with dnsmasq in front as a small local caching layer.

The resolution flow, layer by layer

1. An application calls a resolution function from glibc (the standard C library most Linux programs are built against), which consults NSS, Name Service Switch: a configurable dispatch mechanism (/etc/nsswitch.conf) that decides, for a given lookup type, which sources to check and in what order. The hosts: line in that file, something like hosts: files dns, is what makes /etc/hosts get checked before DNS at all, for every application on the system, without each program needing its own logic for that.
2. If DNS is reached, glibc's resolver reads /etc/resolv.conf for which server(s) to query.
3. On a host running systemd-resolved, /etc/resolv.conf commonly just points at 127.0.0.53, systemd-resolved's own stub listener, rather than at the real upstream DNS servers directly. systemd-resolved itself holds the actual upstream server list (learned from DHCP, static configuration, or per-interface settings) and does its own caching, so the file glibc reads and the servers actually queried over the network are two different things on such a host.
4. NetworkManager, when it manages DNS, can either write /etc/resolv.conf directly with the servers it learned from DHCP or static configuration, or hand the job off to systemd-resolved (a mode controlled by NetworkManager's own dns= setting), depending on how the distribution configured it.
5. dnsmasq, where present, is a small lightweight resolver and DHCP server often used as an additional local caching layer in front of upstream DNS, either standalone or as NetworkManager's local caching backend on some distributions.

Changing DNS, temporarily versus persistently

• Temporary (until the next reconnect, reboot, or manager restart):
  • resolvectl dns eth0 1.1.1.1 on a systemd-resolved host (the modern command; systemd-resolve is the older, now-deprecated name for essentially the same functionality).
  • Editing /etc/resolv.conf directly works immediately for any new resolution, but only until whatever manages that file (systemd-resolved, NetworkManager) next regenerates it and overwrites your edit, which can happen at any reconnect or restart with no warning.
• Persistent:
  • Netplan (Ubuntu): add a nameservers: block under the relevant interface in the YAML config, then netplan apply.
  • NetworkManager: nmcli con mod <connection-name> ipv4.dns "1.1.1.1 8.8.8.8" followed by nmcli con up <connection-name> to apply it.
  • systemd-networkd: add a DNS= line under [Network] in the relevant .network file, then systemctl restart systemd-networkd.

Trade-offs and pitfalls

• The most common confusion is editing /etc/resolv.conf by hand on a host running systemd-resolved or NetworkManager, seeing it work, and then being surprised it reverts on its own later; check whether the file is a symlink (ls -l /etc/resolv.conf) before assuming a manual edit is safe or durable.
• resolvectl status shows you the actual per-interface DNS configuration systemd-resolved is using, which can differ from what a bare cat /etc/resolv.conf shows when the stub resolver is in play; when DNS behavior looks wrong, check resolvectl status, not just the file, or you can chase a mismatch that does not actually explain the resolver's real behavior.
• search domains in resolv.conf can cause surprising, hard-to-debug behavior where an unqualified hostname silently resolves against the wrong domain (or leaks an internal hostname query to an external DNS server it should never have reached), which is a real concern worth checking for on hosts that handle sensitive internal name lookups.

Direct answer
Rolling back a Kubernetes deployment with immutable image tags is kubectl rollout undo, which redeploys the previous ReplicaSet's pod template from Kubernetes' retained revision history; verification means confirming the rollout actually completed and the resulting pods are healthy, not just that the command returned success.

Structured elaboration and worked example

# 1. Confirm the current rollout history and identify the target revision (optional, but good practice)
kubectl rollout history deployment/checkout-api

# 2. Trigger the rollback to the immediately previous revision
kubectl rollout undo deployment/checkout-api

# 3. Watch it complete (this blocks until the rollout finishes or times out)
kubectl rollout status deployment/checkout-api --timeout=120s

# 4. Confirm the pods are actually on the expected previous image tag
kubectl get pods -l app=checkout-api -o jsonpath='{.items[*].spec.containers[*].image}'

# 5. Confirm readiness: all pods Running and Ready, not just Running
kubectl get pods -l app=checkout-api

# 6. Application-level health confirmation beyond kubectl's view of pod state
curl -sf https://checkout-api.internal/healthz

Why each verification step matters: rollout status blocking until completion (rather than assuming success the instant undo returns) catches a rollback that's stuck (for example, the previous image was garbage-collected from the registry and can't be pulled, which undo will happily accept as a command but which then fails to actually schedule). Checking the ACTUAL image tag on running pods, not just trusting the command succeeded, catches a mismatch between what you intended and what's actually running. The final application-level health check matters because Kubernetes' own view (Running, Ready) only confirms the CONTAINER started and passed its readiness probe, not that the application is functioning correctly for real traffic, the same distinction between "process is up" and "service actually works" that motivates smoke testing generally.

Trade-offs and pitfalls
kubectl rollout undo only has as many previous revisions to roll back to as revisionHistoryLimit allows (commonly 10, but sometimes reduced for resource reasons), so a rollback several versions back may fail if that history's been pruned, which is worth checking with rollout history before assuming undo will reach the version you actually want. The most common mistake is treating the undo command's successful return as confirmation the rollback WORKED, rather than explicitly waiting on rollout status and independently verifying pod health, since a rollback that's technically "issued" but stuck (image pull failure, resource constraints preventing new pods from scheduling) looks identical to a successful one until you check.

Direct answer

Build the jar with Maven in a maven builder stage, then copy only the built artifact into a slim JRE (Java Runtime Environment, the minimal Java install needed to run compiled code, as opposed to the JDK which also includes the compiler and development tools) final stage; this both reduces image size (the Maven distribution, the full JDK, the local dependency cache, and the source tree never reach the shipped image) and improves reproducibility (the final image's contents are exactly one artifact, the jar, built from a pinned dependency set, rather than a live build environment whose installed tool versions could drift between builds). On top of that, set JVM (Java Virtual Machine, the runtime that executes compiled Java bytecode) memory flags that are aware of the container's own cgroup limit, not the host's total RAM (cgroup is the Linux kernel mechanism that tracks and caps how much memory and CPU a group of processes, everything running inside one container, is allowed to use), so the JVM sizes its heap correctly when the container is capped well below the host's physical memory.

Structured elaboration

The Dockerfile

FROM maven:3.9-eclipse-temurin-17 AS builder
WORKDIR /build
COPY pom.xml .
RUN mvn -B -q dependency:go-offline
COPY src ./src
RUN mvn -B -q package -DskipTests

FROM eclipse-temurin:17-jre-jammy AS final
RUN useradd --system --uid 10001 --create-home appuser
WORKDIR /app
COPY --from=builder /build/target/*.jar /app/app.jar
USER appuser
ENV JAVA_TOOL_OPTIONS="-XX:+UseContainerSupport -XX:MaxRAMPercentage=75.0 -XX:InitialRAMPercentage=50.0 -XX:+ExitOnOutOfMemoryError"
EXPOSE 8080
ENTRYPOINT ["java", "-jar", "/app/app.jar"]

Copying pom.xml and running dependency:go-offline before copying src is a caching optimization: as long as pom.xml does not change, Docker's build cache reuses the downloaded-dependencies layer even when application source changes on every commit, so ordinary code changes do not re-download the entire dependency tree.

The JVM tuning flags, and why they matter in a memory-limited container

• -XX:+UseContainerSupport (on by default in current JDKs, shown here for explicitness) makes the JVM read the container's own cgroup memory and CPU limits instead of the host's total physical resources when deciding how to size itself.
• -XX:MaxRAMPercentage=75.0 caps the JVM heap at 75% of whatever memory the container is actually allowed (not the host's), leaving room for the JVM's own non-heap overhead (thread stacks, metaspace, JIT compiler buffers, direct byte buffers) inside the same container limit. Without this, a JVM that assumes it has the whole host's RAM available can size its heap far too large for a container capped at, for example, 512 MB, and get OOM-killed by the container's own memory limit despite technically "having plenty of heap" from its own point of view.
• -XX:InitialRAMPercentage=50.0 sets the JVM's starting heap size (rather than growing from a very small default), trading a little extra memory reserved up front for fewer heap-resize pauses early in the process's life.
• -XX:+ExitOnOutOfMemoryError makes the JVM process exit immediately on an out-of-memory condition instead of limping along in a partially broken state; combined with a container restart policy, this turns a silent, slow-degrading failure into a clean, fast, and visible one.

Why the multi-stage build specifically reduces size and improves reproducibility

• Size: the maven:3.9-eclipse-temurin-17 builder image includes the full JDK, the Maven distribution, and (after the build) a populated local repository cache of every dependency; none of that is needed to run the already-built jar, so the final stage, based on a JRE-only image, ships only the runtime and the one artifact it needs.
• Reproducibility: because the final image's file system is built entirely from COPY --from=builder, its contents are deterministic given the same jar; there is no live Maven installation, plugin version, or locally cached dependency state inside the shipped image that could differ between two builds of "the same" image on different machines or at different times. The build-time environment (which can legitimately vary run to run) is fully separated from the runtime artifact (which should not).

Worked example

Building the Dockerfile above (a minimal Spring Boot 3.3 service with one @GetMapping("/hello") endpoint) produces a final image of 405 MB, running as uid=10001(appuser). Starting it with a 512 MB container memory limit (docker run -m 512m ...) and inspecting the JVM's own resolved flags shows MaxHeapSize = 402653184 bytes, which is exactly 75% of 512 MB (536,870,912 x 0.75 = 402,653,184), confirming -XX:MaxRAMPercentage=75.0 sized the heap against the container's cgroup limit rather than the host's full memory. The application answers GET /hello with hello from spring boot immediately after Tomcat (the embedded web server Spring Boot starts by default) reports itself started in under a second.

Trade-offs & pitfalls

• Skipping -XX:MaxRAMPercentage and relying on the JVM's older default heap-sizing behavior (a fraction of the host's physical memory) is a frequent cause of JVM services being OOM-killed in containers despite "looking fine" in local testing on a developer machine with much more RAM available to the same container limit.
• mvn package -DskipTests in the Dockerfile shown here assumes tests already ran as a separate CI gate (mirroring the discrete test-stage pattern used for other languages); baking a full test run into every image build slows down every build and conflates "did the code pass its tests" with "can we produce a deployable artifact," which are better kept as separate, independently rerunnable steps.
• A JRE-only final base image cannot compile anything and has no Maven; if a hotfix workflow assumes you can exec into a running production container and rebuild, this stage will not support that, by design, and the actual fix path is always to rebuild the image from source.

Approach

Three separate hazards have to be handled together here: the timeout has to actually kill the process (not just stop waiting for it), the streaming has to not deadlock on large output, and cleanup has to happen even if the parent itself is killed.

import subprocess, threading, time

def run_cli(cmd, timeout, log_file):
    with open(log_file, "w") as lf:
        proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
                                 text=True, bufsize=1)
        def pump():
            # read line-by-line in a background thread so stdout is drained
            # continuously -- reading ONLY after the process exits risks the
            # child blocking on a full OS pipe buffer if it writes a lot of output
            for line in proc.stdout:
                lf.write(line)
                lf.flush()
        pumper = threading.Thread(target=pump, daemon=True)
        pumper.start()
        try:
            exit_code = proc.wait(timeout=timeout)
        except subprocess.TimeoutExpired:
            proc.kill()      # SIGKILL: don't trust the child to honor SIGTERM
            proc.wait()       # REAP the process -- this is what prevents a zombie
            exit_code = -9
        pumper.join(timeout=1)
        return exit_code

Verified in a sandbox with two cases: a command that prints two lines and exits 0 returns exit code 0 with both lines correctly captured in the log file; a command that sleeps 5 seconds called with a 0.3s timeout is killed and reaped in ~0.3s (confirmed the process does not linger), returning exit code -9.

Key points

• Streaming to a rotating log file in a background thread, rather than only reading output after proc.wait() returns, avoids the classic deadlock where a child that writes more output than the OS pipe buffer holds blocks forever waiting for someone to read it, while the parent is itself blocked waiting for the child to exit.
• proc.kill() (SIGKILL) rather than proc.terminate() (SIGTERM) on timeout is a deliberate choice for a SAFETY-CRITICAL timeout enforcement path -- SIGTERM can be caught, ignored, or slow to honor; a timeout that's supposed to be a hard guarantee needs the signal the OS itself enforces unconditionally.

How resource cleanup on termination is guaranteed

The critical, easy-to-miss step is proc.wait() immediately AFTER proc.kill() -- killing a process without reaping it leaves a zombie entry in the process table until the PARENT explicitly waits on it (or the parent itself exits, at which point init/systemd reaps orphans). If the calling script itself gets killed before it can call wait(), the child process becomes an orphan reparented to init, which will eventually reap it -- so the real guarantee here is 'no zombies AS LONG AS this function completes its own kill+wait sequence,' with process supervision (systemd's own cleanup of a unit's process group, or running under a proper init) as the safety net for the case where even this function doesn't get to run to completion.

Trade-offs and pitfalls

A rotating log file needs its own size/retention policy independent of this function (this function assumes the log file handle is managed correctly, e.g. via Python's logging.handlers.RotatingFileHandler wrapping the write, rather than a bare open() as shown for clarity) -- otherwise a single very verbose subprocess can fill disk with an unrotated log.

Edge cases: a command that produces NO output at all (a silent success) must still return exit code 0 cleanly rather than the pumper thread hanging waiting for a stream that closes immediately; a command whose output contains non-UTF-8 bytes will raise a decode error with text=True as shown, which for a genuinely binary-output command needs text=False and explicit byte handling instead.

Direct answer

Manual rollbacks go wrong because they are rare, rushed, and done by whoever is on call (the engineer currently responsible for responding to alerts), from memory, under stress. I would change the culture so that undoing a change is a normal, rehearsed, low-drama decision shared by developers and operators, and change the process so that every change arrives with its own way back. Blame makes this worse because people hesitate to roll back for fear of being the person who "broke it".

Cultural changes (first, because tooling alone will not fix hesitation)

1. Make "roll back first" the default. If a recent change is the plausible cause, restoring the previous version comes before diagnosis. Say it out loud as a team norm, and make it a decision the incident commander (the person coordinating the response) can order without asking permission.
2. Blameless postmortems. A blameless postmortem examines how the system and process allowed a failure, not who erred. When someone made a bad rollback call, ask what information they lacked.
3. Shared ownership of recovery. Developers are on call for what they ship, or at least join the rollback, so knowledge of how to undo a change sits with the people who wrote it.
4. Practise. Hold regular game days (planned drills where the team deliberately rolls back a harmless change in a safe setting) so the first rollback of the year is not during an outage.

Process changes

1. Rollback plan in the definition of done (the team's checklist for when work counts as finished). A change is not ready for review until the author writes how it is undone, and the reviewer asks "can we undo this, including data changes?"
2. One written rollback path, one owner of the keyboard. Replace several ad hoc methods with a single documented routine, and have one person execute while another reads the steps and watches the effect. That pairing is the main protection against making things worse. Sample exchange:

Reader:   Step 1. Confirm the previous version is v41. What does the dashboard say?
Executor: Previous version is v41.
Reader:   Step 2. Roll back to v41. Say when started.
Executor: Started.
Reader:   Step 3. Error rate should fall below 1% within 5 minutes. What do you see?
Executor: 0.4%. Recovered.
Reader:   Step 4. Log the time and open the follow-up ticket.

3. Separate irreversible changes. Database schema changes and data migrations are flagged as one-way and get extra review and a staged approach, because "just roll back" does not apply.

Worked example (sequence)

Week 1: write down the three rollbacks that went wrong and what each lacked (no reviewer, no plan, unclear authority). Weeks 2 to 4: the norm and the decision right, plus the single routine. Weeks 5 to 8: the first game day, then the definition-of-done change. After that: track failed deployment recovery time (how long it takes to recover from a deployment that failed and needed immediate intervention). It is one of the DORA metrics, the five delivery measures from the DevOps Research and Assessment programme. You compute it per failed deployment as the time service was restored minus the time the failure started.

Illustrative numbers: before the changes, five failed deployments took 95, 40, 180, 60 and 125 minutes to recover, a median of 95 minutes. Two months later, five took 30, 45, 25, 60 and 40 minutes, a median of 40. Also count the rollbacks that caused a second problem, aiming for zero.

Trade-off: "roll back first" can bury the root cause if a change keeps being reverted without analysis; require a follow-up ticket each time.

Direct answer

A recurring metric didn't match my intuition, and nobody had ever actually checked the explanation everyone repeated for it. Instead of arguing about it in a meeting, I pulled the underlying data myself, gave myself a bounded couple of hours to test it, and it turned out the accepted explanation was wrong.

Structured elaboration

What triggers this for me is usually one of three things: a number that doesn't match intuition, an inconsistency between two things that are both supposedly true, or a claim that gets repeated in meetings without anyone citing where it came from. The move that matters is testing it rather than debating it: designing a small, specific data pull or check that would give a clear yes-or-no answer, instead of relying on memory or opinion.

Handling people who are invested in the accepted explanation is the part that actually determines whether the finding goes anywhere. I've found it works best to lead with the method, not the conclusion: show exactly what was pulled and how, invite the person closest to the original explanation to poke holes in it before taking it wider, and frame the result around what it costs or changes rather than around who was wrong. That keeps the disagreement about the data instead of about people.

Keeping it bounded matters just as much: I give myself a fixed, short window, often just a couple of hours, so the detour doesn't quietly become a second, uncommitted project on top of my actual work.

Worked example

A conversion or error-rate number kept coming in lower than expected, and the standing explanation in planning meetings was a vague reference to "seasonality," which nobody had actually verified. I queried the underlying events directly instead of the aggregated report, and found the drop tracked a specific upstream change, not the season at all. Because the explanation directly contradicted what the person who'd offered the seasonality theory had said publicly, I shared the query and the raw numbers with them first, privately, before raising it in the wider meeting, so they had a chance to check my work rather than being contradicted cold in front of others. The team ended up reverting the upstream change, and the metric recovered.

I've also pointed this same instinct outward: looking at what a competitor did differently on a public-facing page to understand why our own numbers were diverging from what we expected, rather than assuming our internal explanation was the only one worth testing.

Trade-offs and pitfalls

The failure mode on the other side of this trait is treating every mildly odd number as worth a detour, which quietly erodes committed work; the discipline of a fixed, short timebox is what keeps curiosity from becoming a distraction. The other pitfall is confirmation-bias digging: designing the check to find evidence for a hunch you already have, rather than genuinely testing whether the accepted explanation holds.