DevOps Engineer (Entry Level) Interview Preparation Guide - Apple

Approach summary
Compare scratch, distroless, and minimal distros for compiled Go services across size, security surface, debugging, library deps, and trade-offs, then give best-practice recommendations for production and debugging.

Image size

• scratch: smallest (single binary) — often <10–20MB for a statically linked Go binary + layers.
• distroless: minimal runtime files (ca certs, passwd) — slightly larger than scratch but still small (20–40MB).
• alpine / debian-slim: larger (30–120+MB) depending on packages.

Security surface

• scratch: minimal attack surface; no package manager or shells to exploit. Fewer CVEs, but you must ensure binary itself and build pipeline are secure.
• distroless: low attack surface; includes only runtime artifacts needed (e.g., CA certs).
• alpine/debian-slim: larger surface due to glibc/musl, shells, package managers — more CVEs and patching burden.

Debugging capability

• scratch/distroless: poor — no shell, no strace, limited runtime introspection. Must build debug images or use ephemeral debug sidecars.
• alpine/debian-slim: good — contains /bin/sh, common tools, easier to exec into container for live troubleshooting.

Library dependencies and cgo

• scratch/distroless: best when Go binary is statically linked (CGO disabled). If your service uses cgo or native libraries, scratch may fail — you must include required .so files or use distroless/busybox-based images. Alpine uses musl which can cause subtle differences versus glibc; debian-slim has glibc compatibility.

Trade-offs

• scratch: maximal minimalism and security, but lower observability and trickier startup (must bake certs, user files into image).
• distroless: balanced — small and secure while providing minimal runtime files, still limited debugging.
• minimal distros: easier troubleshooting and compatibility at cost of size and attack surface.

Recommendations (DevOps perspective)

• Production: prefer distroless for Go services that are statically linked; use scratch only if you need absolute minimal size and have CI-trusted reproducible builds. Ensure multi-stage builds, vulnerability scans, and signed images. Disable CGO or vendor required native libs intentionally.
• Debugging / staging: publish separate debug images based on alpine or debian-slim (same binary but linked/packaged with shells and tooling) or use ephemeral debug sidecars and init containers. Keep debug images out of production registries or tag clearly.
• Pipeline best practices: use multi-stage builds, reproducible build flags (CGO_ENABLED=0, -trimpath, -ldflags="-s -w"), image vulnerability scanning, and automated rebuilds when base images are patched.

Direct answer
Write a resource block of the correct type declaring only the arguments you intend to manage, run terraform import (or a declarative import block) to attach the real resource's ID to that address, then run terraform plan repeatedly, reconciling your HCL against what it reports, until plan shows no diff. Only then does apply touch anything, and by that point it's a no-op. Nothing about the live VPC or database changes during any of this, because import only writes state and plan never mutates anything, so the "without downtime" requirement is really just discipline about not applying until plan is clean.

Structured elaboration

1. Inspect the real resource before writing HCL
Don't write the resource block from memory. Look up the actual current attributes (via the console, CLI, or the provider's data source) so your first draft doesn't propose spurious changes the moment you import.

2. Write the minimal matching resource block
Keep it to the arguments you actually plan to manage going forward; leave out fields you expect the provider to compute (endpoints, generated ARNs) rather than guessing values for them.

resource "aws_db_instance" "example" {
  identifier = "my-db-identifier"
  # do not add attributes you expect AWS to compute (endpoint, address)
}

3. Import

terraform import aws_db_instance.example my-db-identifier

As of Terraform 1.5+ you can express the same step declaratively with an import block plus terraform plan -generate-config-out=generated.tf, which scaffolds a starting resource block for you instead of hand-writing one, useful when importing many resources at once.

4. Reconcile computed and drifted attributes
Run terraform plan. Anything present in the real resource but missing or different in your HCL shows up as a proposed change. For attributes the provider computes and you don't want Terraform fighting over (an RDS endpoint, a final snapshot identifier), either add the real value explicitly if you want to manage it, or mark it in a lifecycle block so Terraform stops treating drift there as something to fix.

lifecycle {
  ignore_changes = [endpoint, address]
}

5. Verify, then only apply if you mean to change something
terraform state show aws_db_instance.example confirms the mapped ID and attributes. Keep iterating on plan until it's clean. If you genuinely want to change a value going forward, that apply is a deliberate, reviewed change, not an accidental side effect of the import.

Nested/child resources aren't imported automatically
A hand-built VPC or database typically has dependent pieces, a DB subnet group, associated security groups, route tables, that each need their own import call. Importing the parent resource doesn't pull its children into management for you.

Worked example
Importing a hand-built RDS instance end to end:

terraform import aws_db_instance.example my-db-identifier
terraform state show aws_db_instance.example
terraform plan

A first plan on a genuinely hand-built instance will usually show a handful of proposed changes on computed fields like endpoint, address, and tags_all, since those were never in your HCL to begin with. You add ignore_changes for the ones you don't want to manage and explicit values for the ones you do, then re-run plan until it's a clean no-op, which is your confirmation that apply is now safe.

Trade-offs & pitfalls
Skipping the "inspect the real resource first" step is the most common way this goes wrong: you import, run plan, and discover it wants to replace the DB instance's engine or storage_type because you guessed a value instead of reading the real one, right after the whole point was to avoid touching it. Always back up state before starting (terraform state pull > backup.tfstate), and where possible rehearse the import against a non-production copy of the resource type first. Bulk-importing dozens of hand-built resources one CLI call at a time doesn't scale, that's what the 1.5+ import block with -generate-config-out is for.

Situation / Task
At my last role I led CI/CD for a microservice (Go) deployed to EKS. The goal was to replace manual builds/releases with a reproducible pipeline using GitHub Actions and Helm.

Action
Repository layout:

• /cmd/* (binaries)
• /pkg/* (packages)
• /charts/service-chart (Helm)
• .github/workflows/ci-cd.yml

Pipeline (GitHub Actions):

• build: checkout, setup-go, docker build + multi-arch images
• test: unit tests, go vet, coverage upload
• scan: Snyk container scan and Trivy image scan
• publish: push images to ECR with image tags (commit SHA, semver)
• deploy: helm upgrade --install to EKS in staging; manual approval for prod

Credentials & env separation:

• Secrets stored in GitHub Actions Secrets (ECR creds, KUBECONFIG encrypted)
• Kube contexts separated: KUBECONFIG_STAGING / KUBECONFIG_PROD; workflows use environment protection rules and required reviewers for prod
• Images tagged with branch and PR refs to avoid collision

Result
Feedback loop reduced from ~45 minutes (manual build + deploy) to ~7 minutes automated for staging; zero-config reproducible releases and 60% fewer deployment-related rollbacks.

Direct answer

When wall-clock timestamps across regions can't be trusted and some traces are missing, don't build one linear timeline from raw timestamps. Instead, anchor on the most reliable time sources you have, order events causally wherever a correlation ID or a request/response pair links them directly, and explicitly attach a confidence level to every event in the reconstructed timeline rather than presenting a single false-precision sequence.

Structured elaboration

Data sources to pull, roughly in order of trust

• A single, centralized ingress point (a global load balancer or CDN edge) if one exists: its clock is one clock, not many, so events it recorded are internally consistent with each other even if every other host disagrees.
• Any surviving distributed traces, since a trace's internal span ordering (parent started before child) is causal, not just timestamp-based, and survives clock skew even when the absolute times are off.
• Metrics and alert history, which are often still complete even when detailed traces were dropped, and which give you a coarse but reliable window for when things degraded.
• NTP (Network Time Protocol, the standard system-clock synchronization protocol) or chrony logs, if collected, which tell you how far each host's clock had actually drifted at the relevant time.
• Deploy and configuration-change records, usually from a system with its own independent clock, useful as an anchor point that doesn't depend on the skewed hosts at all.

Correlating and reconstructing

• Join everything you can on a request or correlation ID first; causal links from an ID match are far more trustworthy than "these two lines have similar timestamps."
• Where you know or can estimate a host's clock offset (from NTP logs, or by comparing a request's arrival time at an ingress point against that same request's logged time on the skewed host), apply the correction and keep the original alongside it, don't overwrite the raw evidence.
• For traces that were dropped entirely, use surrounding evidence (the request's presence in an access log, a metric spike, a queue-depth anomaly) to infer that something happened in that window even without a full span, being explicit that this is an inference, not a direct observation.

Confidence levels

Assign each event or causal link an explicit confidence tier rather than a single flat timeline: high confidence for events tied together by a shared trace or request ID with no clock dependency at all, medium confidence for events correlated by corrected/estimated time plus a plausible causal story, and low confidence for anything relying on raw uncorrected timestamps alone. State the tier next to each claim in the final writeup.

Worked example

Suppose a request enters through a single global load balancer at 14:02:00.100 (the load balancer's own clock, one source, trusted), and two downstream services in different regions log timestamps of 14:01:58 and 14:02:03 for what should be the same request. Rather than concluding the second service ran before the first, or that there's a 5-second gap, you'd check whether both services log the same request/correlation ID; if they do, you know the causal order (load balancer received it, then downstream service A, then downstream service B) regardless of what the raw clocks say, and you'd note the load-balancer-anchored event as high confidence while flagging the two regional timestamps as unreliable without a computed clock offset.

Trade-offs & pitfalls

The pitfall that produces the most confidently wrong postmortems is presenting a single merged timeline built from raw, uncorrected timestamps as if it were ground truth. It looks authoritative and it usually is not; readers act on it as if every ordering claim were equally certain. The discipline of tagging confidence per event is more work up front, but it's what keeps a plausible-looking but wrong causal story from becoming the official root cause. A second pitfall is treating "dropped trace" as "nothing happened": absence of evidence in a system known to lose data under load is not evidence of absence, and inferring from surrounding signals (however lower-confidence) is usually better than leaving a silent gap in the narrative.

Approach

Three separate hazards have to be handled together here: the timeout has to actually kill the process (not just stop waiting for it), the streaming has to not deadlock on large output, and cleanup has to happen even if the parent itself is killed.

import subprocess, threading, time

def run_cli(cmd, timeout, log_file):
    with open(log_file, "w") as lf:
        proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
                                 text=True, bufsize=1)
        def pump():
            # read line-by-line in a background thread so stdout is drained
            # continuously -- reading ONLY after the process exits risks the
            # child blocking on a full OS pipe buffer if it writes a lot of output
            for line in proc.stdout:
                lf.write(line)
                lf.flush()
        pumper = threading.Thread(target=pump, daemon=True)
        pumper.start()
        try:
            exit_code = proc.wait(timeout=timeout)
        except subprocess.TimeoutExpired:
            proc.kill()      # SIGKILL: don't trust the child to honor SIGTERM
            proc.wait()       # REAP the process -- this is what prevents a zombie
            exit_code = -9
        pumper.join(timeout=1)
        return exit_code

Verified in a sandbox with two cases: a command that prints two lines and exits 0 returns exit code 0 with both lines correctly captured in the log file; a command that sleeps 5 seconds called with a 0.3s timeout is killed and reaped in ~0.3s (confirmed the process does not linger), returning exit code -9.

Key points

• Streaming to a rotating log file in a background thread, rather than only reading output after proc.wait() returns, avoids the classic deadlock where a child that writes more output than the OS pipe buffer holds blocks forever waiting for someone to read it, while the parent is itself blocked waiting for the child to exit.
• proc.kill() (SIGKILL) rather than proc.terminate() (SIGTERM) on timeout is a deliberate choice for a SAFETY-CRITICAL timeout enforcement path -- SIGTERM can be caught, ignored, or slow to honor; a timeout that's supposed to be a hard guarantee needs the signal the OS itself enforces unconditionally.

How resource cleanup on termination is guaranteed

The critical, easy-to-miss step is proc.wait() immediately AFTER proc.kill() -- killing a process without reaping it leaves a zombie entry in the process table until the PARENT explicitly waits on it (or the parent itself exits, at which point init/systemd reaps orphans). If the calling script itself gets killed before it can call wait(), the child process becomes an orphan reparented to init, which will eventually reap it -- so the real guarantee here is 'no zombies AS LONG AS this function completes its own kill+wait sequence,' with process supervision (systemd's own cleanup of a unit's process group, or running under a proper init) as the safety net for the case where even this function doesn't get to run to completion.

Trade-offs and pitfalls

A rotating log file needs its own size/retention policy independent of this function (this function assumes the log file handle is managed correctly, e.g. via Python's logging.handlers.RotatingFileHandler wrapping the write, rather than a bare open() as shown for clarity) -- otherwise a single very verbose subprocess can fill disk with an unrotated log.

Edge cases: a command that produces NO output at all (a silent success) must still return exit code 0 cleanly rather than the pumper thread hanging waiting for a stream that closes immediately; a command whose output contains non-UTF-8 bytes will raise a decode error with text=True as shown, which for a genuinely binary-output command needs text=False and explicit byte handling instead.

Approach (DevOps perspective)
Find processes for a given user older than 24h (86_400s), skip whitelisted commands and essential system PIDs/users, send SIGTERM, wait up to configurable timeout, then SIGKILL if still alive. Log each step with timestamps.

Safe Bash snippet

#!/usr/bin/env bash
set -euo pipefail

USER="${1:-deploy}"                  # target user
WHITELIST=("sshd" "systemd" "init")  # commands NOT to kill
LOG="/var/log/prune-procs.log"
GRACE=30   # seconds to wait before SIGKILL
THRESH=86400

ts(){ date +"%F %T"; }

# list pid, etimes, cmd
ps -u "$USER" -o pid=,etimes=,comm= | while read -r pid etimes comm; do
  # sanity checks
  [[ -z "$pid" || "$pid" -eq 1 ]] && continue
  # skip system accounts (optional)
  if id -u "$USER" &>/dev/null; then
    :
  fi
  # whitelist check
  skip=false
  for w in "${WHITELIST[@]}"; do [[ "$comm" == "$w" ]] && skip=true && break; done
  if $skip; then
    echo "$(ts) SKIP pid=$pid cmd=$comm reason=whitelist" >> "$LOG"
    continue
  fi
  if (( etimes > THRESH )); then
    echo "$(ts) TERM pid=$pid cmd=$comm etimes=$etimes" >> "$LOG"
    kill -TERM "$pid" 2>/dev/null || continue
    # wait up to GRACE for process exit
    for i in $(seq 1 $GRACE); do
      if ! kill -0 "$pid" 2>/dev/null; then
        echo "$(ts) EXITED pid=$pid" >> "$LOG"
        break
      fi
      sleep 1
    done
    if kill -0 "$pid" 2>/dev/null; then
      echo "$(ts) KILL pid=$pid" >> "$LOG"
      kill -KILL "$pid" 2>/dev/null || echo "$(ts) FAILED_KILL pid=$pid" >> "$LOG"
    fi
  fi
done

Notes & best practices

• Run as root/privileged to see all user's processes, but be cautious — restrict to specific users.
• Extend whitelist with full command paths or use pattern matching.
• Test in staging; run dry-run by echoing actions before actual kill.
• Integrate with monitoring/alerting so service owners are notified before termination.

Direct answer
Rolling back a Kubernetes deployment with immutable image tags is kubectl rollout undo, which redeploys the previous ReplicaSet's pod template from Kubernetes' retained revision history; verification means confirming the rollout actually completed and the resulting pods are healthy, not just that the command returned success.

Structured elaboration and worked example

# 1. Confirm the current rollout history and identify the target revision (optional, but good practice)
kubectl rollout history deployment/checkout-api

# 2. Trigger the rollback to the immediately previous revision
kubectl rollout undo deployment/checkout-api

# 3. Watch it complete (this blocks until the rollout finishes or times out)
kubectl rollout status deployment/checkout-api --timeout=120s

# 4. Confirm the pods are actually on the expected previous image tag
kubectl get pods -l app=checkout-api -o jsonpath='{.items[*].spec.containers[*].image}'

# 5. Confirm readiness: all pods Running and Ready, not just Running
kubectl get pods -l app=checkout-api

# 6. Application-level health confirmation beyond kubectl's view of pod state
curl -sf https://checkout-api.internal/healthz

Why each verification step matters: rollout status blocking until completion (rather than assuming success the instant undo returns) catches a rollback that's stuck (for example, the previous image was garbage-collected from the registry and can't be pulled, which undo will happily accept as a command but which then fails to actually schedule). Checking the ACTUAL image tag on running pods, not just trusting the command succeeded, catches a mismatch between what you intended and what's actually running. The final application-level health check matters because Kubernetes' own view (Running, Ready) only confirms the CONTAINER started and passed its readiness probe, not that the application is functioning correctly for real traffic, the same distinction between "process is up" and "service actually works" that motivates smoke testing generally.

Trade-offs and pitfalls
kubectl rollout undo only has as many previous revisions to roll back to as revisionHistoryLimit allows (commonly 10, but sometimes reduced for resource reasons), so a rollback several versions back may fail if that history's been pruned, which is worth checking with rollout history before assuming undo will reach the version you actually want. The most common mistake is treating the undo command's successful return as confirmation the rollback WORKED, rather than explicitly waiting on rollout status and independently verifying pod health, since a rollback that's technically "issued" but stuck (image pull failure, resource constraints preventing new pods from scheduling) looks identical to a successful one until you check.

Direct answer

When legal or compliance flags a possible regulatory problem on something about to ship, that flag is new information, not an attack on the project. The first move is to separate the specific risk from the whole feature: find out exactly what triggers the concern, then look for a way to ship everything outside that blast radius (the specific data, users, or markets the flagged concern actually touches) while the risky piece gets handled properly. Treating the flag as either a full block to fight or a formality to route around are both weak answers; the senior move is to make the freeze as small as the actual risk.

Structured elaboration

1. Turn the flag into a scoped, written finding

Ask for the specific clause or regulation, the specific data flow or behavior it applies to, and which markets or user segments are affected. A flag that sounds like 'this violates a regulation' often narrows down to 'this one data field, in these two markets.' Until that scoping happens, nobody can reason about mitigation, they can only argue about the abstract freeze.

2. Sort what's actually blocked from what's just slow

Once scoped, most flags fall into three buckets: genuinely unsafe to ship anywhere (rare, but real, treat it as a hard stop); unsafe in specific markets or for specific data (the common case, often scoped out with a flag or market-level rule); or unsafe as currently designed but fixable with a smaller change than a full freeze (needs a scoped rework, not a blanket delay).

3. Bring a mitigation, not just a constraint

Offer a concrete option: disable the flagged behavior for the affected markets, gate it behind a feature flag (a toggle that turns a piece of functionality on or off without a new deployment), or ship a version that omits the specific data flow while the rest proceeds. This turns the conversation from 'can we go or not' into 'does this mitigation satisfy the concern,' which moves much faster.

4. Get joint, written sign-off before proceeding

Both the business owner and compliance need to agree in writing on what shipped, what did not, the remaining risk, and who owns closing it. This protects everyone if the interpretation is questioned later and prevents the same argument from recurring next release.

5. If a real freeze can't be avoided, negotiate the timeline explicitly

Sometimes there is no safe scoped path and the freeze has to hold for the affected piece. Here the negotiation shifts to: what's the minimum change needed to clear the concern, who is assigned to it, and can the review be fast-tracked with a dedicated reviewer instead of sitting in a general queue. A freeze with a committed, shrinking timeline is a very different conversation from an open-ended one.

Worked example

A team is about to ship a feature that logs a new field for product analytics, and legal flags that collecting that field may violate a data-protection rule in one region. Scoping the flag shows the issue is narrow: one field, one region. Instead of freezing the whole release, the team ships everywhere else immediately, and for the flagged region ships the same feature with that one field's collection disabled behind a config switch. Legal signs off on the scoped version in writing. The team opens a follow-up item, with an owner and a target date, to redesign how that field is collected (for example, aggregating it instead of storing it per user), so the region isn't stuck without the feature indefinitely.

Trade-offs and pitfalls

• Treating every compliance flag as either a full block or a nuisance to route around is the most common mistake here; both extremes erode trust with the compliance function over time.
• Scoped mitigations (flags, market gating, field exclusions) are good short-term tools but can quietly become permanent if nobody owns the follow-up fix. The sign-off should name an owner and a date, not just describe a workaround.
• Escalating past compliance to force a ship date, without addressing the underlying concern, tends to resurface later as a bigger problem: a real violation or a regulator inquiry. Speed gained by skipping the process rarely survives contact with the risk it was protecting against.
• The strongest signal of seniority isn't how fast the team got to yes, it's whether the final decision is something both sides would still defend the same way months later.

The core risk in this migration is not containerization itself, it is state: a monolith on VMs with local disk and in-process sessions has state living in three places, local disk, in-memory sessions, and the database, that Kubernetes' pod model does not preserve across restarts, rescheduling, or rolling updates. The plan externalizes each of those in turn, in order of risk, before extracting pieces of the monolith into separate services with the strangler pattern, so the system never depends on a single big-bang cutover.

1. Containerize the monolith as-is first

Build an image for the existing Java monolith with externalized configuration (environment variables, ConfigMaps and Secrets instead of files baked into the VM), add liveness and readiness probes so Kubernetes can tell a hung JVM (the Java Virtual Machine, the runtime the monolith executes in) from a healthy one, and handle SIGTERM correctly (the termination signal Kubernetes sends a container before killing it): drain in-flight requests before exit, matching the JVM's shutdown hook to Kubernetes' terminationGracePeriodSeconds so a rolling update does not drop in-flight traffic. Get this version running and stable on Kubernetes before touching architecture.

2. Externalize local disk state

Kubernetes pods are ephemeral and can be rescheduled to a different node at any time, so anything the monolith wrote to local disk and expected to still be there tomorrow has to move:

• Transient, pod-lifetime-only files: an EmptyDir volume, deleted with the pod, is fine.
• Durable files the application actually needs to persist: a networked or object store. Blob-shaped data (uploads, reports) moves to an object store; anything requiring POSIX file semantics moves to a ReadWriteMany-capable networked filesystem.
• Migrate existing files with a one-time copy job, verify with checksums, and cut reads and writes over behind a feature flag so a bad migration can be reverted without a second migration.

3. Externalize session state

The same ephemeral-pod problem applies to in-process HTTP sessions: a session held only in the JVM's memory on one pod disappears the moment that pod is rescheduled, evicted, or replaced during a rolling update, which shows up to a user as an unexplained logout. The fix at the workload-design level is to externalize sessions to an external store, Redis is the common choice, so any pod can serve any request. Routing a user's requests back to the same pod is a different, ingress-layer session-affinity technique with its own trade-offs; it does not solve the underlying problem that a rescheduled pod still loses in-process state, so it is not a substitute for externalizing it.

4. Database migration strategy

Move the database carefully, since it is both the hardest state to externalize and the one most likely to be gotten wrong:

• Add logical replication (change-data-capture tooling such as Debezium, or the database's native replication) from the source to the target so the target stays current while the application still writes to the source.
• Use the expand-contract pattern for schema changes so old and new code can both run correctly during the transition: expand (add new columns or tables without touching old ones), migrate and backfill data, dual-write or dual-read as needed, then only after the new code path is validated in production, contract (stop writing the old shape, then drop it).
• Cut writes over in a single, short, controlled window once replication lag is at or near zero and read paths have already been validated against the target; keep the source writable-but-idle for a defined rollback window before decommissioning it.

5. Incremental rollout: the strangler pattern

Rather than extracting everything at once, put a facade, typically at the ingress layer, in front of the monolith that can route specific paths to a newly extracted service while everything else still goes to the monolith unchanged. Extract modules with low fan-in first, not the highest-traffic module first, because fan-in (how many other parts of the system call into a module) is what determines how much of the system breaks if the extraction has a bug, independent of how much traffic the module carries.

Worked example: choosing extraction order

Say three candidate modules have the following measured properties:

Using a simple priority heuristic of request share divided by fan-in, favoring modules that matter but are not deeply entangled:

$\text{Auth: } \frac{15}{2} = 7.5$

$\text{Reporting: } \frac{5}{1} = 5$

$\text{Checkout: } \frac{40}{10} = 4$

Auth extracts first (highest score), Reporting second, and Checkout last, despite carrying the most traffic, because its fan-in of 10 means a bug in the extraction has ten times the blast radius of Reporting's. This is the general shape of the reasoning a strangler-pattern plan should show, not a fixed formula: traffic volume alone is the wrong axis to sequence on.

6. Rollback strategy

Rollback needs to work at three independent layers, because a single "revert" button does not exist across all of them:

• Code: Kubernetes Deployments keep revision history, so kubectl rollout undo reverts to the previous ReplicaSet; tune maxSurge and maxUnavailable so a bad rollout is caught by readiness probes before it receives significant traffic.
• Data: because schema changes were expand-contract, the previous code version still works against the current schema right up until the contract step, so a code rollback does not also force an emergency schema rollback.
• Extracted services: keep the facade able to route a given path back to the monolith if the newly extracted service misbehaves, for as long as the monolith still contains that functionality; do not delete monolith code paths the moment a service is extracted, retire them on a delay after the extraction is proven.

Primary risks and mitigations

Direct answer

A recurring metric didn't match my intuition, and nobody had ever actually checked the explanation everyone repeated for it. Instead of arguing about it in a meeting, I pulled the underlying data myself, gave myself a bounded couple of hours to test it, and it turned out the accepted explanation was wrong.

Structured elaboration

What triggers this for me is usually one of three things: a number that doesn't match intuition, an inconsistency between two things that are both supposedly true, or a claim that gets repeated in meetings without anyone citing where it came from. The move that matters is testing it rather than debating it: designing a small, specific data pull or check that would give a clear yes-or-no answer, instead of relying on memory or opinion.

Handling people who are invested in the accepted explanation is the part that actually determines whether the finding goes anywhere. I've found it works best to lead with the method, not the conclusion: show exactly what was pulled and how, invite the person closest to the original explanation to poke holes in it before taking it wider, and frame the result around what it costs or changes rather than around who was wrong. That keeps the disagreement about the data instead of about people.

Keeping it bounded matters just as much: I give myself a fixed, short window, often just a couple of hours, so the detour doesn't quietly become a second, uncommitted project on top of my actual work.

Worked example

A conversion or error-rate number kept coming in lower than expected, and the standing explanation in planning meetings was a vague reference to "seasonality," which nobody had actually verified. I queried the underlying events directly instead of the aggregated report, and found the drop tracked a specific upstream change, not the season at all. Because the explanation directly contradicted what the person who'd offered the seasonality theory had said publicly, I shared the query and the raw numbers with them first, privately, before raising it in the wider meeting, so they had a chance to check my work rather than being contradicted cold in front of others. The team ended up reverting the upstream change, and the metric recovered.

I've also pointed this same instinct outward: looking at what a competitor did differently on a public-facing page to understand why our own numbers were diverging from what we expected, rather than assuming our internal explanation was the only one worth testing.

Trade-offs and pitfalls

The failure mode on the other side of this trait is treating every mildly odd number as worth a detour, which quietly erodes committed work; the discipline of a fixed, short timebox is what keeps curiosity from becoming a distraction. The other pitfall is confirmation-bias digging: designing the check to find evidence for a hunch you already have, rather than genuinely testing whether the accepted explanation holds.