Senior DevOps Engineer Interview Preparation Guide for Apple

Direct answer

CAP theorem says a distributed system cannot simultaneously guarantee all three of Consistency (every read returns the most recent write), Availability (every request receives a non-error response), and Partition tolerance (the system keeps working when some nodes cannot reach others). Because real networks drop packets and links fail, partition tolerance is not optional, so the actual choice during a partition is between consistency and availability. Systems like Google Cloud Spanner lean CP (some requests are blocked or rejected to stay correct); systems like Cassandra or Riak in their common tuning lean AP (every request gets an answer immediately, even a possibly stale one).

Structured elaboration

The three letters, precisely

• Consistency: every node that answers a read returns the same, most-recent value; there is no "which replica did you happen to hit" effect.
• Availability: every request that reaches a non-failed node gets a response, in bounded time, even if that response can't be proven up to date.
• Partition tolerance: the system continues operating even though some messages between nodes are lost or delayed.

Why "CA" is not a real option
A system that isn't partition tolerant is only correct as long as the network never drops a message, which no real, multi-node network guarantees. So "CA" only describes a single-node system, or a distributed system that hasn't yet been tested by a real partition; it is not a deployable choice for a genuinely distributed service.

CP vs AP

PACELC, the extension worth naming
PACELC (if Partitioned, choose Availability or Consistency; Else, choose Latency or Consistency) points out that CAP only describes what happens during a partition. Even with no partition at all, a system still has to decide, on every write, how synchronously to replicate: wait for every replica to acknowledge (favors consistency, costs latency) or acknowledge after one replica and replicate the rest in the background (favors latency, costs consistency). An AP system's day-to-day behavior is really governed by its PACELC answer, not its CAP answer, since partitions are rare and normal operation is not.

Quorum tuning as the middle ground
Many stores let you tune how many replicas must acknowledge a write (W) and how many must respond to a read (R) out of N total replicas, sliding the same system between CP-like and AP-like behavior per operation rather than committing the whole deployment to one label. The mechanics of quorum protocols are their own topic; the point for this answer is that the CP/AP choice is a dial, not a single global switch.

Worked example

Take three replicas (N = 3) with a write quorum W = 2 and a read quorum R = 2, so that any write and any read are guaranteed to overlap on at least one replica:

$W + R > N \quad\Rightarrow\quad 2 + 2 > 3$

Now a partition splits the three replicas into a group of 2 and a group of 1.

• The group of 2 can still assemble a quorum (2 out of the required 2), so it keeps serving reads and writes: consistent and available, on that side.
• The lone replica cannot assemble a quorum (1 is short of 2), so it must refuse requests rather than answer from possibly-stale local data: it sacrifices availability to avoid sacrificing consistency.

If the same store were instead configured for W = 1, R = 1 (an AP-leaning setting), both sides could keep accepting writes independently during the same partition, and the two sides would need to reconcile any conflicting writes once the partition heals; nothing here changes the network, only the dial.

Two alternate ways to see the same trade, folded in from adjacent scenarios: to a non-engineer stakeholder, the choice can be framed without the theorem's name at all: "during an outage, would you rather the app show a slightly out-of-date number and stay up, or freeze and show an error until we're sure the number is right?" That question is CAP in plain language. The same trade also shows up in machine learning infrastructure: an online feature store serving features to a live model at inference time typically leans AP (serve the last known values within a freshness window rather than block the request), while the offline pipeline that snapshots those same features for a training run typically leans CP (a training set built from a partially-written snapshot silently corrupts the model, so the pipeline waits for a consistent point-in-time snapshot instead).

Trade-offs & pitfalls

• Treating "pick 2 of 3" as one permanent, whole-system architecture decision, instead of a per-operation or even per-data-type decision a single platform can make differently in different places.
• Conflating CAP's "C" with the "C" in ACID (atomicity, consistency, isolation, durability): CAP consistency is about replicas agreeing with each other; ACID consistency is about the database enforcing its own integrity constraints. They answer different questions and can both be true or false independently.
• Forgetting PACELC: an AP system still owes an answer to "how synchronously do you replicate during normal operation," since that decision affects latency and staleness every day, not just during rare partitions.
• Citing "CA" as if it were a legitimate label for a real distributed deployment, rather than a description of a single-node system or an untested assumption.

Direct answer

Treat each of the four process changes as a time-boxed trial with its own measurable hypothesis, not a permanent mandate, so at quarter end there's real evidence for which ones to keep, adapt, or drop rather than a gut call.

Structured elaboration

1. Continuous integration and continuous delivery (CI/CD) for models. Automate training, testing, containerized deployment, and a versioned model registry. Measure: lead time from merge to production, deployment frequency, and rollback rate. Safeguard: start with one non-critical pipeline, require automated unit and integration tests before any automatic deploy, use feature flags for gradual rollout, and monitor for performance drift after deployment.

2. Backlog grooming with a sprint-ready Definition of Ready (DoR, an explicit checklist a task must meet before it can enter a sprint). Weekly grooming with product managers that confirms data availability, success metrics, and task estimates before a story enters a sprint. Measure: the percentage of stories accepted as ready before sprint start, sprint completion rate, and the number of blocked tasks mid-sprint. Safeguard: time-box grooming sessions, rotate the facilitator so it doesn't become one person's gate, and keep a separate "research" lane for genuine exploratory work that doesn't fit the DoR checklist.

3. Clear acceptance criteria and success metrics per experiment. Every ticket states its data sources, evaluation metric, minimum improvement threshold, and deployment criteria before work starts. Measure: acceptance pass rate, rework hours, and post-deployment metric lift versus what was expected going in. Safeguard: use templates so criteria don't become overly strict busywork, and have a review committee for genuinely disputed cases rather than one person's judgment call.

4. Code review standards with lightweight pairing for critical components. Enforce linting, unit tests, docstrings, and two-person review for anything reaching production; encourage short pairing sessions early in a piece of work rather than only reviewing at the end. Measure: pull request (PR) lead time, defects found after merge, and codebase maintainability (concentration of complex, hard-to-change code). Safeguard: keep reviews time-limited (for example a 60-90 minute service-level target), exempt prototype branches from the full standard, and allow a senior engineer to fast-track a genuinely urgent fix.

Measuring overall impact. Compare the quarter's baseline to its end on cycle time, deployment frequency, model quality (a business-relevant metric, not just an offline accuracy number), a developer-satisfaction pulse survey, and incident count. Run the four changes as staggered or team-split trials where practical, collect metrics weekly, and decide at quarter end which to adopt, adapt, or abandon based on what the data actually shows.

Worked example

These numbers are the hypothesis a trial is designed to test, not a promised outcome, since nothing has run yet at the start of the quarter.

Baseline: 40% of stories entering a sprint meet the Definition of Ready (8 of 20 stories in a typical two-week sprint), and average lead time from a model being merge-ready to running in production is 6 weeks.

Trial target for the quarter: DoR compliance at 80% or higher (16 of 20 stories) by grooming out the ambiguity before sprint start, and CI/CD-driven deployment lead time down to 2 weeks, a reduction of about 67% ((6-2)/6 ≈ 0.67), by removing the manual handoffs between training, testing, and deployment.

At quarter end, if DoR compliance actually reached, say, 15 of 20 stories (75%), that's short of the 80% target but a real improvement over the 40% baseline, worth continuing and tuning rather than declaring either a full success or a failure. The decision point is the trend and the gap to target, not a single pass/fail number.

Trade-offs and pitfalls

Running all four changes at once makes it hard to attribute a velocity improvement to any single one; if attribution matters, stagger the rollout or split it across squads so each change can be evaluated somewhat independently. Code-review standards heavy enough for production code can slow down the genuinely exploratory work research needs, which is why prototype branches are explicitly exempted. And a Definition of Ready that's stricter than the team actually needs becomes a new bottleneck wearing the costume of quality, so the DoR checklist itself should be revisited at quarter end alongside the metrics it was meant to improve.

Command

docker run -d --name web --restart always -p 8080:80 -v /srv/www:/usr/share/nginx/html:ro nginx:latest

What this does (flag-by-flag)

• -d
  • Runs the container in detached mode (background). As a DevOps engineer, use this for long-running services.
• --name web
  • Assigns the container the human-readable name "web" for easier management (docker ps, logs, restart).
• --restart always
  • Ensures the container is automatically restarted by Docker if it exits, and on daemon reboot — suitable for service reliability.
• -p 8080:80
  • Publishes/mapped host port 8080 to container port 80 so HTTP on host:8080 reaches nginx inside the container.
• -v /srv/www:/usr/share/nginx/html:ro
  • Binds host directory /srv/www into the container at nginx's web root as read-only (prevents container from modifying host content).
• nginx:latest
  • Image to run (explicit tag recommended; pin to a digest or specific version for production).

Notes / best practices

• Ensure host directory permissions allow the container to read files.
• For production, consider using a pinned image tag and orchestrator-managed restarts (systemd/k8s) for advanced control.

Direct answer

A cross-team CLI earns trust the same way a good API does: predictable structure, safe defaults, and no surprises. The two commands (backup, restore) should be subcommands of one binary, each with its own scoped flags, plus a small set of global flags that apply everywhere.

Command surface

backuptool [global flags] <command> [command flags]

Global flags:
  --verbose, -v          increase log verbosity (repeatable: -vv)
  --dry-run              show what would happen, make no changes
  --config PATH          explicit config file path
  --output json|text     machine-readable vs human-readable output

Commands:
  backup   --target NAME [--full|--incremental] [--retention DAYS]
  restore  --target NAME --snapshot ID [--to PATH]
  list     [--target NAME]                 # discoverability: what CAN I restore?
  status   [--run-id ID]                   # is a backup/restore in flight, did it succeed?

list and status aren't in the original ask but are worth calling out explicitly: a tool that can only DO things and never tell you what it's already done gets treated as a black box, and operators route around black boxes.

Exit codes and help

Use the conventional split: 0 success, 1 for a runtime failure (the operation was attempted and failed), 2 for a usage error (bad flags -- the operation was never attempted). This distinction matters operationally: a wrapper script or a monitoring check needs to tell 'you typo'd a flag' apart from 'the backup actually failed', and a single generic non-zero exit code collapses that distinction. Every command needs -h/--help text with at least one example invocation; a flag list with no example is where most CLI usability complaints come from.

Idempotent --dry-run

--dry-run should exercise the exact same code path as a real run up to (and not including) the side-effecting step, so what it prints is genuinely what would happen, not a separately-maintained approximation that can drift out of sync. Concretely: build the full plan (which files, which destination, computed retention actions), print the plan, and return before calling anything that mutates state. This is 'idempotent' in the sense that running --dry-run a hundred times in a row changes nothing and always shows the same plan for the same inputs -- if it isn't, the dry-run path has drifted from the real path and can no longer be trusted.

Config precedence

Standard, least-surprising order (highest wins): explicit CLI flag > environment variable > config file > built-in default. Document this order once, prominently, because 'why didn't my env var take effect' is the single most common CLI support question, and it's almost always a precedence surprise.

Libraries

Python: argparse (stdlib, sufficient for subcommands) or click/typer for richer UX with less boilerplate. Go: the standard library's flag package is thin; cobra (used by kubectl, docker CLI) is the de facto standard for subcommand-heavy tools and is a defensible default recommendation for a cross-team tool that other engineers will extend.

Trade-offs and pitfalls

The most common mistake is over-fitting the CLI's flags to whatever the FIRST team that adopts it needs, then discovering every subsequent team wants a slightly different flag shape for the same underlying concept -- naming flags around the general operation (--target, --source) rather than one team's specific vocabulary pays off once the tool is genuinely cross-team. Edge case: a flag that's valid for one subcommand but silently ignored (not rejected) for another is a common source of confused bug reports; validate flag-subcommand compatibility explicitly rather than letting argparse accept anything globally-defined everywhere.

Direct answer

Treat the budget as a portfolio: rank candidate investments by how much measured downtime each one removes per dollar, fund the highest-ROI items first until the budget is spent, then translate the resulting hours of downtime avoided into a revenue or cost-avoidance figure a non-engineer can act on. The common mistake is funding the biggest-sounding project instead of the one with the best marginal return.

Structured elaboration

Step 1: build a downtime-hours ledger from real incidents

Pull the last 12 months of incidents and categorize each by root cause (correlated infrastructure failure, human/process delay, insufficient capacity, and so on). Sum the downtime attributable to each category. This ledger is the input to everything that follows, without it, prioritization is intuition, not evidence.

Step 2: price each candidate fix against the hours it removes

For every candidate investment, estimate its annual cost and the hours of downtime it removes, ideally by fully eliminating a category rather than vaguely helping it. Compute dollars per hour-year removed and rank from cheapest to most expensive.

Step 3: fund down the ranked list until the budget is spent

This is a knapsack-style trade-off: sometimes a slightly worse-ROI item is still funded because a better one does not fit the remaining budget. Treat that as an explicit, stated judgment call rather than a strict greedy pick.

Step 4: translate hours removed into a business number

Multiply hours removed by an estimated revenue-or-cost-per-hour-of-downtime figure, sourced from finance or product and stated as an assumption if not yet confirmed, to produce a dollar risk-reduction figure comparable against the spend.

Worked example

Assume the incident ledger for the last 12 months shows two categories:

Two candidate investments:

• A: multi-AZ redundancy for the data layer. Removes the correlated-failure category entirely (4.5 hours/year), costs $60k/year.
• B: automated failover tooling. Cuts manual-delay MTTR from 25 minutes to 5 minutes across the same 12 incidents/year, removing $(25-5) \times 12 / 60 = 4$ hours/year, costs $25k/year.

Dollars per hour-year removed:

B has the better marginal ROI, fund it first. With a $500k baseline infra spend, a 20% increase is a $100k budget: B ($25k) plus A ($60k) totals $85k, funding both, with $15k left for monitoring and alerting improvements that support both fixes.

Total downtime removed: $4.5 + 4 = 8.5$ hours/year, against a baseline of 9.5 hours/year attributable to these two categories ($4.5 + 5$), an 89% reduction in that measured downtime.

Business translation: assume, subject to finance confirmation, $50k of revenue at risk per hour of downtime during business hours:

That is the pitch for the non-engineer conversation: "$85k of this budget removes $425k a year of downtime-driven revenue risk, based on our own incident history."

Trade-offs & pitfalls

• The ROI math only covers failure categories with existing data; it says nothing about the tail risk of a novel failure mode not yet seen, do not present it as a complete risk model.
• Averaging MTTR across incidents of very different severity can hide the one incident that actually mattered; check the distribution, not just the mean, before trusting the "hours removed" estimate.
• A dollar-per-hour-of-downtime figure that is not sourced from finance is a guess dressed as a number; label it explicitly as an assumption when presenting it, and get it validated before it becomes the headline of the pitch.
• Diminishing returns are structural: the third and fourth items on the ranked list will have worse ROI than the first two by construction, do not extrapolate the 5x multiple to the whole budget.

ImagePullBackOff is the pod status shown when the kubelet has repeatedly failed to pull a container's image and is now backing off further attempts; like CrashLoopBackOff, it names the symptom, not the cause. The four usual causes are registry authentication, DNS resolution to the registry, a wrong image name or tag, and network reachability from the node to the registry.

Causes and their signatures

Representative event text you'd actually see:

Failed to pull image "myrepo/app:v2": rpc error: code = NotFound desc = failed to pull and unpack image "myrepo/app:v2": failed to resolve reference "myrepo/app:v2": myrepo/app:v2: not found

Failed to pull image "123456789012.dkr.ecr.us-east-1.amazonaws.com/app:v1": rpc error: code = Unknown desc = failed to authorize: 401 Unauthorized

Step-by-step: cluster, node, or registry?

1. Read events first.

kubectl describe pod <pod> -n <ns>
kubectl get events -n <ns> --sort-by=.metadata.creationTimestamp

2. Check the pull secret and service account actually being used.

kubectl get sa <sa> -n <ns> -o yaml
kubectl get secret <secret> -n <ns> -o yaml

3. Test from inside the cluster (isolates cluster-level DNS/NetworkPolicy problems from node-level ones):

kubectl run -i --rm debug --image=alpine --restart=Never -- sh
# inside: apk add --no-cache curl bind-tools; nslookup registry.example.com; curl -v https://registry.example.com/v2/

4. Test from the node itself. Since dockershim was removed in Kubernetes 1.24, the node-level container runtime on essentially every current cluster is containerd (or CRI-O), so the right tool is crictl, not docker. docker pull only applies if you're intentionally still running Mirantis cri-dockerd or looking at a pre-1.24 node.

kubectl get pod <pod> -o wide -n <ns>
# on the node:
sudo crictl pull <image>
sudo journalctl -u kubelet -e

5. Decide scope:
   • Registry-related: the error is 401/403/manifest unknown both from the debug pod and from the node directly.
   • Cluster-level: the in-cluster debug pod can't resolve or reach the registry, but the node itself can (curl succeeds from the node); points at CoreDNS or a NetworkPolicy blocking egress.
   • Node-level: only some nodes fail; others pull the same image fine. Points at a node-specific proxy, MTU, or firewall rule.

Two variants worth naming explicitly

Registry rate limiting across many pods at once. If ImagePullBackOff appears on many unrelated pods across the cluster at roughly the same time, rather than one pod, suspect the registry itself: unauthenticated pulls against a public registry are commonly rate-limited (429 Too Many Requests or an equivalent 401 once the anonymous quota is spent), and every node hitting that same registry trips it simultaneously. The fix is authenticated pulls, a pull-through cache/mirror, or spreading pulls out over time, not chasing one pod's config.

ECR token expiry as a time-correlated failure. On EKS Amazon Elastic Kubernetes Service, if pulls are authenticated with a static, manually-refreshed token stored in an imagePullSecret rather than through IRSA (IAM Roles for Service Accounts), that token is only valid for 12 hours. Pulls succeed for a while and then start failing cluster-wide at almost exactly the token's issuance time plus 12 hours, which is the tell that distinguishes this from a one-off misconfiguration: correlate the failure's onset timestamp against when the credential was last refreshed. The fix is to stop relying on a static Secret and either use IRSA so kubelet-side credential retrieval is automatic, or run a credential-refresh mechanism (such as the ECR credential helper or a scheduled Secret-rotation job) instead of a manually maintained token.

Trade-offs and pitfalls

• Don't assume "it's the registry" just because the error message is generic; a node-level DNS or proxy issue can produce an error that looks identical to a registry outage.
• A fix that works from a debug pod but not from the node (or vice versa) is itself diagnostic information: it tells you the failure is layer-specific, not systemic.
• Long-lived static pull secrets are an operational trap even when they work today; anything time-boxed (ECR tokens, expiring service-account tokens) will eventually fail in a way that looks like a random new bug unless the expiry is tracked explicitly.

Direct answer

This question is testing whether you own mistakes honestly and turn them into concrete, lasting process or tooling improvements, not whether you've never caused an outage. A strong answer names a real trigger, a real diagnosis process, and a specific change that prevents the same class of mistake, not just this exact one.

Structured elaboration

The interviewer is listening for: a credible, specific trigger (what pipeline change, and why did it cause the outage, described precisely enough to show you actually understood the mechanism, not just 'a bad deploy happened'); a real diagnosis narrative (how you or the team figured out the pipeline change was the cause, including any false leads you initially chased); a concrete mitigation in the moment (what you actually did to restore service, distinct from the longer-term fix); and, most importantly, a specific systemic change afterward that would have caught this class of problem earlier, not just fixed this one instance.

A weak answer stops at 'we rolled back and it was fine,' which describes the immediate mitigation but skips the part that actually demonstrates growth: what changed about the pipeline, the review process, or the testing strategy so the same shape of mistake is now caught automatically, before it ever reaches production again.

Worked example

A credible shape: 'A pipeline change I made added a new deployment step that skipped the smoke-test gate for a specific service, because I'd mentally modeled it as low-risk. It shipped a config change that silently broke the service's connection pool sizing under production load, which we didn't see in staging because staging's traffic volume never exercised the pool exhaustion path. We noticed within 15 minutes via error-rate alerting, rolled back to the previous deployment, and the immediate incident was over quickly. Afterward, I removed the smoke-test exception for that service (the actual mistake: assuming any service could be safely exempted from the standard gate), and separately added a load-shaped smoke test that exercises realistic concurrency, not just a single health-check request, specifically because staging's low-traffic smoke test wouldn't have caught this class of bug either.' This is credible because the mechanism is specific, the diagnosis is described honestly (including that staging didn't catch it, which is a real and common gap), and the fix addresses the actual root cause (an exemption that shouldn't have existed) rather than a surface-level patch.

Trade-offs and pitfalls

The most common weak answer blames the deployment or the tooling ('the pipeline just broke') rather than owning the specific decision that caused it, which reads as deflecting responsibility rather than demonstrating the self-awareness the question is actually probing for. A second common gap is describing a detailed incident but a vague, generic follow-up ('we improved our testing'), when a strong answer names the exact gap the incident revealed and the exact change that closed it.

Direct answer

Measuring tail latency across six hops means separating two questions: which hop is actually responsible for a given slow request (attribution), and is the tail getting worse over time (detection). Attribution needs per-hop distributed tracing with sampling that preserves slow traces even when it drops fast ones; detection needs P95/P99 (95th- and 99th-percentile latency, the response times only the slowest 5% and 1% of requests exceed) tracked as their own alertable series, since a stable median (this is the common trap: P99 spikes while the median looks completely healthy) hides exactly this class of problem. Once a hop is identified, the fix is rarely "make everything faster" but a targeted mitigation such as hedged requests, request prioritization, resource partitioning, or admission control aimed at that specific hop, each of which trades some cost or complexity for the latency it buys back.

Measurement and attribution

• Instrumentation: every one of the six services emits a span per request with start/end timestamps, propagated trace context, and enough metadata (host, downstream call outcome, queue wait time) to distinguish "this hop was slow" from "this hop was waiting on the next one."
• Sampling strategy: pure random sampling at low rates (say 1%) will almost never happen to capture a P99 request, since by definition only 1% of requests qualify and the sample and the tail rarely overlap. Use tail-preserving (tail-based) sampling: buffer a trace briefly and only decide to keep it once you know whether any span exceeded a latency threshold, so slow traces are captured close to 100% of the time while typical traces are still sampled cheaply.
• Attribution: once a slow trace is captured, break its total duration into a waterfall of per-hop contributions to see which hop consumed the largest share.
• Spike detection: alert on the rate of change of P95/P99 against a rolling baseline (for example, a sustained jump relative to the trailing window), not on a single fixed threshold, since normal traffic variation would otherwise cause constant false alarms.

Worked example: attributing a P99 spike across six hops

As an illustrative example, not measured data, suppose a captured slow trace shows an 800 ms end-to-end duration split across the six hops as follows:

Hop B accounts for $\frac{300}{800} = 0.375 = 37.5\%$ of the total, the single largest share, so it is the first place to investigate and the first place a mitigation should target, rather than spreading effort evenly across all six services.

Mitigation techniques, their overhead, and their risk

Trade-offs and pitfalls

• Chasing every hop at once instead of attributing first. Without the waterfall breakdown, teams tend to optimize the hop that's easiest to touch rather than the one actually driving the P99.
• Sampling uniformly at a low rate and concluding tail latency "looks fine" because slow traces were simply never captured. The sampling strategy has to be tail-aware, not just cheap.
• Applying hedging without a cap. It is the mitigation most likely to backfire under genuine overload, since it adds load exactly when the system can least afford it.
• Treating any single mitigation here as free. Every row in the table above buys latency at the cost of either infrastructure efficiency, code complexity, or operational risk; picking one should follow from what the attribution step actually showed, not from familiarity with the technique.

Approach (high level)

Use OPA as a gate that evaluates SBOM and vulnerability scan outputs during CI. Fail the job when: any CVE severity >= threshold (e.g., HIGH) or SBOM metadata (supplier, version, or components) is missing. Run OPA evaluation after image build + scanning step, before pushing or deploying.

Example Rego policy

package ci.guard

default allow = true

# deny if SBOM missing required fields
deny[msg] {
  sbom := input.sbom
  not sbom
  msg = "SBOM missing"
}
deny[msg] {
  sbom := input.sbom
  not sbom.artifacts
  msg = "SBOM artifacts missing"
}

# deny if any vulnerability >= threshold
severity_score(s) = n {
  scores := {"NONE":0, "LOW":1, "MEDIUM":2, "HIGH":3, "CRITICAL":4}
  n = scores[s]
}

deny[msg] {
  vuln := input.vulnerabilities[_]
  severity_score(vuln.severity) >= severity_score("HIGH")
  msg = sprintf("vuln %s severity %s exceeds threshold", [vuln.id, vuln.severity])
}

Integration points & timing

• After image build: run SBOM generator (syft) and vuln scanner (grype/Trivy).
• CI step: call opa eval -i scan-output.json --data policy.rego "data.ci.guard.deny" or use OPA Gatekeeper/Conftest library.
• If evaluation returns any deny messages, exit non-zero to fail the build.
• Optionally run policy as a pre-commit or pre-merge check for fast feedback; enforce in release pipeline for blocking.

Notes / best practices

• Keep policies in repo, versioned with code.
• Parameterize severity threshold via input or environment.
• Combine with attestations and allowlist for known acceptable CVEs.

Direct answer

Before you delete anything, you need proof the migration actually succeeded (not just that the new system is live), a defined retention and archival plan for whatever legal or audit obligations outlive the system itself, a rollback path in case something surfaces after decommission that the pre-cutover testing missed, and a communication plan that reaches every stakeholder who might still depend on the old system, including the ones you do not already know about.

Structured elaboration

A decommissioning plan has four parts, and skipping any of them is how "the migration is done" turns into "we deleted data we needed":

1. Verification that the migration is actually complete. Not "the new system works," but "nothing depends on the old one anymore." This means auditing traffic and access logs on the legacy system for weeks after the cutover, not just at the moment of cutover, because low-frequency dependencies (a monthly batch job, a quarterly report) will not show up in a one-week traffic sample.
2. Legal and audit retention requirements. Many systems have data that has to remain queryable for years after the system itself is gone, for regulatory or contractual reasons. That means an archival strategy decided before decommission, not scrambled together after someone asks for five-year-old records the week after you deleted the database.
3. A rollback plan for the decommission itself, distinct from the rollback plan for the original migration. If something surfaces after you have shut the legacy system down (a caller nobody knew about, a data discrepancy only visible under a rare condition), you need a defined path back, even if that path is "restore from the last verified backup and re-enable the legacy code path for a bounded window," not "we have no idea, we deleted it."
4. Third-party integrations and stakeholder communication. External partners often integrate with systems in ways your internal traffic logs cannot see (a partner polling an API you exposed to them specifically). The communication plan needs to reach them with enough lead time to migrate on their side, not just notify internal teams.

Worked example

A team decommissioning a legacy order-management system after a successful migration:

• They keep the legacy system read-only (not deleted) for 90 days post-cutover, monitoring access logs the whole time. In week six, they find a quarterly compliance report job still reading directly from the legacy database, which nobody had flagged as a dependency because it only runs four times a year.
• They export the full historical dataset to a queryable archive with a retention period matching the company's seven-year audit requirement, and verify a sample of archived records against the live system before the live system goes away, since an archive nobody has tested is not actually a safety net.
• They notify the three external partners who integrate with the legacy system's API directly, giving them 60 days' notice and a migration guide, rather than assuming internal migration alone covers everyone with a dependency.
• Only after all of this, and after a final confirmed zero-traffic week on the legacy system, do they actually shut it down, with the archived data and a documented restore procedure kept in case something surfaces later.

Trade-offs and pitfalls

The tempting shortcut is to declare victory the moment the new system handles 100% of live traffic and decommission immediately, but "no traffic this week" is not the same as "no dependencies," and the cost of being wrong (deleted data you needed, a partner integration silently broken) is far higher than the cost of a monitored grace period before deletion. The other common mistake is treating archival as a technical afterthought rather than a compliance requirement with its own sign-off, which is how companies end up unable to produce records a regulator or auditor asks for.