Infrastructure as Code and Automation Questions

Direct answer
Sequence by dependency layer, not by application: foundational services first (network connectivity, identity, DNS, logging), shared services next, then non-prod workloads to prove the IaC modules work, and only then production, one tier at a time with a canary before full cutover. For each system, decide import versus rebuild based on whether it's a well-understood, cloud-native-mappable resource, in which case import, or a piece of accumulated config drift where reproducing it faithfully would just encode the debt, in which case rebuild.

Structured elaboration

Phase 1: Discovery and inventory
Automated discovery beats a spreadsheet built from memory: pull from the CMDB (configuration management database, the org's system-of-record inventory) if one exists and is trustworthy, cross-reference with what's actually running (Ansible facts or an SCCM/WSUS export for Windows, network scans for anything undocumented). The output is one canonical list per host/service: what it is, what it depends on, what depends on it, current backup/replication status, and an owner. Systems with no identifiable owner get flagged for a decision, migrate as-is or decommission, before they get IaC'd at all; don't automate ownership questions away.

Deciding import versus rebuild

• Import when the resource maps cleanly onto a cloud construct and its current config is trusted (a well-behaved VM that's really just a Linux box running a stateless service maps onto a cloud VM/instance with no surprises).
• Rebuild when the current config has drifted far enough from any documented baseline that reproducing it faithfully would mean writing IaC for years of undocumented manual tweaks, when a cloud-native replacement removes an entire operational burden (a hand-managed on-prem DB server becoming a managed database service), or when the security posture is bad enough that carrying it forward as-is isn't acceptable.
• The uncomfortable middle case, a system nobody fully understands but that's too risky to touch, gets isolated (its own network segment, tightly scoped IAM) and migrated last, with extra testing budget, rather than either extreme.

Sequencing to keep risk low

1. Network and identity: VPN/DirectConnect (or equivalent) between on-prem and cloud, IAM/AD federation, so hybrid connectivity exists before anything depends on it.
2. Shared services: DNS, NTP, centralized logging, backup targets, so everything migrated afterward has somewhere to report to.
3. Non-production workloads: prove the IaC modules (network, compute, IAM patterns) work end to end where a mistake is cheap.
4. Staging/canary for production-bound applications, using production-shaped data volumes and traffic patterns where feasible.
5. Stateful systems (databases, anything with data gravity), with explicit replication and cutover planning, not treated as "just another VM."
6. Production cutover, app by app, in dependency order.

Testing at each stage

• Terraform-level: validate, fmt, tflint/checkov in CI before anything runs against real infrastructure.
• Integration: apply into a sandbox, run smoke tests against the actual service and functional tests exercising real workflows, not just "did terraform apply succeed."
• Cutover rehearsal: run the actual runbook against staging at least once before running it against production, timing each step so production isn't the first execution.

The cutover runbook
An ordered checklist with explicit go/no-go gates:

1. Pre-checks: final data sync started, config drift on the source system confirmed at zero, backups verified restorable via an actual test restore, on-call notified.
2. Provision cloud infra via the pinned IaC module version, not "latest."
3. Final data sync completes, replication lag confirmed near zero.
4. Health checks and smoke tests pass against the new environment before any real traffic touches it.
5. Traffic shift via DNS/load-balancer weighting, gradual, with metrics watched between each step.
6. Go/no-go gate: if error rate or latency degrades beyond the pre-agreed threshold at any step, weight shifts back to on-prem immediately, no debugging in place first.
7. Once fully shifted and stable for the agreed validation window, decommission the on-prem system, keeping backups per the org's retention policy.

Worked example
A concrete piece of Phase 1 for one Linux host, using two purpose-built Ansible modules rather than plain gather_facts (the standard setup module never collects open ports or installed packages, referencing a fact it doesn't produce fails silently, not loudly, which is its own trap):

# discovery-playbook.yml
- hosts: legacy_estate
  gather_facts: true
  tasks:
    - name: Gather listening port facts
      community.general.listen_ports_facts:

    - name: Gather package facts
      ansible.builtin.package_facts:

    - name: Record OS, packages, and listening ports
      ansible.builtin.set_fact:
        inventory_record:
          hostname: "{{ ansible_hostname }}"
          os: "{{ ansible_distribution }}{{ ansible_distribution_version }}"
          packages: "{{ ansible_facts.packages.keys() | list if ansible_facts.packages is defined else [] }}"
          listening_ports: "{{ ansible_facts.tcp_listen | default([]) | map(attribute='port') | list }}"

    - name: Append to canonical inventory
      ansible.builtin.lineinfile:
        path: ./inventory.jsonl
        line: "{{ inventory_record | to_json }}"
        create: true

community.general.listen_ports_facts (it needs netstat or ss on the target and is Linux-only, matching this "legacy_estate" group) is what actually populates ansible_facts.tcp_listen, a list of dicts with a port field, among others, which is why the field is only meaningful after this task runs, not from gather_facts alone. ansible.builtin.package_facts auto-detects the box's package manager and populates ansible_facts.packages as a dict of package name to installed-version list. lineinfile needs create: true or it refuses to touch a file that doesn't exist yet; without it, the very first run of this playbook fails before the inventory file is ever created.

Running this across the estate produces one JSON line per host, which feeds the import-versus-rebuild decision: a host whose listening ports and packages match a known, simple service profile is a strong import candidate; one that doesn't match anything documented goes into the "investigate before deciding" bucket.

Trade-offs & pitfalls

• Treating discovery as a one-time step is a common mistake: on-prem estates drift during the months a migration takes, so re-run discovery periodically through the project, not just at kickoff.
• Rebuilding too aggressively, using the migration as an excuse to modernize everything at once, multiplies risk; separate "move" from "improve" and do the improvement afterward, once the system is stable on the new platform.
• The go/no-go gate only works if the threshold is agreed before the cutover; under pressure, teams tend to talk themselves into pushing through a borderline metric rather than rolling back.
• Stateful systems are where hybrid migrations actually go wrong; budget disproportionately more testing and rehearsal time for databases and anything with data gravity than for stateless compute.

Direct answer

Stop making further changes, protect the stateful resource immediately with a fresh snapshot (or a replica), and then use a CloudFormation change set, never a blind update, to see exactly what any reconciling update would do before it happens. Keep the database itself out of that reconciliation entirely by setting DeletionPolicy/UpdateReplacePolicy to Retain and, if the template and the live resource have drifted apart, adopting the existing database into the stack via resource import rather than letting CloudFormation replace it. Only cut traffic over to anything new once it is validated, so the database is never touched by an apply that could replace it.

Structured elaboration

Assess before touching anything further

• Check stack events (aws cloudformation describe-stack-events) to see exactly which resources were replaced, which were updated in place, and which are in a failed or rollback state.
• Run drift detection (aws cloudformation detect-stack-drift then describe-stack-resource-drifts) to see how far the live resources have diverged from the template's understanding of them.

Protect the stateful resource first

• Take an immediate manual snapshot: aws rds create-db-snapshot --db-instance-identifier mydb --db-snapshot-identifier pre-reconcile-YYYYMMDD.
• If the engine supports it, stand up a read replica as a warm standby before making any further changes to the stack.
• Confirm automated backups and the earliest available restore point, in case a snapshot restore ends up being the fallback.

Prepare a non-destructive plan via a change set

• Generate a change set against the current template rather than applying directly: aws cloudformation create-change-set --stack-name my-stack --template-body file://new.yml --change-set-name reconcile-cs --capabilities CAPABILITY_NAMED_IAM.
• Inspect it specifically for any Replace action on the database. If the change set would replace it, stop and rework the template rather than proceeding, this is the single most important check in the whole process.

Reconciling the stack's model of reality with what is actually running

Cutover for anything that does need to move

Where the database tier genuinely has to change (a forced replacement, a major version upgrade), prefer promoting a replica or restoring a snapshot into a new instance, validating it (schema checks, smoke tests) in parallel, and only then repointing the application via its endpoint during a short, controlled cutover, rather than letting a stack update do it as a side effect of resolving the CloudFormation inconsistency.

Worked example

Given a stack where an EC2 Auto Scaling Group was replaced successfully but the RDS instance update rolled back partway, leaving the stack in UPDATE_ROLLBACK_FAILED: first, snapshot the RDS instance immediately. Second, run drift detection to confirm the RDS instance's live properties versus what the template currently declares. Third, generate a change set against a corrected template that sets DeletionPolicy: Retain on the RDS resource and matches its declared properties to the live instance's actual configuration, confirm the change set shows zero actions against the RDS resource. Fourth, apply that change set, since it only touches the already-replaced ASG's dependent resources (for example, security group references) and leaves the database untouched. Fifth, once the stack is back to a clean state, address the original intended change (whatever caused the RDS property to want replacing) as its own follow-up change set, reviewed specifically for that risk.

Trade-offs & pitfalls

• DeletionPolicy: Retain protects against accidental replacement, but if a team forgets to set it before a risky update, that protection simply is not there when it is needed; it has to be a template default for stateful resources, not something added reactively.
• Change sets add real process overhead compared to a direct update, but that overhead is the entire point for a stateful resource; skipping it to move faster is how partial-failure states like this one happen in the first place.
• Resource import is strict: even a small mismatch between the template's declared properties and the live resource's actual configuration means CloudFormation still believes a change is needed, which can silently reintroduce the same class of risk this whole process is meant to avoid.

I would separate shared logic from environment-specific entrypoints. Right now one large root module is forcing unrelated changes into the same file set, which creates merge conflicts and increases blast radius.

New structure

• modules/network
• modules/app
• modules/database
• envs/dev
• envs/staging
• envs/prod

Each environment folder has its own backend and only calls the modules it needs. Shared values like tags or account IDs can live in small local files or a common variable file.

Why this helps

• Two engineers can edit dev and prod in parallel with less conflict
• A change to app compute does not force a network review unless inputs change
• Cross-environment mistakes are harder because each root has its own state

Example
Dev can use t3.small and prod can use m6i.large, but both call the same modules/app. That gives consistency without a monolith. I would also keep module outputs explicit so one environment cannot accidentally consume another environment’s state.

Direct answer

Treat a partially-applied Terraform run as a state-integrity incident, not something you fix by re-running apply. Stop, snapshot the current state before touching anything, and do all reconciliation work in an isolated recovery copy of state so a mistake can't reach production. Then bring state back into agreement with what's actually running, mainly with terraform import, terraform state mv, and terraform state rm, and only promote the recovered state once a plan against it shows zero unexpected creates or destroys.

The recovery framework: which command fixes which symptom

The golden rule underneath all of it: import, state mv, and state rm only ever touch the state file. apply touches the real world. When state is wrong, reach for the state-only commands first; apply (even targeted) is the last step, run only after a refresh-only plan shows the picture is clean.

Worked example: rebuilding state after a partial apply

1. Preserve and isolate. Copy the current (possibly corrupted) state and the last known-good backup to a separate, write-protected location. Lock the real backend so nothing else can touch it while you work.
2. Stand up a recovery workspace pointed at a copy, not production:

terraform init \
  -backend-config="bucket=my-tfstate-recovery-bucket" \
  -backend-config="key=prod-recovery.tfstate" \
  -backend-config="region=us-east-1"

3. Inventory what's actually live. Use the cloud CLI or console to list real resource IDs (instance IDs, ARNs, bucket names) for everything the config is supposed to manage, and build a mapping of state address to real ID.
4. Reconcile with import and state mv:

terraform import module.web.aws_instance.web[0] i-0123456789abcdef0
terraform import aws_s3_bucket.static my-bucket-name

# if an address changed (e.g. moved into a module) after backup:
terraform state mv 'aws_instance.old_name' 'module.web.aws_instance.web[0]'

5. Handle anything state points at that's truly gone, after confirming it wasn't just renamed: terraform state rm <address>.
6. Validate without changing anything:

terraform plan -refresh-only -out=plan.tfplan
terraform show -json plan.tfplan

Review the plan carefully. Any create or destroy action means the mismatch isn't fully resolved yet; a diff limited to attribute drift (tags, a computed field) is expected and can be reconciled by updating config or accepting the read value.
7. Promote once clean: back up production state again, acquire the real lock, and push the recovered state (terraform state push) or copy it into the production backend key. Run terraform plan once more against production; it should show no changes.
8. Close out: release locks, notify the team, rotate any temporary credentials used for the recovery, and write down what happened.

Stale lock vs corrupted state

Everything above assumes Terraform can already acquire the state lock and the problem is what's inside the state file. A stale lock is a different failure mode: the pipeline can't even start, because a prior run crashed (or was killed) mid-apply and never released its lock.

How it shows up. On an S3+DynamoDB backend, a stale lock surfaces as:

Error: Error acquiring the state lock

Lock Info:
  ID:        7f3a1c2e-9e2b-4b7a-9b1a-3f2e7c5a1d90
  Path:      my-tfstate-bucket/prod/terraform.tfstate
  Operation: OperationTypeApply
  Who:       ci-runner@build-42
  Version:   1.x.x
  Created:   2026-07-18 03:14:22 UTC
  Info:

That row lives in the DynamoDB lock table; a crashed apply never gets to delete it, so every later run keeps hitting the same lock.

Before running terraform force-unlock <LOCK_ID>, verify:

• The owning run is actually dead. Check the CI system for the job matching Who and confirm its status is failed or terminated, not still executing.
• The Operation and Created timestamp are consistent with a crash, not a live apply. A Created time that lines up with a known crash a few minutes ago supports a stale lock; a Created time from seconds ago means an apply is genuinely in flight.
• Check with the team (deploy channel, on-call) that nobody else is intentionally mid-apply against this same state, since the lock record alone can't tell you that.
• Snapshot the current state object before unlocking, in case whatever crashed left a partial write behind.

Once all of that checks out, run terraform force-unlock <LOCK_ID>, using the exact ID from the error, against the same workspace.

Why a blind force-unlock is dangerous: the DynamoDB lock exists to serialize writes to the S3 state object. Force-unlock while a run is genuinely still applying leaves two processes that both believe they hold exclusive write access; they can both write to the same state key, the last write silently wins, and the loser's changes vanish. That is exactly how you manufacture the missing or out-of-sync state entries the rest of this answer is about recovering from, so force-unlock is a last resort after verification, not a first response to the error.

Trade-offs, pitfalls, and how the cause changes the fix

The mechanics above are the same regardless of cause, but what actually happened changes which step matters most:

• If the corruption is from an operator accidentally running terraform state rm on the wrong resource, rather than a crash, there's no drift to reconcile from a failed apply, the fix is narrower: re-import just the specific resource(s) the operator removed. The real fix is a guardrail, not a recovery step: gate state rm behind peer review or a break-glass process so a fat-fingered command doesn't silently look identical to a crash.
• If the partial failure came from hitting a cloud provider quota limit mid-apply, the resources created before the limit hit are already correctly in state; the danger is retrying blindly. Retrying the same apply while the quota is still exhausted just fails again on the same resource, or worse, races a partial retry against the resources that already succeeded. Fix the quota first (request an increase, free up existing capacity), confirm with a refresh-only plan, then retry.
• If the failure spans two providers (say the AWS side of an apply succeeded but a paired GCP resource failed), a naive retry re-applies the whole graph, which is only safe if every resource involved is genuinely idempotent to re-apply. Frame this recovery around idempotent retries specifically: confirm each resource type is safe to re-create-or-noop before retrying, and for anything that isn't (a resource whose create isn't idempotent, like one that always generates a new ID), reconcile it manually with import instead of retrying the apply.
• Common wrong turn under pressure: reaching for terraform apply (even -targeted) to "fix" a mismatch. Apply mutates the world; only the state subcommands are safe to use while you don't yet trust what's actually there.

Direct answer

Stop an accidental production destroy with layered guardrails, not one control: a built-in per-resource lock in the config itself, an automated policy check that fails the pipeline if a plan touches a protected resource, and a human approval gate for anything that still gets through. None of these alone is sufficient, a resource-level lock protects one resource but not a whole environment, a policy check is only as good as its rule coverage, and a human approver gets fatigued reviewing the same plan every day. Layered together, each one catches what the others miss.

The guardrail layers

Worked example: policy-as-code and a built-in resource lock

The cheapest guardrail, right in the resource block:

resource "aws_db_instance" "prod" {
  # ...
  lifecycle {
    prevent_destroy = true
  }
}

This alone stops the single most common accident, an unintended destroy of one named resource, without needing any pipeline tooling at all. It doesn't help if the resource is later removed from the config entirely (which sidesteps the lifecycle block), so it's a floor, not a ceiling.

Sentinel policy (Terraform Enterprise), denying any destroy on a prod-tagged resource:

import "tfplan/v2" as tfplan

prod_resources = func() {
  result = []
  for tfplan.resource_changes as rc {
    if rc.change.actions contains "delete" {
      if rc.change.before.tags["environment"] == "prod" {
        result.append(rc)
      }
    }
  }
  return result
}()

main = rule {
  length(prod_resources) is 0
}

OPA/Rego equivalent for a Kubernetes admission flow, denying deletion of a prod-labeled object:

package kubernetes.admission

deny[msg] {
  input.request.operation == "DELETE"
  input.request.object.metadata.labels["env"] == "prod"
  msg := "Deleting a prod-labeled resource requires an approved exception"
}

CI staging: unauthenticated static checks (terraform validate, tflint, checkov) run on every PR; the policy check above runs against the generated plan before any apply job can start; non-prod environments auto-apply on merge; prod applies require a recorded manual approval step.

Trade-offs and pitfalls

• Every added gate slows deploys; apply risk-based gating, strict for prod, lighter for dev and staging, rather than the same friction everywhere, or engineers will look for ways around it.
• Tags are a control surface, not a guarantee: a resource missing its environment tag silently escapes tag-based policy rules, so treat "no tag" as a policy failure in its own right, not as "assume dev."
• prevent_destroy protects a resource that's still in the config; it does nothing once that resource block is deleted from the config entirely, plan-level policy is what catches that case.
• If a destroy does slip through: the recovery path is the same discipline as any state incident, restore from the most recent backup or snapshot (RDS snapshot, EBS snapshot, versioned S3 object), reconcile Terraform state to match what's actually there, and only then resume normal applies. Have that runbook written down before an incident, not during one.
• Policies that are too strict without an exception path get quietly bypassed under deadline pressure; build a reviewable, logged exception mechanism into the policy itself rather than leaving break-glass as an undocumented workaround.