Infrastructure as Code and Automation Questions

Direct answer

Terraform has no built-in transactional rollback because it applies resources one at a time in dependency order, not as a single atomic unit, so "rolling back" is really a set of separate patterns you choose between based on blast radius: reapplying a known-good, version-pinned configuration, restoring a snapshotted state file, or targeted, resource-level remediation. Automated rollback is the right call for reversible, idempotent changes with a clear health signal (a stateless service behind an ASG, a container image swap); a human needs to reconcile by hand whenever the change is destructive, touches stateful data, or the exact post-failure state is ambiguous.

Patterns for walking back a change

Versioned modules and state backups

• Pin every module call to a semantic version tag, so "roll back the infrastructure" can mean "redeploy the previous module version" rather than reconstructing a diff by hand.
• Keep state backend versioning on (S3/GCS object versioning) and snapshot state before every apply, tagged with the pipeline run ID and commit.

Partial rollback versus full rollback

• A full rollback (restoring the entire state file to the last-good snapshot) is the blunt instrument: fast, but it can silently undo any other legitimate change another engineer applied in the meantime, since it doesn't know which parts of the diff are the problem.
• A partial rollback (fixing or re-targeting just the resource(s) that actually broke, using terraform apply -target sparingly, or hand-correcting one resource's config) is narrower and safer when you know exactly what failed, but requires that you've actually diagnosed the failure first, guessing wrong under a full state restore is less catastrophic than guessing wrong under a targeted one.

Reapplying configuration management state after an infra rollback

If a configuration management tool (Ansible, Chef, Puppet) runs on top of Terraform-provisioned instances, rolling back the Terraform layer alone (say, reverting an instance's AMI or instance type) does not automatically put that instance's OS-level configuration back in sync. The CM tool's last run was against the instance in its post-change form; after an infra rollback you need to rerun the CM playbook or run-list against the rolled-back instance so both layers agree again, otherwise you end up with infra that matches the old Terraform state but application configuration that still reflects the change you just reverted.

Keeping a multi-resource logical change atomic

A single logical change that spans several resources, for example provisioning a new database instance together with the IAM policies that grant an application access to it, needs to stay consistent even if one part fails. Two practical approaches: sequence the change so a partial failure is safe by construction (grant the IAM policy before the database exists, so a database-creation failure just leaves an unused policy rather than a database nothing can reach), or make each piece individually idempotent so re-running the apply after a partial failure converges to the same end state rather than erroring on "already exists."

Worked example

Say an apply plans five resources in this order: a security group, a DB subnet group, an RDS instance, an IAM policy, and an IAM role-policy attachment. The first four create successfully; the fifth fails because the policy ARN referenced in the attachment has a typo. Terraform writes state incrementally as each resource completes, not only at the end of the run, so at the moment of failure, state already correctly records the security group, subnet group, RDS instance, and IAM policy as created; only the role-policy attachment is missing. Running terraform plan again reflects this accurately: it proposes creating only the missing attachment, not recreating the other four, because state already matches reality for those. The fix is to correct the ARN and reapply; nothing about the first four resources needs to be touched or reasoned about again. This is the diagnostic step that has to happen before deciding between "just reapply" (safe here, since the failure was config-only and everything else succeeded) and "restore from snapshot" (which would be the wrong call here, since it would needlessly discard four correctly-created resources).

Trade-offs and pitfalls

• Automated rollback assumes the operation is reversible and idempotent, true for stateless compute and image swaps, false for anything with an irreversible side effect (a dropped database column, deleted data, a webhook already fired to an external system). Forcing an automated revert onto that class of change is how you turn one incident into two.
• A blanket "restore the old state file" full rollback is dangerous in a team setting precisely because it can undo unrelated, legitimate concurrent changes, prefer targeted remediation once you've actually diagnosed which resource is broken, as in the worked example above.
• Forgetting the configuration-management layer after an infra-level rollback is a common miss: the infra looks reverted, but the OS-level config on that instance can silently stay in the post-change state until the CM tool is rerun.

I would consume the shared network with data sources or, if the network team publishes outputs, with terraform_remote_state. A data source is Terraform’s read-only lookup for existing infrastructure. I would prefer explicit outputs for IDs, because they are less ambiguous than searching by tags.

What I would check

• The VPC ID matches the expected CIDR, for example 10.20.0.0/16
• The subnet count is what I need, for example 2 private subnets in us-east-1a and us-east-1b
• Subnet tags match my assumptions, such as tier=private and env=prod
• All subnets belong to the same VPC

Fail loudly
I would add variable validation plus precondition checks so the plan stops before apply if the layout is wrong. For example, if I expect exactly 2 private subnets and only find 1, the module should error instead of guessing.

That approach keeps the module reusable, but still safe when the shared network changes.

Direct answer
A strangler-pattern migration: freeze new provisioning through the scripts, pick one resource type at a time, write declarative config that matches what already exists, import the live resource into the new tool's state without recreating it, and only then let the script's write path go dark for that resource type. You retire a script once every resource it used to own is imported, a plan against the new config comes back clean, and nobody still depends on running it manually.

Structured elaboration

What "legacy shell scripts" actually means for the migration
Bash wrapping the cloud CLI and Python wrapping the SDK (boto3, azure-mgmt, etc.) are the two common shapes, and for migration purposes they behave the same way: neither leaves behind a state file, so the IaC tool has no idea these resources exist until you tell it. The first job isn't picking a tool, it's building an inventory.

Phase 1: Inventory and classify
For every script, record:

• What resource(s) it creates or mutates (a name isn't enough, you need the actual resource IDs/ARNs it touches).
• Idempotency: does re-running it error, no-op, or duplicate the resource?
• Side effects beyond provisioning: does it also register the resource somewhere else (DNS, a CMDB, a monitoring config) that the new tool would need to replicate?
• Blast radius: shared resource (a VPC, an IAM role many things depend on) versus isolated (one team's dev bucket).

Order the migration by blast radius, smallest first.

Phase 2: Pilot on one low-risk resource type
Pick something isolated and cheap to redo if wrong (a dev-only S3 bucket, a single security group). Write the declarative config to describe it, then import rather than recreate.

Phase 3: Import without recreating
This is the step teams get wrong: writing config first and letting apply create a duplicate resource next to the one the script made. The safe order is config, import, plan, and only apply if the plan shows zero changes.

Phase 4: Dual-track cutover
Disable the script's write path (comment out the create/update calls, or pull the cron that reruns it) as soon as a resource type is imported, even before the whole estate is converted. Leaving both paths live is how you get drift: someone reruns the old script out of habit and the state file no longer matches reality.

Phase 5: Retirement criteria
Retire a script for a resource type when all of these hold:

• Every resource that script used to own has been imported and a plan against it is clean.
• The script's write path has been disabled for at least a full deploy cycle with no incidents traced back to needing it.
• Nobody on the team can name a workflow that still calls it manually.
• A rollback reference exists: the script itself, kept read-only in version control, in case you need to reconstruct what it used to do.

Worked example
Say a bash script provisioned an S3 bucket with aws s3api create-bucket --bucket acme-prod-assets --region us-east-1. The safe Terraform import sequence:

# 1. Write the resource config that should describe the existing bucket
resource "aws_s3_bucket" "assets" {
  bucket = "acme-prod-assets"
}

# 2. Declare the import (Terraform 1.5+ import block, no separate CLI step needed)
import {
  to = aws_s3_bucket.assets
  id = "acme-prod-assets"
}

Run terraform plan. If the plan shows changes (say, the bucket has versioning enabled in reality but your config didn't set it), that's the config being wrong, not the import: fix the config until plan shows zero diff, then apply. Only after that is the resource genuinely under Terraform's control.

If the resource later gets refactored into a module, use a moved block instead of destroying and recreating it, so state history is preserved:

moved {
  from = aws_s3_bucket.assets
  to   = module.storage.aws_s3_bucket.assets
}

Trade-offs & pitfalls

• Import only checks that the ID exists; it doesn't validate that your config matches every real attribute. A plan that isn't empty after import means your config is wrong, and applying it anyway can modify or, in the worst case, replace the live resource.
• The dual-track period is the highest-risk window: a stray cron rerun of the old script can silently drift the state Terraform thinks it owns. Disable the script's write path per resource type as soon as it's imported, don't wait for the whole estate to finish.
• Not every script maps cleanly to a resource. Ones with side effects (paging a team, writing to an external system) need that side effect re-homed somewhere (a CI step, a webhook) rather than papered over.
• Rewriting from scratch instead of importing is sometimes the right call: when the existing resource's config has drifted so far from any documented baseline that reproducing it faithfully would just be encoding technical debt. In that case, treat it as a planned recreation, with the downtime/cutover that implies, rather than an import.

Direct answer

On my last project I codified an AWS microservices platform, a set of backend services running on managed servers with their own database, load balancer, and access controls (VPC, EKS, RDS, IAM, ALB, monitoring) in Terraform, using versioned reusable modules composed per environment, remote state (the file Terraform uses to track what it created, kept separate per environment so a mistake in one can't touch another) isolated per environment, secrets pulled from AWS Secrets Manager and Parameter Store rather than stored in code, and a PR-based workflow where a machine-generated plan had to be reviewed and approved before an apply job with a separate, more privileged role could run.

How I structured it

Module layout and versioning

• Modules lived in a private registry: vpc, eks, rds, iam, alb, monitoring, each with a narrow set of inputs/outputs and no hidden side effects.
• Root configurations per environment composed these modules and pinned each one to a semantic version tag (for example vpc ~> 2.3), so a change to a module's source could not silently change an environment that had not explicitly bumped its pin.

Environment separation and state

• Each environment (dev, staging, prod) had its own remote state file in an S3 backend with a DynamoDB lock table, keyed roughly as s3://infra-state/{env}/{component}.tfstate.
• Isolating state per environment (rather than one shared state with workspaces) meant a mistake in dev could not touch prod's state, and the blast radius of a single apply was limited to one component in one environment.

Secrets handling

• Database credentials lived in AWS Secrets Manager, app configuration lived in Parameter Store encrypted with KMS.
• Terraform read them at apply time through data sources (for example aws_secretsmanager_secret_version) rather than having them typed anywhere in .tf files or CI variables.
• The CI runner itself never held a long-lived key: it assumed a role via OIDC scoped to the minimum permissions needed for that environment's apply.

Change approval and safe apply

• Every change went through a PR. CI ran terraform plan and posted the JSON plan (with sensitive values redacted) as an artifact, plus a readable summary on the PR.
• A change needed sign-off from both the infra owner and the owning service team before the apply stage would even unlock.
• Apply ran as a separate pipeline stage using a more privileged, MFA-gated role for prod, and every apply was logged to CloudTrail.
• Destructive changes (anything showing a resource replacement or delete in the plan) required an explicit manual confirmation step, and we took a fresh snapshot or backup first.

Worked example

Concretely: adding a read replica to an existing RDS instance meant one PR that touched only the rds module's call site, a plan that showed one resource create and zero destroys, review from the database owner since it touched a stateful resource, and an apply that ran with the prod-apply role only after both approvals landed. Because the module was version-pinned and state for that component was isolated, the blast radius of that single PR was exactly one RDS resource in one environment: nothing else in the account could be affected by that apply.

Trade-offs and pitfalls

• Splitting state by component reduces blast radius but adds cross-stack coordination cost (remote state lookups or SSM parameters to pass values between components); too many tiny state files becomes its own operational burden.
• A plan-then-approve workflow is only as safe as the reviewers actually reading the plan. Without a policy-as-code gate (something like OPA/Rego or Sentinel evaluating the plan JSON for guardrails such as "no public security groups" or "no unencrypted volumes"), review can degrade into rubber-stamping on a busy day.
• Reading secrets via data sources at apply time keeps them out of source control, but the values still land in the Terraform state file in plaintext, so state encryption and tightly scoped state-read IAM matter just as much as the CI-side handling.

Direct answer

Split this into three layers: a versioned, registry-published modules repo that no environment applies directly; per-account (or per-environment) root configurations, each with its own remote state backend and key; and a promotion pipeline that moves a module version bump from dev through staging to production behind plan review and a human approval gate. Cross-account access is done with short-lived STS role assumption from CI (OIDC, not long-lived IAM keys) into a narrowly-scoped deployer role in each target account. Promotion is a version bump plus a gated pipeline stage, not a copy-paste of resources between account configs.

Repository and state layout

• modules/ (its own repo or clearly separated path): network/, iam/, compute/, eks/, rbac/. Each module is small, single-responsibility, semantically versioned, and published to a module registry (Terraform Registry, or an internal artifact store) so consumers pin a version instead of tracking a branch.
• envs/ (one repo or directory per account or per environment): org-utilities/, dev/, staging/, prod/. Each has its own backend configuration. org-utilities/ is the shared-services account, not just another spoke, see "Network topology across accounts" below for what it holds and how it's wired to the rest.
• A platform repo owning the shared CI/CD pipeline definitions and policy-as-code rules that every env repo inherits.

Remote state, one per account/environment, not shared:

• S3 (or GCS) bucket + DynamoDB lock table per account/region, named and scoped to that account, encrypted with a KMS key owned by a central security account.
• IAM policy on the state bucket/table restricts access to the CI deployer role for that account and the owning team, nobody else.

graph LR
  P["Platform repo: module registry + shared CI/CD"] --> DEVR["Dev env repo"]
  P --> STGR["Stage env repo"]
  P --> PRODR["Prod env repo"]
  DEVR --> DEVS["Dev state backend"]
  STGR --> STGS["Stage state backend"]
  PRODR --> PRODS["Prod state backend"]
  DEVS --> PLANROLE["tf-plan role per account, assumed via OIDC on any PR"]
  STGS --> PLANROLE
  PRODS --> PLANROLE
  PLANROLE --> GATE["Manual approval gate"]
  GATE --> APPLYROLE["tf-apply role per account, assumed via OIDC only for approved runs"]
  APPLYROLE --> PRODACC["Prod AWS account"]

Network topology across accounts

• Every spoke account (dev/, staging/, prod/, and any future team account) runs the same network module from the registry: one VPC, subnets, and route tables, laid out identically. The only thing that differs per account is the vpc_cidr input.
• CIDR ranges are allocated centrally, not picked ad hoc per account: a single CIDR registry (a cidr-allocations.tf map or table in the platform repo, or an actual AWS IPAM pool, a service that centrally allocates and tracks IP address ranges, if you want enforcement instead of just documentation) assigns each account a non-overlapping block up front, e.g. dev = 10.0.0.0/16, staging = 10.1.0.0/16, prod = 10.2.0.0/16, org-utilities = 10.255.0.0/16. Non-overlapping ranges are what make it possible to peer or Transit-Gateway-attach any two accounts later without a re-IP.
• org-utilities/ is the shared-services account: it owns a Transit Gateway (or a hub VPC with peering, for a smaller org) that every spoke account attaches to; centralized egress (a NAT/firewall fleet spokes route their outbound traffic through, so you're not paying for and securing a NAT gateway fleet in every account); centralized DNS (a Route 53 Resolver with forwarding rules shared to spokes via AWS RAM, Resource Access Manager, a service for sharing resources across accounts, not memory, so on-prem and private-zone resolution is consistent everywhere); and centralized logging/monitoring aggregation (VPC Flow Logs, CloudTrail, and metrics from every spoke shipped to one log/metrics account).
• Wiring: each spoke's network module output includes its VPC ID, and a small tgw-attachment module in that spoke requests attachment to the shared Transit Gateway in org-utilities/; org-utilities/ accepts the attachment and associates it with the shared route table, and each spoke's route tables get a route pointing anything outside their own CIDR at the Transit Gateway.

Cross-account access

• Each target account has two deployer roles, not one: a tf-plan role with read-only/describe permissions (plus any iam:PassRole scoped only to what plan-time data sources need) that any CI job on any branch can assume, and a separate tf-apply role with the actual write permissions, scoped to the resource types and namespaces that account's environment needs, never a blanket admin role.
• CI authenticates to a central identity (GitHub Actions OIDC, or similar). Every PR assumes the tf-plan role via STS to run terraform plan and post the diff, this can run unattended on any branch since it can't change anything. Only a merge to the environment's protected branch, after the required human approval on that environment's deployment gate (a GitHub Environment with required reviewers, or a Terraform Cloud/Enterprise run task), is allowed to assume the tf-apply role and run terraform apply. The trust policy on tf-apply itself enforces this, it only accepts assumption from the specific CI workflow/job context tied to an approved deployment, not from an arbitrary PR run. No long-lived static credentials live in CI configuration either way.
• SCPs (Service Control Policies, org-wide permission ceilings that apply even if a role's own IAM policy allows more) at the organizational-unit level are an additional guardrail on top of IAM: even a misconfigured tf-apply role can't exceed what the OU-level SCP allows.

Module design and reuse

• Modules expose typed inputs, sensible defaults, and stable outputs; each is reviewed and tested (terraform test or Terratest) before a new version tag is cut.
• Environment root configs pin an explicit module version constraint, so a module fix doesn't silently propagate to prod the moment it merges, prod picks it up on its own promotion cycle.

Promotion pipeline, dev to prod

• CI runs plan on every PR to an env repo and posts the diff for review.
• Merge to the dev environment's branch auto-applies to the dev account.
• Promotion to staging and then prod is a version bump (of the module pin, or of a shared "release" tag) plus a pipeline stage that requires a passing plan, automated policy checks, and a manual approval before apply runs against that account.
• Prefer immutable-infrastructure patterns (new instances/versions rather than in-place mutation) and blue/green or canary rollout where the resource type supports it, so a bad compute promotion is a matter of shifting traffic back, not an emergency Terraform surgery.
• Rolling back a bad Terraform promotion itself, not just compute traffic: Terraform has no native "undo" for state. The default path is a forward-fix: re-run the pipeline against the last-good git tag or module version pin, so a normal plan/apply reconciles infrastructure back to that prior declared config, and treat that as the standard rollback, not a special emergency procedure. If state itself got corrupted, or the bad change can't be cleanly re-declared forward, fall back to restoring the state object's previous version from the backend bucket's object versioning (kept enabled on the state bucket specifically for this) and re-planning against the restored state to reconcile real infrastructure with it. Blue/green covers the common case of a bad compute rollout; it does not cover a bad promotion to something stateful like an RDS parameter, an IAM policy, or a networking change, those need the tag-revert-and-reapply path above.

If you're using CloudFormation instead of Terraform

The topology problem is the same, roll one change across many AWS accounts and regions, but CloudFormation gives you two different native mechanisms with different partial-failure behavior:

• Nested stacks: a parent stack in one account/region calls child stacks as part of its own dependency graph. This does not natively fan out across accounts or regions on its own, you'd still need a pipeline stage (for example, one CodePipeline stage per account) to run the same template repeatedly. Because it's all one stack transaction, a failure rolls back the parent and any children it had already started updating within that single operation.
• StackSets: an administrator account deploys one template as "stack instances" into many target accounts and regions in a single StackSet operation. Each account's per-account update still goes through its own change set under the hood, so you get the same create/update/delete preview terraform plan gives you, just scoped to that one account. Partial failure is fundamentally different from nested stacks: a StackSet operation does not roll back instances that already succeeded, each stack instance is its own independent stack update. If instance 12 of 50 fails, instances 1 through 11 stay updated, instance 12 is where the operation stops (or continues, depending on settings), and you're left reconciling a fleet in a mixed state rather than a clean all-or-nothing rollback.
• Limiting blast radius on a StackSet rollout: set a low FailureToleranceCount (or FailureTolerancePercentage), for example 0 or 1, so the operation halts on the first failure instead of continuing through dozens of accounts; deploy in waves by targeting a canary OU first before the rest of the organization; and use MaxConcurrentCount (or MaxConcurrentPercentage) to cap how many accounts update in parallel, so a bad template can't hit every account before anyone notices.

Governance and guardrails

• Policy-as-code (OPA/Gatekeeper-style checks, or Terraform Cloud/Sentinel policy checks) enforces tagging, encryption, and network restrictions before apply is allowed to run.
• Scheduled drift detection (periodic plan runs with alerting on any diff) catches out-of-band changes; route safe fixes to automated remediation and anything ambiguous to a human ticket.
• Cost guardrails: tag-based budgets and alerts, pre-deploy cost estimates in CI, automated shutdown policies for non-production resources outside business hours.

Trade-offs and pitfalls

• Mono-repo for modules simplifies discoverability but needs stricter CI (required reviews, version-tag discipline) to avoid noisy, hard-to-review changes landing across many modules at once.
• Centralizing state encryption keys under one security account increases control but makes that account a higher-value target, mitigate with strict key rotation, access logging, and alerting on key usage, not just "it's centralized so it's secure."
• Strict SCPs reduce blast radius but slow down legitimate one-off changes; pair them with a documented, fast exception process rather than leaving teams to work around them informally.
• The most common real failure mode in this kind of setup isn't a Terraform bug, it's an overly broad deployer role IAM policy that lets a dev-account CI job touch prod resources it was never meant to reach; review those policies as carefully as the Terraform code itself.

Team responsibilities

• Platform team: owns the module registry, shared CI/CD, central IAM/KMS boundaries, and policy-as-code.
• Security and governance: defines the policies, runs audits, owns SCPs and account-level guardrails.
• Application teams: own their environment repos, build infrastructure from approved modules, and are accountable for their own runtime operations and cost.
• Shared ops: on-call for platform-wide and cross-account incidents, first responder for drift and security alerts.