Infrastructure as Code and Automation Questions

Direct answer

Decouple the schema change from the infra rollout and sequence it in stages using the expand-contract pattern: expand first (an additive, backward-compatible schema change applied through the IaC/migration pipeline while the old application code keeps running unchanged), then migrate data, then only contract (drop or tighten the old shape) once the new application version has been fully rolled out and validated against the new schema. The fallback if something goes wrong partway through is usually to just stop and hold at whatever phase you are in and flip a feature flag off, because every step up through migrate is additive and non-destructive; nothing forces an emergency down-migration unless you jump straight to contract before it is safe.

Structured elaboration

The expand-contract pattern

• Expand: add the new column, table, or index in a way that does not require old code to change, nullable or defaulted, no rename, no drop.
• Migrate: backfill or dual-write so the new shape is populated while the old shape is still being read and written by the currently deployed application.
• Contract: once the new application code is fully rolled out and only depends on the new shape, remove the old column or constraint in a later, separate release.

Sequencing against the infra rollout

The schema migration and the infra/app rollout are two different releases, not one. The expand step ships through the IaC/migration pipeline on its own, ahead of any application change that depends on it. The application code that reads or writes the new column stays behind a feature flag even after the column exists, so the schema release and the app release are decoupled: you can flip the flag on independently of any deploy, and flip it back off instantly if something looks wrong, without touching the database again.

Comparing rollout strategies against a shared, stateful database

Long-running migrations on large tables

For a table too large to migrate in one blocking DDL, use an online schema-change approach: either the database's own online DDL if the engine and change support it, or a chunked backfill job that processes bounded batches (for example by primary-key range) with built-in throttling to bound replication lag and lock contention. Each batch should be written so it is idempotent, re-running a batch that already succeeded should be a no-op, so the job can pause and resume safely instead of needing to restart from scratch.

Worked example

Say we need to add a NOT NULL orders.shipped_at column, no default, to a 50 million row table.

1. Expand: add shipped_at as nullable, no default. This is an additive, non-blocking change with no dependency on any read path changing.
2. Backfill: run a batched UPDATE in chunks of 10,000 rows ordered by primary key, each batch scoped as WHERE shipped_at IS NULL AND id BETWEEN :start AND :end. That bounds every single transaction's lock and redo footprint to 10,000 rows regardless of total table size, and because the WHERE clause only matches unfinished rows, re-running any batch that already completed touches zero rows, so the job is safely resumable after a failure or pause. At 10,000 rows per batch, 50 million rows means roughly 5,000 batches; that number is only there to show the backfill decomposes into a bounded, resumable unit of work, it is not a timing claim.
3. Dual-write: the new application code, behind a feature flag, writes shipped_at on every new order while still tolerating it being null on read for older rows.
4. Validate: confirm there are zero unexpected NULL rows outside the ones that legitimately have not shipped yet, before proceeding.
5. Contract: only after the new application version is fully rolled out and the flag has been on and stable, add the NOT NULL constraint in its own, separate release, and only then remove any old fallback code path that tolerated null.

Trade-offs & pitfalls

• "Rollback" here mostly means flipping the feature flag off and pausing, not a destructive down-migration, because everything through the migrate step is additive. Keep an explicit rollback script for the rare case an expand step itself needs reverting, for example dropping the new column, which is only safe precisely because nothing depends on it being NOT NULL yet.
• Coordinating flag state with schema state is itself a source of bugs: flipping the flag on before the backfill has finished sends null values into code paths that were not written to expect them.
• Canarying the app version alongside a shared database only tells you about app-level regressions; canary and control traffic hit the exact same schema, so schema-level correctness has to be validated independently, during the migrate phase, not inferred from canary metrics.

Direct answer

All three are lifecycle meta-arguments that override Terraform's default create/update/destroy behavior for a specific resource. create_before_destroy reorders a replacement so the new resource is created before the old one is destroyed. prevent_destroy makes any plan that would delete the resource fail outright. ignore_changes tells Terraform to stop comparing specific attributes against configuration, so changes made outside Terraform to those fields don't get reverted or flagged.

What each one does

Worked example

resource "aws_db_instance" "primary" {
  identifier        = "app-primary"
  engine            = "postgres"
  instance_class    = "db.r6g.large"
  allocated_storage = 100

  lifecycle {
    prevent_destroy = true
    ignore_changes  = [password]
  }
}

Here prevent_destroy stops an accidental terraform destroy (or a refactor that renames/removes this block) from deleting the production database, and ignore_changes = [password] accepts that the password is rotated outside Terraform (by a Secrets Manager rotation Lambda, for instance) without Terraform trying to reset it back to whatever value is in state on every apply.

Trade-offs and pitfalls

• create_before_destroy and prevent_destroy are opposites in intent (one enables safe replacement, the other blocks replacement/deletion entirely) and shouldn't be combined on the same resource without thinking through what "replace" even means when destroy is also forbidden.
• Removing prevent_destroy to allow a legitimate teardown is a manual code change and re-apply, budget time for it rather than treating decommissioning as instant.
• ignore_changes is best scoped to the narrowest possible attribute list and documented inline with why it's there; treat it as a deliberate, reviewed exception, and pair it with periodic drift detection so ignored fields are still checked occasionally, just not fought over on every apply.

Direct answer

Moving a patched-in-place VM fleet to an immutable image model means separating "how the image is built" from "how the fleet is rolled over": build and validate a versioned image with a tool like Packer (a CLI that scripts building a machine image from a template), roll it out behind a second autoscaling group with a gradual canary before a full cutover rather than a single big-bang swap, and pull anything stateful (databases, session data, files that must survive an instance being replaced) off the instance entirely so instances become disposable and the rollout doesn't have to babysit data.

Getting there

Building and validating the image

Use Packer to bake a versioned AMI: install packages, apply the same hardening the current fleet has via provisioners, and run automated image tests (boot check, service health, a smoke test hitting the app) before an AMI is eligible to deploy. Nothing instance-specific gets baked in, secrets and per-instance config are still fetched at boot time from Secrets Manager/SSM, exactly as they would be for the existing fleet, only the delivery mechanism for "what's installed" moves from a patch run to a new image build.

Rolling the fleet over without a big-bang cutover

graph LR
  A[Base OS image] --> B[Packer build + provisioners]
  B --> C[Automated image tests]
  C --> D[New Auto Scaling Group on new AMI]
  D --> E[Canary: shift small percent of traffic]
  E --> F{Healthy?}
  F -->|Yes| G[Full cutover: shift remaining traffic]
  F -->|No| H[Rollback: route back to old ASG]
  G --> I[Drain and terminate old instances]

Stand up a second ASG on the new AMI behind the same load balancer as the existing fleet, and start by shifting only a small slice of traffic to it (a canary), watching health checks and error-rate signals before widening further. Doing the migration itself as a canary rollout, not just the day-to-day deploys after migration is done, is what avoids a big-bang cutover: a bad image only affects the canary slice, not the whole fleet, and you can route traffic back to the untouched old ASG immediately if it isn't healthy.

Handling the stateful pieces

• Databases: move to a managed service (RDS) if not already there, or use replication plus a coordinated cutover if migrating the DB itself.
• Local/session state: externalize to something the new, disposable instances can all reach, S3 or EFS for files, Redis for session data, so any instance can be swapped without losing anything tied to it.
• Anything acting as a "leader" or holding a lock: move that coordination to something external (DynamoDB, etcd, Consul) instead of relying on a specific instance's identity, since that identity won't survive the cutover.

Worked example

Say the existing fleet is a set of long-running EC2 instances patched in place by a configuration management tool. The team builds a new AMI with Packer that layers the same packages via provisioners, with nothing instance-specific baked in. They stand up a second ASG on that AMI behind the existing load balancer, initially sized to take roughly a tenth of traffic, a chosen canary slice, not a measured result, while the old ASG keeps the rest. Health checks and error-rate alarms on the new ASG are watched before the split is widened in further steps, and only once the new ASG has fully absorbed traffic is the old ASG scaled to zero and terminated. In this case session data was already externalized to Redis and the primary database was already RDS before the migration started, so neither had to move as part of the cutover, only the compute layer changed.

Trade-offs and pitfalls

• Immutable, image-based fleets trade a fast "SSH in and fix it" patch for a slower "bake, test, redeploy" cycle, a real cost during an urgent security patch if the image build pipeline itself is slow or manual; mitigate by keeping that pipeline fast and fully automated rather than something someone has to babysit.
• Running two ASGs side by side during the migration roughly doubles compute spend for the duration of the cutover; that's usually an acceptable trade for safety, but size and time-box it deliberately rather than leaving both running indefinitely.
• Anything still implicitly relying on instance identity (a hardcoded IP allowlist, a cron job assuming it's the same box every day) breaks silently during cutover unless it's found during the inventory step; incomplete discovery of what depends on the old fleet's stability is usually the biggest risk in this migration, not the image-build mechanics themselves.

I would put both human review and automation around the plan. A Terraform plan is a preview of create, update, destroy, and replace actions. Replace is especially risky because Terraform deletes and recreates a resource.

Guardrails

• Run terraform fmt -check and terraform validate in CI
• Save the plan as an artifact and require approval before apply
• Fail the pipeline if the plan includes unexpected destroys or replacements on critical resources
• Use policy checks for things like public exposure, open security groups, or deletion of databases
• Add prevent_destroy to the few resources that should almost never be removed

Example
If the plan shows aws_db_instance.main will be replaced because of an engine change, I would force manual review from an engineer and, ideally, a service owner. If the plan only adds an autoscaling instance, that can follow the normal path.

This catches surprises before users feel them, which is the real goal.

Direct answer

Terraform has no built-in transactional rollback because it applies resources one at a time in dependency order, not as a single atomic unit, so "rolling back" is really a set of separate patterns you choose between based on blast radius: reapplying a known-good, version-pinned configuration, restoring a snapshotted state file, or targeted, resource-level remediation. Automated rollback is the right call for reversible, idempotent changes with a clear health signal (a stateless service behind an ASG, a container image swap); a human needs to reconcile by hand whenever the change is destructive, touches stateful data, or the exact post-failure state is ambiguous.

Patterns for walking back a change

Versioned modules and state backups

• Pin every module call to a semantic version tag, so "roll back the infrastructure" can mean "redeploy the previous module version" rather than reconstructing a diff by hand.
• Keep state backend versioning on (S3/GCS object versioning) and snapshot state before every apply, tagged with the pipeline run ID and commit.

Partial rollback versus full rollback

• A full rollback (restoring the entire state file to the last-good snapshot) is the blunt instrument: fast, but it can silently undo any other legitimate change another engineer applied in the meantime, since it doesn't know which parts of the diff are the problem.
• A partial rollback (fixing or re-targeting just the resource(s) that actually broke, using terraform apply -target sparingly, or hand-correcting one resource's config) is narrower and safer when you know exactly what failed, but requires that you've actually diagnosed the failure first, guessing wrong under a full state restore is less catastrophic than guessing wrong under a targeted one.

Reapplying configuration management state after an infra rollback

If a configuration management tool (Ansible, Chef, Puppet) runs on top of Terraform-provisioned instances, rolling back the Terraform layer alone (say, reverting an instance's AMI or instance type) does not automatically put that instance's OS-level configuration back in sync. The CM tool's last run was against the instance in its post-change form; after an infra rollback you need to rerun the CM playbook or run-list against the rolled-back instance so both layers agree again, otherwise you end up with infra that matches the old Terraform state but application configuration that still reflects the change you just reverted.

Keeping a multi-resource logical change atomic

A single logical change that spans several resources, for example provisioning a new database instance together with the IAM policies that grant an application access to it, needs to stay consistent even if one part fails. Two practical approaches: sequence the change so a partial failure is safe by construction (grant the IAM policy before the database exists, so a database-creation failure just leaves an unused policy rather than a database nothing can reach), or make each piece individually idempotent so re-running the apply after a partial failure converges to the same end state rather than erroring on "already exists."

Worked example

Say an apply plans five resources in this order: a security group, a DB subnet group, an RDS instance, an IAM policy, and an IAM role-policy attachment. The first four create successfully; the fifth fails because the policy ARN referenced in the attachment has a typo. Terraform writes state incrementally as each resource completes, not only at the end of the run, so at the moment of failure, state already correctly records the security group, subnet group, RDS instance, and IAM policy as created; only the role-policy attachment is missing. Running terraform plan again reflects this accurately: it proposes creating only the missing attachment, not recreating the other four, because state already matches reality for those. The fix is to correct the ARN and reapply; nothing about the first four resources needs to be touched or reasoned about again. This is the diagnostic step that has to happen before deciding between "just reapply" (safe here, since the failure was config-only and everything else succeeded) and "restore from snapshot" (which would be the wrong call here, since it would needlessly discard four correctly-created resources).

Trade-offs and pitfalls

• Automated rollback assumes the operation is reversible and idempotent, true for stateless compute and image swaps, false for anything with an irreversible side effect (a dropped database column, deleted data, a webhook already fired to an external system). Forcing an automated revert onto that class of change is how you turn one incident into two.
• A blanket "restore the old state file" full rollback is dangerous in a team setting precisely because it can undo unrelated, legitimate concurrent changes, prefer targeted remediation once you've actually diagnosed which resource is broken, as in the worked example above.
• Forgetting the configuration-management layer after an infra-level rollback is a common miss: the infra looks reverted, but the OS-level config on that instance can silently stay in the post-change state until the CM tool is rerun.