Infrastructure Automation with Terraform/CloudFormation Questions

Writing production-quality infrastructure code. Topics include module design, variable validation, error handling, conditional logic, dynamic blocks, outputs for dependent systems, testing infrastructure changes, and documenting code. Writing code that's maintainable for other engineers and handles edge cases gracefully.

EasyTechnical

75 practiced

Explain Terraform lifecycle meta-arguments such as `create_before_destroy`, `prevent_destroy`, and `ignore_changes`. Provide examples of when each should be used and describe their risks and operational effects during apply and destroy operations.

Sample Answer

**Brief overview**Terraform lifecycle meta-arguments modify how resource replacements and updates are handled: create_before_destroy, prevent_destroy, and ignore_changes. As a DevOps engineer you use them to protect state, avoid downtime, and permit safe manual edits — but each has operational trade‑offs.

**create_before_destroy**- Purpose: ensure a new resource is created before the old one is destroyed (useful for immutable infra).- Example:

hcl

lifecycle {
  create_before_destroy = true
}

- Use when replacing load balancers, VMs with ephemeral IPs, or DNS records to minimize downtime.- Risks: can exceed provider quotas (duplicate resources), may fail if unique names are required.

**prevent_destroy**- Purpose: blocks Terraform from destroying a resource unless overridden.- Example:

hcl

lifecycle {
  prevent_destroy = true
}

- Use for critical data stores, production databases, or sensitive IAM roles.- Operational effect: terraform plan/apply will error on any destroy action; to destroy you must remove the meta-arg or use -target/force flag — careful with automation.

**ignore_changes**- Purpose: tells Terraform to ignore specific attribute diffs (often when managed externally).- Example:

hcl

lifecycle {
  ignore_changes = [metadata[0].annotations, tags]
}

- Use when Kubernetes injects annotations, or autoscaling modifies instance count.- Risks: drift can accumulate; hidden diffs may mask real config divergence.

**Summary of effects during apply/destroy**- create_before_destroy: apply creates replacement then destroys old; destroy behaves normally.- prevent_destroy: apply errors if destruction required; destroy is blocked.- ignore_changes: apply won’t change ignored attributes even if plan shows diffs; destroy unaffected.

Always document reasons in code and limit scope to specific resources/attributes.

HardTechnical

90 practiced

Provide a concrete Terraform plan for migrating state from one S3 backend to another with minimal downtime and without losing resources. Include steps for locking, snapshotting, migrating the state file, validating resource addresses, and rolling back if the migration introduces issues.

Sample Answer

**Plan summary (goal):** Move Terraform state from S3 bucket A to bucket B with DynamoDB locking, ensure snapshot, validate addresses, and be able to roll back with minimal downtime.

**Pre-reqs**- AWS CLI + Terraform installed- IAM perms to S3, DynamoDB, and KMS used for state- New S3 bucket (bucket-b) and DynamoDB table (tf-locks) created

**Steps**

1. Create locking table & bucket- Create DynamoDB lock table:

bash

aws dynamodb create-table --table-name tf-locks --attribute-definitions AttributeName=LockID,AttributeType=S \
--key-schema AttributeName=LockID,KeyType=HASH --provisioned-throughput ReadCapacityUnits=5,WriteCapacityUnits=5

- Ensure bucket-b exists and correct encryption/policies.

2. Snapshot current state (immutable backup)- Lock current state by creating a manual lock item (optional) OR just ensure no active runs.- Download latest state:

bash

aws s3 cp s3://bucket-a/path/terraform.tfstate ./terraform.tfstate.snapshot

- Store snapshot in versioned safe location:

bash

aws s3 cp ./terraform.tfstate.snapshot s3://bucket-a/backups/terraform.tfstate.$(date -u +%Y%m%dT%H%M%SZ)

3. Validate resource addresses locally- In workspace with current backend config, run:

bash

terraform state list > state-list.txt

- Inspect addresses for unusual modules, count resources:

bash

wc -l state-list.txt

- Optionally run `terraform plan` (read-only) to confirm no drift.

4. Migrate backend with locking and minimal downtime- Update backend configuration in terraform backend block to point to bucket-b and dynamodb table.- Initialize and migrate state atomically:

bash

terraform init -reconfigure -migrate-state

Terraform will acquire locks (using new DynamoDB table if configured) and transfer state from old backend to new backend.

5. Post-migration validation- Immediately run:

bash

terraform state list > new-state-list.txt
diff state-list.txt new-state-list.txt
terraform plan -refresh-only

- Confirm identical resource addresses and zero-change plan.

6. Rollback procedure (if issues)- Re-lock by preventing users from running Terraform (notify, CI pause).- Restore snapshot to original bucket:

bash

aws s3 cp ./terraform.tfstate.snapshot s3://bucket-a/path/terraform.tfstate --acl bucket-owner-full-control

- Revert backend config to original S3/DynamoDB and run:

bash

terraform init -reconfigure -migrate-state

This moves state back. Validate with `terraform plan`.

**Notes & trade-offs**- Using `terraform init -migrate-state` is preferred to avoid manual parsing/pushing.- Ensure CI pipelines updated to use new backend config and DynamoDB locking.- Test migration in staging with same resource count before production.- Keep snapshots and enable S3 versioning and MFA-delete for extra safety.

MediumTechnical

121 practiced

When and how should you use CloudFormation custom resources (Lambda-backed or other) to extend functionality? Describe common failure modes for custom resources, how to design idempotent custom resources, and strategies to test and monitor them in production.

HardSystem Design

62 practiced

Propose a guardrail and policy system to prevent accidental destroy of production resources when using Terraform and CloudFormation. Include policy-as-code examples (Sentinel, OPA/Gatekeeper), CI checks, manual approvals, tagging protections, and how to recover quickly if an accidental destroy occurs.

Sample Answer

**Approach — goals**- Prevent accidental destroys of production resources while keeping dev agility.- Multi-layered: IaC lint/policy -> CI gate -> runtime protections (tags, cloud safeguards) -> manual approvals -> fast recovery playbook.

**Policy-as-code examples**- Sentinel (Terraform Enterprise) — forbid destroy on prod-tagged resources:

hcl

# sentinel
import "tfplan/v2" as tfplan

prod_resources = func() {
  r = []
  for _, a in tfplan.resources {
    if a.mode is "managed" {
      for _, inst in a.instances {
        if inst.attributes.tags["environment"] == "prod" {
          r.append({address: a.address, change: inst.change})
        }
      }
    }
  }
  return r
}()

# disallow any resource with prod tag that will be destroyed
main = rule {
  all prod_resources as pr {
    pr.change.actions not contains "delete"
  }
}

- OPA / Gatekeeper (K8s example) — deny deletion if label env=prod:

rego

package kubernetes.admission

deny[msg] {
  input.request.operation == "DELETE"
  input.request.object.metadata.labels["env"] == "prod"
  msg = "Deleting prod resources is denied without approval"
}

**CI checks & pipeline**- Pre-merge: terraform validate, tflint, checkov/terrascan for security, unit tests.- PR policy: Terraform plan must be posted; automated script scans plan JSON for "delete" on prod-tagged resources and fails PR.- Staged apply: non-prod auto-apply; prod applies require manual approval step (GitHub/GitLab approval or Terraform Cloud run approvals).

**Tagging & cloud safeguards**- Enforce mandatory tags via policy (owner, environment, cost-center).- Cloud-level protections: AWS termination protection (EC2), Azure soft-delete/locks, GCP IAM conditions to prevent resource deletion unless via privileged service account.- Least-privilege: separate service accounts for plan vs apply; apply account requires step-up MFA/approval.

**Manual approvals & audits**- Require human approval for any plan that contains deletes or taints for resources with environment=prod.- Approval recorded in audit log (Terraform Cloud runs, PR history).

**Fast recovery playbook**- Automated runbook: snapshot/AMI, RDS snapshots, EBS volume snapshot, S3 versioning enabled.- If accidental destroy: 1) Quarantine pipeline (revoke apply SA keys), 2) Recreate infra from last successful state: use Terraform state backup (remote state with versioning), run terraform apply with state restored, or restore from snapshots, 3) Data restore: restore DB from latest snapshot, reattach volumes, 4) Post-mortem and add stricter guardrail if gap found.

**Trade-offs**- More checks = slower deploys. Use risk-based gating: blocking for prod, lightweight for lower envs.- Keep policies testable and versioned; simulate policies in CI to avoid surprises.

This layered system balances safety, recoverability, and operational flow expected from a DevOps engineer.

EasyTechnical

136 practiced

What should be included in a Terraform module README to make it production-quality and maintainable by other engineers? Provide a checklist covering inputs, outputs, examples, versioning notes, prerequisites, and testing instructions.

Unlock Full Question Bank

Get access to hundreds of Infrastructure Automation with Terraform/CloudFormation interview questions and detailed answers.

Join thousands of developers preparing for their dream job.