Infrastructure as Code and Automation Questions

A module other teams can safely adopt needs three things: a small, well-documented interface of inputs and outputs that hides the resource internals behind it, a clear versioning discipline so consumers can tell a breaking change from a safe one before they upgrade, and a directory layout that keeps the module itself separate from how any one environment consumes it. Get those right and teams can pin a version, read the README instead of the source, and upgrade on their own schedule instead of being afraid to touch it.

Directory layout & boundaries

/modules/
  /network/     main.tf variables.tf outputs.tf README.md examples/
  /compute/     main.tf variables.tf outputs.tf README.md examples/
/examples/
  dev/  staging/  prod/   (minimal consumption examples with realistic tfvars)

Keep each module scoped to one concern, network, compute, or identity, so a boundary maps to something a consumer would naturally reason about as one unit; a module that bundles unrelated concerns like network plus security plus app forces consumers into all-or-nothing dependencies. Providers should be configured by the root module, not hardcoded inside a reusable module, so the same module works against a consumer's own provider aliases and account.

Interface design

• Required inputs should be the minimum a consumer must decide, the things that genuinely vary per environment; everything else gets a sensible default.
• Group related settings into an object or map variable instead of exploding the parameter list, and use variable descriptions plus validation blocks so misuse fails fast with a clear message.
• Expose only what a consumer needs to wire into other modules, IDs, ARNs, endpoints; don't leak internal resource addresses a consumer could accidentally start depending on.
• Write the README well enough that another team can adopt the module from the README and examples alone, without reading main.tf: what it creates, every input and its default, every output, and one runnable example per environment tier.
• Default to least privilege: any IAM role or security group the module creates should default to the tightest reasonable policy, with broadening an explicit opt-in input, not the default.

Versioning strategy

Pin the module's own provider version requirements so a consumer doesn't get a surprise provider upgrade alongside a module upgrade. Before publishing a new major version, run compatibility tests, a plan against each example environment using the new version, confirming the diff is only the intended change, so you catch a breaking change you didn't mean to ship. Publish to a registry with a changelog per release, and give consumers advance notice, a deprecation window on the old major version, before a breaking release, so nobody upgrades blind.

Worked example: a VPC module
Take a concrete requirement: a VPC with public and private subnets across availability zones, NAT gateways, and route tables. A well-designed module exposes inputs like vpc_cidr, azs, public_subnet_cidrs and private_subnet_cidrs (one entry per AZ), and single_nat_gateway (a cost-versus-availability choice left to the consumer), and outputs vpc_id, public_subnet_ids, private_subnet_ids, and nat_gateway_ids, named plainly enough that another engineer can guess what they contain without opening the source.

An application module then needs those subnet IDs. Three ways to wire that together:

1. Direct module composition: the root module calls both the network and app modules and passes the network module's subnet output straight into the app module's input. Simplest, but couples both modules' lifecycles to the same root and the same apply.
2. Remote-state data source: the app module, a separate root or workspace, reads the network module's outputs via a remote-state lookup. Decouples the two lifecycles so each team applies independently, at the cost of an implicit dependency on the network stack's state layout staying stable.
3. Shared-value registry lookup: both stacks read published values, for example from a parameter store or a data source backed by tags, rather than reading each other's state directly. Most decoupled, and avoids one team's state format becoming another team's contract, at the cost of an extra piece of infrastructure to keep in sync.
   Direct composition is fine when one team owns both layers; independent teams releasing on different schedules are usually better served by remote-state or a published-values lookup, so a network apply doesn't require touching the app stack's plan.

Trade-offs & pitfalls
Over-parameterizing a module, an input for every conceivable variation, makes it harder to document and harder for a consumer to use correctly; keep it opinionated and add inputs when a real second consumer needs them, not speculatively. Safe consumption matters as much as safe design: a shared compute-cluster module should be parameterized differently for non-prod versus prod, tighter autoscaling ceilings and spot instances allowed in non-prod, higher limits and on-demand in prod, through explicit, validated inputs, so a consumer can't accidentally instantiate an expensive always-on cluster in a dev account just because the module's default happened to be prod-shaped.

Direct answer

A totally clean .tf source doesn't guarantee no plaintext secrets, because Terraform itself creates two more places a value can end up: the state file, which stores full resource attributes (including provider-generated secrets like an RDS master password) in plaintext by default regardless of anything in source, and any tooling built around Terraform's own output, like a CI job that posts a plan summary, which can print exactly the values you thought were protected. Defending against this means treating state as sensitive data (encrypt it, restrict who can read it) and understanding precisely what the sensitive flag does and does not cover, rather than assuming "no secrets in the .tf files" is the whole job.

Where it leaks and what to do about it

The state file itself

Whenever Terraform reads back a resource's real attributes, an auto-generated RDS password, a private key from a tls_private_key resource, an API key a provider returns, those values land in terraform.tfstate as plaintext JSON, independent of whether any .tf file ever mentions a secret. Mitigation: server-side encryption on the backend (S3 with SSE-KMS, GCS with CMEK), IAM/access policy that limits who can read the state object, and state locking so nobody's casually pulling and re-uploading a stray copy.

What sensitive = true actually redacts

• What it DOES do: masks the value in the interactive plan/apply diff, in terraform output (prints <sensitive> instead of the value), and in most default log output.
• What it does NOT do: it does not encrypt or redact the value inside the state file itself, the raw state, and terraform show -json, still contain the real value. It's a display-layer guard, not a storage-layer or access-control guard.

The PR-plan-summary-bot leak

A common pattern: a CI job runs terraform plan -out=tfplan, then terraform show -json tfplan, and a script turns that JSON into a human-readable comment on the pull request. If that script naively reads the plan's after (or before) attribute values to build the summary, a value marked sensitive in source shows up in plain text in a PR comment, a location that's usually far more widely readable (the whole engineering org, sometimes an external contractor) than the CI system's own console output. The same risk shows up in the CI job's own logs if the job runs with verbose/debug logging (for example TF_LOG=trace) that dumps provider request/response bodies. Mitigation: any tool consuming terraform show -json output must read the plan's after_sensitive / before_sensitive markers and redact those keys before rendering anything to a comment or log, rather than assuming the CLI's own masking has already been applied to the JSON, it has not.

Worked example

Suppose an aws_db_instance uses manage_master_user_password = true so AWS/Terraform manages a generated password, and a separate random_password output is used elsewhere:

output "app_db_password" {
  value     = random_password.app_db.result
  sensitive = true
}

Running terraform output app_db_password prints <sensitive> in the terminal, the protection working as designed. But terraform show -json terraform.tfstate (or simply opening the state file if it's stored unencrypted) prints the resource's values block including the real password string, because state serialization does not consult the sensitive attribute at all, that flag only affects the CLI/UI rendering layer, exactly as described above.

Trade-offs and pitfalls

• Treating sensitive = true as "the secret is now safe" is the single most common mistake, it only changes what's printed to a screen, not what's stored.
• Encrypting the backend protects data at rest but not against a legitimately-authorized reader with over-broad IAM simply opening the state.
• Building any custom tooling on top of terraform show -json (dashboards, PR bots, drift reporters) reintroduces the exact leak the CLI's own masking prevents, unless that tooling explicitly re-implements redaction using the sensitivity markers in the JSON.
• The most robust long-term fix is minimizing what actually needs to be provider-generated and stored in state at all, preferring external secret managers with dynamic, short-lived credentials over resources that bake a static secret into Terraform's own bookkeeping.

Direct answer

Terraform has no built-in transactional rollback because it applies resources one at a time in dependency order, not as a single atomic unit, so "rolling back" is really a set of separate patterns you choose between based on blast radius: reapplying a known-good, version-pinned configuration, restoring a snapshotted state file, or targeted, resource-level remediation. Automated rollback is the right call for reversible, idempotent changes with a clear health signal (a stateless service behind an ASG, a container image swap); a human needs to reconcile by hand whenever the change is destructive, touches stateful data, or the exact post-failure state is ambiguous.

Patterns for walking back a change

Versioned modules and state backups

• Pin every module call to a semantic version tag, so "roll back the infrastructure" can mean "redeploy the previous module version" rather than reconstructing a diff by hand.
• Keep state backend versioning on (S3/GCS object versioning) and snapshot state before every apply, tagged with the pipeline run ID and commit.

Partial rollback versus full rollback

• A full rollback (restoring the entire state file to the last-good snapshot) is the blunt instrument: fast, but it can silently undo any other legitimate change another engineer applied in the meantime, since it doesn't know which parts of the diff are the problem.
• A partial rollback (fixing or re-targeting just the resource(s) that actually broke, using terraform apply -target sparingly, or hand-correcting one resource's config) is narrower and safer when you know exactly what failed, but requires that you've actually diagnosed the failure first, guessing wrong under a full state restore is less catastrophic than guessing wrong under a targeted one.

Reapplying configuration management state after an infra rollback

If a configuration management tool (Ansible, Chef, Puppet) runs on top of Terraform-provisioned instances, rolling back the Terraform layer alone (say, reverting an instance's AMI or instance type) does not automatically put that instance's OS-level configuration back in sync. The CM tool's last run was against the instance in its post-change form; after an infra rollback you need to rerun the CM playbook or run-list against the rolled-back instance so both layers agree again, otherwise you end up with infra that matches the old Terraform state but application configuration that still reflects the change you just reverted.

Keeping a multi-resource logical change atomic

A single logical change that spans several resources, for example provisioning a new database instance together with the IAM policies that grant an application access to it, needs to stay consistent even if one part fails. Two practical approaches: sequence the change so a partial failure is safe by construction (grant the IAM policy before the database exists, so a database-creation failure just leaves an unused policy rather than a database nothing can reach), or make each piece individually idempotent so re-running the apply after a partial failure converges to the same end state rather than erroring on "already exists."

Worked example

Say an apply plans five resources in this order: a security group, a DB subnet group, an RDS instance, an IAM policy, and an IAM role-policy attachment. The first four create successfully; the fifth fails because the policy ARN referenced in the attachment has a typo. Terraform writes state incrementally as each resource completes, not only at the end of the run, so at the moment of failure, state already correctly records the security group, subnet group, RDS instance, and IAM policy as created; only the role-policy attachment is missing. Running terraform plan again reflects this accurately: it proposes creating only the missing attachment, not recreating the other four, because state already matches reality for those. The fix is to correct the ARN and reapply; nothing about the first four resources needs to be touched or reasoned about again. This is the diagnostic step that has to happen before deciding between "just reapply" (safe here, since the failure was config-only and everything else succeeded) and "restore from snapshot" (which would be the wrong call here, since it would needlessly discard four correctly-created resources).

Trade-offs and pitfalls

• Automated rollback assumes the operation is reversible and idempotent, true for stateless compute and image swaps, false for anything with an irreversible side effect (a dropped database column, deleted data, a webhook already fired to an external system). Forcing an automated revert onto that class of change is how you turn one incident into two.
• A blanket "restore the old state file" full rollback is dangerous in a team setting precisely because it can undo unrelated, legitimate concurrent changes, prefer targeted remediation once you've actually diagnosed which resource is broken, as in the worked example above.
• Forgetting the configuration-management layer after an infra-level rollback is a common miss: the infra looks reverted, but the OS-level config on that instance can silently stay in the post-change state until the CM tool is rerun.

I would move production to a remote backend that supports both shared access and state locking. State is Terraform’s record of what it created, and locking means only one writer can change that record at a time.

Design choices

• Use one backend per environment, so prod does not share state with dev
• Turn on encryption at rest for the backend storage
• Restrict write access to CI and a small break-glass admin group
• Give most engineers read-only access to plans and state history
• Require an exclusive lock for apply, with a short timeout and clear failure message

Recovery
I would also enable versioning or history so I can recover a prior state file if an apply fails halfway through. If a run breaks, I would not edit state first. I would inspect the lock, compare the current cloud resources with terraform plan -refresh-only, and then decide whether to fix drift, import an orphan, or roll back.

Example
If a colleague applies from their workstation while CI is planning, the lock should stop the second write. That prevents two people from corrupting the same prod state.

I would use count or for_each for the feature flag, but I would prefer for_each for the per-item objects. count is a simple on or off switch. for_each creates one instance per stable key, which is better when the list order changes.

Pattern

• For the feature-flagged singleton, create either one instance or none
• For the repeated objects, convert the caller’s list into a map keyed by a stable ID, such as name
• Avoid indexing directly into a list, because reordering ['api', 'worker'] can cause unnecessary replacement

Example
If the caller passes ['api', 'worker'] today and ['worker', 'api'] tomorrow, keys like api and worker still point to the same resources. That keeps Terraform from churning objects just because the order changed.

Rule of thumb
Use count for a single optional resource, and for_each for anything that should survive list reordering. That makes the module much easier to maintain as the list grows.