Infrastructure as Code and Automation Questions

I would put both human review and automation around the plan. A Terraform plan is a preview of create, update, destroy, and replace actions. Replace is especially risky because Terraform deletes and recreates a resource.

Guardrails

• Run terraform fmt -check and terraform validate in CI
• Save the plan as an artifact and require approval before apply
• Fail the pipeline if the plan includes unexpected destroys or replacements on critical resources
• Use policy checks for things like public exposure, open security groups, or deletion of databases
• Add prevent_destroy to the few resources that should almost never be removed

Example
If the plan shows aws_db_instance.main will be replaced because of an engine change, I would force manual review from an engineer and, ideally, a service owner. If the plan only adds an autoscaling instance, that can follow the normal path.

This catches surprises before users feel them, which is the real goal.

I would use count or for_each for the feature flag, but I would prefer for_each for the per-item objects. count is a simple on or off switch. for_each creates one instance per stable key, which is better when the list order changes.

Pattern

• For the feature-flagged singleton, create either one instance or none
• For the repeated objects, convert the caller’s list into a map keyed by a stable ID, such as name
• Avoid indexing directly into a list, because reordering ['api', 'worker'] can cause unnecessary replacement

Example
If the caller passes ['api', 'worker'] today and ['worker', 'api'] tomorrow, keys like api and worker still point to the same resources. That keeps Terraform from churning objects just because the order changed.

Rule of thumb
Use count for a single optional resource, and for_each for anything that should survive list reordering. That makes the module much easier to maintain as the list grows.

I would redesign this so Terraform manages secret references, not secret values, whenever possible. A secret is the actual password or API key. A secret manager is a service like Vault or AWS Secrets Manager that stores the value and gives apps access at runtime.

Workflow

• Store the real secret in a secret manager outside the repo
• Pass only the secret ARN or path into Terraform, not the plaintext value
• Let the application read the secret at runtime using its IAM role or service identity
• Protect state with encryption, tight backend access, and separate prod permissions

Important caveat
If a provider resource truly needs the raw secret during create, Terraform may still record that value in state. Marking a variable sensitive only hides it from CLI output, not from state. So I would avoid designs that require Terraform to own the secret value unless there is no alternative.

Example
Terraform creates prod/db/password as a managed secret container, and the app reads it later. That keeps repo access from becoming secret access.

Decision framework
I first determine whether Terraform should own the object. A resource should be imported only if the team wants Terraform to manage its lifecycle and we can make the current cloud object match code with low risk. If ownership is unclear or another team runs it, I leave it alone and document the boundary.

How I decide

• Import when the resource already exists, is needed long term, and its current settings are close enough to model in code.
• Recreate when the live object is too drifted, unsafe, or impossible to express cleanly, and downtime or cutover is acceptable.
• Leave alone when it is intentionally outside this stack, or the business risk of taking ownership is higher than the benefit.

Concrete example
Say an existing prod-logs bucket was created manually. If the app depends on it and the bucket policy, versioning, and encryption are acceptable, I would import it, then update the module so the code matches the real settings. If the bucket has years of data and a clean import is possible, import is safest. If the bucket name, policy, or region is wrong and fixing it in place would be risky, I would create a new bucket, migrate traffic or data, then delete the old one after validation.

What I check first
I compare state, config, and live attributes, then run a plan to see whether the desired change is truly just drift or a bigger ownership problem.

Direct answer

On my last project I codified an AWS microservices platform, a set of backend services running on managed servers with their own database, load balancer, and access controls (VPC, EKS, RDS, IAM, ALB, monitoring) in Terraform, using versioned reusable modules composed per environment, remote state (the file Terraform uses to track what it created, kept separate per environment so a mistake in one can't touch another) isolated per environment, secrets pulled from AWS Secrets Manager and Parameter Store rather than stored in code, and a PR-based workflow where a machine-generated plan had to be reviewed and approved before an apply job with a separate, more privileged role could run.

How I structured it

Module layout and versioning

• Modules lived in a private registry: vpc, eks, rds, iam, alb, monitoring, each with a narrow set of inputs/outputs and no hidden side effects.
• Root configurations per environment composed these modules and pinned each one to a semantic version tag (for example vpc ~> 2.3), so a change to a module's source could not silently change an environment that had not explicitly bumped its pin.

Environment separation and state

• Each environment (dev, staging, prod) had its own remote state file in an S3 backend with a DynamoDB lock table, keyed roughly as s3://infra-state/{env}/{component}.tfstate.
• Isolating state per environment (rather than one shared state with workspaces) meant a mistake in dev could not touch prod's state, and the blast radius of a single apply was limited to one component in one environment.

Secrets handling

• Database credentials lived in AWS Secrets Manager, app configuration lived in Parameter Store encrypted with KMS.
• Terraform read them at apply time through data sources (for example aws_secretsmanager_secret_version) rather than having them typed anywhere in .tf files or CI variables.
• The CI runner itself never held a long-lived key: it assumed a role via OIDC scoped to the minimum permissions needed for that environment's apply.

Change approval and safe apply

• Every change went through a PR. CI ran terraform plan and posted the JSON plan (with sensitive values redacted) as an artifact, plus a readable summary on the PR.
• A change needed sign-off from both the infra owner and the owning service team before the apply stage would even unlock.
• Apply ran as a separate pipeline stage using a more privileged, MFA-gated role for prod, and every apply was logged to CloudTrail.
• Destructive changes (anything showing a resource replacement or delete in the plan) required an explicit manual confirmation step, and we took a fresh snapshot or backup first.

Worked example

Concretely: adding a read replica to an existing RDS instance meant one PR that touched only the rds module's call site, a plan that showed one resource create and zero destroys, review from the database owner since it touched a stateful resource, and an apply that ran with the prod-apply role only after both approvals landed. Because the module was version-pinned and state for that component was isolated, the blast radius of that single PR was exactly one RDS resource in one environment: nothing else in the account could be affected by that apply.

Trade-offs and pitfalls

• Splitting state by component reduces blast radius but adds cross-stack coordination cost (remote state lookups or SSM parameters to pass values between components); too many tiny state files becomes its own operational burden.
• A plan-then-approve workflow is only as safe as the reviewers actually reading the plan. Without a policy-as-code gate (something like OPA/Rego or Sentinel evaluating the plan JSON for guardrails such as "no public security groups" or "no unencrypted volumes"), review can degrade into rubber-stamping on a busy day.
• Reading secrets via data sources at apply time keeps them out of source control, but the values still land in the Terraform state file in plaintext, so state encryption and tightly scoped state-read IAM matter just as much as the CI-side handling.