Infrastructure as Code and Automation Questions

I would split the solution by responsibility, not by environment. A module should do one job well.

Module layout

• network: VPC, subnets, routes, NAT, and network tags
• compute: app instances, ECS or ASG, load balancer, security groups
• database: managed DB, subnet group, parameter group, DB security group
• root stack: wires the outputs together

Why this works
The network changes slowly, compute changes often, and the database has its own lifecycle and risk. Keeping them separate reduces blast radius and makes reviews easier.

Safe interface
I would expose only what callers need:

• From network: vpc_id, private_subnet_ids, public_subnet_ids
• From compute: alb_dns_name, app_sg_id
• From database: endpoint, port, and maybe a secret reference, not a password

Example
The root module can pass private_subnet_ids = ["subnet-101", "subnet-202"] into compute and database, while dev and prod use different sizes through variables. That keeps composition flexible without letting one team edit module internals.

Direct answer

A totally clean .tf source doesn't guarantee no plaintext secrets, because Terraform itself creates two more places a value can end up: the state file, which stores full resource attributes (including provider-generated secrets like an RDS master password) in plaintext by default regardless of anything in source, and any tooling built around Terraform's own output, like a CI job that posts a plan summary, which can print exactly the values you thought were protected. Defending against this means treating state as sensitive data (encrypt it, restrict who can read it) and understanding precisely what the sensitive flag does and does not cover, rather than assuming "no secrets in the .tf files" is the whole job.

Where it leaks and what to do about it

The state file itself

Whenever Terraform reads back a resource's real attributes, an auto-generated RDS password, a private key from a tls_private_key resource, an API key a provider returns, those values land in terraform.tfstate as plaintext JSON, independent of whether any .tf file ever mentions a secret. Mitigation: server-side encryption on the backend (S3 with SSE-KMS, GCS with CMEK), IAM/access policy that limits who can read the state object, and state locking so nobody's casually pulling and re-uploading a stray copy.

What sensitive = true actually redacts

• What it DOES do: masks the value in the interactive plan/apply diff, in terraform output (prints <sensitive> instead of the value), and in most default log output.
• What it does NOT do: it does not encrypt or redact the value inside the state file itself, the raw state, and terraform show -json, still contain the real value. It's a display-layer guard, not a storage-layer or access-control guard.

The PR-plan-summary-bot leak

A common pattern: a CI job runs terraform plan -out=tfplan, then terraform show -json tfplan, and a script turns that JSON into a human-readable comment on the pull request. If that script naively reads the plan's after (or before) attribute values to build the summary, a value marked sensitive in source shows up in plain text in a PR comment, a location that's usually far more widely readable (the whole engineering org, sometimes an external contractor) than the CI system's own console output. The same risk shows up in the CI job's own logs if the job runs with verbose/debug logging (for example TF_LOG=trace) that dumps provider request/response bodies. Mitigation: any tool consuming terraform show -json output must read the plan's after_sensitive / before_sensitive markers and redact those keys before rendering anything to a comment or log, rather than assuming the CLI's own masking has already been applied to the JSON, it has not.

Worked example

Suppose an aws_db_instance uses manage_master_user_password = true so AWS/Terraform manages a generated password, and a separate random_password output is used elsewhere:

output "app_db_password" {
  value     = random_password.app_db.result
  sensitive = true
}

Running terraform output app_db_password prints <sensitive> in the terminal, the protection working as designed. But terraform show -json terraform.tfstate (or simply opening the state file if it's stored unencrypted) prints the resource's values block including the real password string, because state serialization does not consult the sensitive attribute at all, that flag only affects the CLI/UI rendering layer, exactly as described above.

Trade-offs and pitfalls

• Treating sensitive = true as "the secret is now safe" is the single most common mistake, it only changes what's printed to a screen, not what's stored.
• Encrypting the backend protects data at rest but not against a legitimately-authorized reader with over-broad IAM simply opening the state.
• Building any custom tooling on top of terraform show -json (dashboards, PR bots, drift reporters) reintroduces the exact leak the CLI's own masking prevents, unless that tooling explicitly re-implements redaction using the sensitivity markers in the JSON.
• The most robust long-term fix is minimizing what actually needs to be provider-generated and stored in state at all, preferring external secret managers with dynamic, short-lived credentials over resources that bake a static secret into Terraform's own bookkeeping.

Direct answer
Rotate and revoke the credential immediately, everywhere it's used, before doing anything else: that's the only step that actually stops the bleeding, and forensics, cleanup, and policy changes all happen after containment, not instead of it. Separately, because this keeps happening across the industry, fix the two structural causes: encrypt the state backend so a leaked state file isn't plaintext-readable, and prefer ephemeral, short-lived credentials so secrets stop landing in state to begin with, rather than relying on catching every leak after the fact.

Structured elaboration

Immediate containment, before root-causing anything

• Revoke or rotate the exposed credential at the source (disable the IAM user/key, rotate the DB password) before investigating how it leaked. Every minute it stays valid is exposure.
• Remove the credential's blast radius: rotating a DB password and updating the app's connection string via the secret manager, not a manual edit, locks out anyone who copied it.
• Pull repository access temporarily if the state file is still sitting in a shared, readable location, so no one can pull a fresh copy while cleanup is in progress.

Investigate blast radius
Was the credential actually used by someone who shouldn't have had it? Check the target system's own access logs (DB connection logs, cloud provider audit logs) for activity from unfamiliar IPs or unusual times in the window since the state file was pushed. This determines whether it's "credential exposed, no evidence of misuse" or "confirmed unauthorized access," which changes the notification obligations.

Two different remediation paths depending on how long it's been exposed
A state file pushed an hour ago and one that's been sitting in git history for months are different problems.

• Fresh push, hours old: rotate, remove the offending commit before it's widely pulled, and a normal coordinated git push --force plus history cleanup is enough, because the exposure window is small and you likely know who has pulled it.
• Buried in history for months: assume it's been exposed the entire time, not just since discovery, because you can't retroactively know who cloned, forked, or cached the repo, including CI runner caches and backup snapshots, during that window. This needs a full history rewrite (git filter-repo to strip the file/commit from every ref), coordinated re-clone by every developer and CI system, since a plain force-push doesn't fix clones that already exist, and treating the credential as compromised regardless of what the access logs show, since log retention may not even cover the full exposure window.

Structural fix 1: encrypt the state backend
Even after rotation, the old, now-invalid but still informative, value sitting in a plaintext state file is a problem for anyone auditing what leaked. Use a state backend with encryption at rest by default (S3 with SSE-KMS and a bucket policy denying unencrypted puts, Terraform Cloud/Enterprise's built-in encryption, or a Vault-backed backend) so a copy of the state file alone isn't enough to read secrets even if it leaks again.

Structural fix 2: prefer ephemeral credentials so nothing long-lived is ever in state
The deeper issue is that a static, long-lived credential value, a real DB password, not a reference to one, was ever an attribute in the resource's config to begin with. The fix is architectural, not just procedural: use dynamic, short-TTL credentials issued at apply time (Vault, or a cloud-native equivalent) instead of static passwords, so the value that ever touches Terraform is valid for minutes, not indefinitely. Even that has a gap worth knowing: a value fetched via a data source still gets written into the state file's plaintext by default, because Terraform's state format persists every attribute of every resource and data source it manages, dynamic or not. This is exactly the gap Terraform 1.10+ addressed with ephemeral resources and write-only arguments: values marked ephemeral are used during the apply but are deliberately never written to state or plan files at all, a stronger guarantee than "the credential expires quickly" on its own.

Structural fix 3: a pre-commit/CI scanning gate
Rotation and history rewriting fix one incident; a scanning gate stops the next one from merging in the first place. Run a secret-scanning tool (gitleaks, truffleHog, or similar) both as a pre-commit hook, for fast author feedback, and as a required CI check on every PR, the actual enforcement point since pre-commit hooks can be skipped locally. Scan the Terraform plan/state output specifically, not just source files, since state and plan JSON are exactly where a secret shows up even when the .tf source never had it hardcoded.

Notification and postmortem
If the credential's compromise could have exposed customer data, loop in compliance/legal early, since this determines regulatory notification timelines, often measured in days. Regardless of scale, write a postmortem with a timeline, what was exposed, what was rotated, and concrete owners/dates for the structural fixes above, not just "we rotated it and moved on."

Worked example
Concrete before/after for structural fix 2, showing the actual problem, a static password as a plain resource argument, versus the pattern that avoids it:

# Before: static secret is a plain attribute, gets written to state in plaintext
resource "aws_db_instance" "app" {
  identifier = "app-prod"
  password   = var.db_password   # a real, long-lived value, now permanently in state history
}

# After: value is marked ephemeral (Terraform 1.10+), used during apply,
# never persisted to state or plan output
ephemeral "random_password" "db" {
  length = 24
}

resource "aws_db_instance" "app" {
  identifier           = "app-prod"
  password_wo          = ephemeral.random_password.db.result
  password_wo_version  = 1
}

The practical difference: if this state file leaks again after the fix, there's no password in it to rotate in a panic, because the value was never written there.

Trade-offs & pitfalls

• Rotating before investigating feels backwards to people who want to preserve evidence first, but a live credential someone else might be actively using is a bigger risk than losing a few minutes of forensic cleanliness; contain first.
• Treating "we rotated it" as the end of the incident misses that the old value is still sitting in every git clone and CI cache that pulled it before rotation; a fresh push and a months-old buried secret need genuinely different remediation, not the same checklist applied at different speeds.
• Ephemeral values and write-only arguments require the specific resource/provider to support them; not every provider has adopted the pattern yet, so this is a direction to move toward for new resources, not a guarantee that already covers the whole estate.
• A pre-commit hook alone isn't enforcement, since --no-verify skips it; the CI check on the PR is what actually blocks a merge, and that's the one to treat as required rather than advisory.

I would separate shared logic from environment-specific entrypoints. Right now one large root module is forcing unrelated changes into the same file set, which creates merge conflicts and increases blast radius.

New structure

• modules/network
• modules/app
• modules/database
• envs/dev
• envs/staging
• envs/prod

Each environment folder has its own backend and only calls the modules it needs. Shared values like tags or account IDs can live in small local files or a common variable file.

Why this helps

• Two engineers can edit dev and prod in parallel with less conflict
• A change to app compute does not force a network review unless inputs change
• Cross-environment mistakes are harder because each root has its own state

Example
Dev can use t3.small and prod can use m6i.large, but both call the same modules/app. That gives consistency without a monolith. I would also keep module outputs explicit so one environment cannot accidentally consume another environment’s state.

Approach
I treat a shared Terraform module like an API. First I classify the change: additive and backward-compatible, or breaking. If it is additive, I release a new minor version, keep existing variables and outputs unchanged, and make the new capability opt-in with a default that preserves current behavior. If I must rename or remove something, I publish a new major version and keep the old one available for a transition period.

How I keep upgrades predictable

• Use semantic versioning: patch for fixes, minor for new optional features, major for breaking changes.
• Pin module versions in callers, for example ~> 1.4, so teams only receive compatible updates.
• Add tests that run old examples and new examples in CI.
• Publish a changelog with migration notes and deprecation dates.
• Announce the change early, then give teams a canary path in one workspace before broad rollout.

Concrete example
If the module currently creates an S3 bucket and I want to add optional access logging, I would add enable_access_logging = false and a new logging block. Existing consumers get the same bucket as before, while teams that want logging can opt in. After a release or two, I can deprecate any old workaround variables without breaking them immediately.

Result
That approach lets teams upgrade on their schedule, keeps state changes predictable, and makes ownership clear.