Infrastructure as Code and Automation Questions

A module other teams can safely adopt needs three things: a small, well-documented interface of inputs and outputs that hides the resource internals behind it, a clear versioning discipline so consumers can tell a breaking change from a safe one before they upgrade, and a directory layout that keeps the module itself separate from how any one environment consumes it. Get those right and teams can pin a version, read the README instead of the source, and upgrade on their own schedule instead of being afraid to touch it.

Directory layout & boundaries

/modules/
  /network/     main.tf variables.tf outputs.tf README.md examples/
  /compute/     main.tf variables.tf outputs.tf README.md examples/
/examples/
  dev/  staging/  prod/   (minimal consumption examples with realistic tfvars)

Keep each module scoped to one concern, network, compute, or identity, so a boundary maps to something a consumer would naturally reason about as one unit; a module that bundles unrelated concerns like network plus security plus app forces consumers into all-or-nothing dependencies. Providers should be configured by the root module, not hardcoded inside a reusable module, so the same module works against a consumer's own provider aliases and account.

Interface design

• Required inputs should be the minimum a consumer must decide, the things that genuinely vary per environment; everything else gets a sensible default.
• Group related settings into an object or map variable instead of exploding the parameter list, and use variable descriptions plus validation blocks so misuse fails fast with a clear message.
• Expose only what a consumer needs to wire into other modules, IDs, ARNs, endpoints; don't leak internal resource addresses a consumer could accidentally start depending on.
• Write the README well enough that another team can adopt the module from the README and examples alone, without reading main.tf: what it creates, every input and its default, every output, and one runnable example per environment tier.
• Default to least privilege: any IAM role or security group the module creates should default to the tightest reasonable policy, with broadening an explicit opt-in input, not the default.

Versioning strategy

Pin the module's own provider version requirements so a consumer doesn't get a surprise provider upgrade alongside a module upgrade. Before publishing a new major version, run compatibility tests, a plan against each example environment using the new version, confirming the diff is only the intended change, so you catch a breaking change you didn't mean to ship. Publish to a registry with a changelog per release, and give consumers advance notice, a deprecation window on the old major version, before a breaking release, so nobody upgrades blind.

Worked example: a VPC module
Take a concrete requirement: a VPC with public and private subnets across availability zones, NAT gateways, and route tables. A well-designed module exposes inputs like vpc_cidr, azs, public_subnet_cidrs and private_subnet_cidrs (one entry per AZ), and single_nat_gateway (a cost-versus-availability choice left to the consumer), and outputs vpc_id, public_subnet_ids, private_subnet_ids, and nat_gateway_ids, named plainly enough that another engineer can guess what they contain without opening the source.

An application module then needs those subnet IDs. Three ways to wire that together:

1. Direct module composition: the root module calls both the network and app modules and passes the network module's subnet output straight into the app module's input. Simplest, but couples both modules' lifecycles to the same root and the same apply.
2. Remote-state data source: the app module, a separate root or workspace, reads the network module's outputs via a remote-state lookup. Decouples the two lifecycles so each team applies independently, at the cost of an implicit dependency on the network stack's state layout staying stable.
3. Shared-value registry lookup: both stacks read published values, for example from a parameter store or a data source backed by tags, rather than reading each other's state directly. Most decoupled, and avoids one team's state format becoming another team's contract, at the cost of an extra piece of infrastructure to keep in sync.
   Direct composition is fine when one team owns both layers; independent teams releasing on different schedules are usually better served by remote-state or a published-values lookup, so a network apply doesn't require touching the app stack's plan.

Trade-offs & pitfalls
Over-parameterizing a module, an input for every conceivable variation, makes it harder to document and harder for a consumer to use correctly; keep it opinionated and add inputs when a real second consumer needs them, not speculatively. Safe consumption matters as much as safe design: a shared compute-cluster module should be parameterized differently for non-prod versus prod, tighter autoscaling ceilings and spot instances allowed in non-prod, higher limits and on-demand in prod, through explicit, validated inputs, so a consumer can't accidentally instantiate an expensive always-on cluster in a dev account just because the module's default happened to be prod-shaped.

Direct answer

Decouple the schema change from the infra rollout and sequence it in stages using the expand-contract pattern: expand first (an additive, backward-compatible schema change applied through the IaC/migration pipeline while the old application code keeps running unchanged), then migrate data, then only contract (drop or tighten the old shape) once the new application version has been fully rolled out and validated against the new schema. The fallback if something goes wrong partway through is usually to just stop and hold at whatever phase you are in and flip a feature flag off, because every step up through migrate is additive and non-destructive; nothing forces an emergency down-migration unless you jump straight to contract before it is safe.

Structured elaboration

The expand-contract pattern

• Expand: add the new column, table, or index in a way that does not require old code to change, nullable or defaulted, no rename, no drop.
• Migrate: backfill or dual-write so the new shape is populated while the old shape is still being read and written by the currently deployed application.
• Contract: once the new application code is fully rolled out and only depends on the new shape, remove the old column or constraint in a later, separate release.

Sequencing against the infra rollout

The schema migration and the infra/app rollout are two different releases, not one. The expand step ships through the IaC/migration pipeline on its own, ahead of any application change that depends on it. The application code that reads or writes the new column stays behind a feature flag even after the column exists, so the schema release and the app release are decoupled: you can flip the flag on independently of any deploy, and flip it back off instantly if something looks wrong, without touching the database again.

Comparing rollout strategies against a shared, stateful database

Long-running migrations on large tables

For a table too large to migrate in one blocking DDL, use an online schema-change approach: either the database's own online DDL if the engine and change support it, or a chunked backfill job that processes bounded batches (for example by primary-key range) with built-in throttling to bound replication lag and lock contention. Each batch should be written so it is idempotent, re-running a batch that already succeeded should be a no-op, so the job can pause and resume safely instead of needing to restart from scratch.

Worked example

Say we need to add a NOT NULL orders.shipped_at column, no default, to a 50 million row table.

1. Expand: add shipped_at as nullable, no default. This is an additive, non-blocking change with no dependency on any read path changing.
2. Backfill: run a batched UPDATE in chunks of 10,000 rows ordered by primary key, each batch scoped as WHERE shipped_at IS NULL AND id BETWEEN :start AND :end. That bounds every single transaction's lock and redo footprint to 10,000 rows regardless of total table size, and because the WHERE clause only matches unfinished rows, re-running any batch that already completed touches zero rows, so the job is safely resumable after a failure or pause. At 10,000 rows per batch, 50 million rows means roughly 5,000 batches; that number is only there to show the backfill decomposes into a bounded, resumable unit of work, it is not a timing claim.
3. Dual-write: the new application code, behind a feature flag, writes shipped_at on every new order while still tolerating it being null on read for older rows.
4. Validate: confirm there are zero unexpected NULL rows outside the ones that legitimately have not shipped yet, before proceeding.
5. Contract: only after the new application version is fully rolled out and the flag has been on and stable, add the NOT NULL constraint in its own, separate release, and only then remove any old fallback code path that tolerated null.

Trade-offs & pitfalls

• "Rollback" here mostly means flipping the feature flag off and pausing, not a destructive down-migration, because everything through the migrate step is additive. Keep an explicit rollback script for the rare case an expand step itself needs reverting, for example dropping the new column, which is only safe precisely because nothing depends on it being NOT NULL yet.
• Coordinating flag state with schema state is itself a source of bugs: flipping the flag on before the backfill has finished sends null values into code paths that were not written to expect them.
• Canarying the app version alongside a shared database only tells you about app-level regressions; canary and control traffic hit the exact same schema, so schema-level correctness has to be validated independently, during the migrate phase, not inferred from canary metrics.

Direct answer

Moving a patched-in-place VM fleet to an immutable image model means separating "how the image is built" from "how the fleet is rolled over": build and validate a versioned image with a tool like Packer (a CLI that scripts building a machine image from a template), roll it out behind a second autoscaling group with a gradual canary before a full cutover rather than a single big-bang swap, and pull anything stateful (databases, session data, files that must survive an instance being replaced) off the instance entirely so instances become disposable and the rollout doesn't have to babysit data.

Getting there

Building and validating the image

Use Packer to bake a versioned AMI: install packages, apply the same hardening the current fleet has via provisioners, and run automated image tests (boot check, service health, a smoke test hitting the app) before an AMI is eligible to deploy. Nothing instance-specific gets baked in, secrets and per-instance config are still fetched at boot time from Secrets Manager/SSM, exactly as they would be for the existing fleet, only the delivery mechanism for "what's installed" moves from a patch run to a new image build.

Rolling the fleet over without a big-bang cutover

graph LR
  A[Base OS image] --> B[Packer build + provisioners]
  B --> C[Automated image tests]
  C --> D[New Auto Scaling Group on new AMI]
  D --> E[Canary: shift small percent of traffic]
  E --> F{Healthy?}
  F -->|Yes| G[Full cutover: shift remaining traffic]
  F -->|No| H[Rollback: route back to old ASG]
  G --> I[Drain and terminate old instances]

Stand up a second ASG on the new AMI behind the same load balancer as the existing fleet, and start by shifting only a small slice of traffic to it (a canary), watching health checks and error-rate signals before widening further. Doing the migration itself as a canary rollout, not just the day-to-day deploys after migration is done, is what avoids a big-bang cutover: a bad image only affects the canary slice, not the whole fleet, and you can route traffic back to the untouched old ASG immediately if it isn't healthy.

Handling the stateful pieces

• Databases: move to a managed service (RDS) if not already there, or use replication plus a coordinated cutover if migrating the DB itself.
• Local/session state: externalize to something the new, disposable instances can all reach, S3 or EFS for files, Redis for session data, so any instance can be swapped without losing anything tied to it.
• Anything acting as a "leader" or holding a lock: move that coordination to something external (DynamoDB, etcd, Consul) instead of relying on a specific instance's identity, since that identity won't survive the cutover.

Worked example

Say the existing fleet is a set of long-running EC2 instances patched in place by a configuration management tool. The team builds a new AMI with Packer that layers the same packages via provisioners, with nothing instance-specific baked in. They stand up a second ASG on that AMI behind the existing load balancer, initially sized to take roughly a tenth of traffic, a chosen canary slice, not a measured result, while the old ASG keeps the rest. Health checks and error-rate alarms on the new ASG are watched before the split is widened in further steps, and only once the new ASG has fully absorbed traffic is the old ASG scaled to zero and terminated. In this case session data was already externalized to Redis and the primary database was already RDS before the migration started, so neither had to move as part of the cutover, only the compute layer changed.

Trade-offs and pitfalls

• Immutable, image-based fleets trade a fast "SSH in and fix it" patch for a slower "bake, test, redeploy" cycle, a real cost during an urgent security patch if the image build pipeline itself is slow or manual; mitigate by keeping that pipeline fast and fully automated rather than something someone has to babysit.
• Running two ASGs side by side during the migration roughly doubles compute spend for the duration of the cutover; that's usually an acceptable trade for safety, but size and time-box it deliberately rather than leaving both running indefinitely.
• Anything still implicitly relying on instance identity (a hardcoded IP allowlist, a cron job assuming it's the same box every day) breaks silently during cutover unless it's found during the inventory step; incomplete discovery of what depends on the old fleet's stability is usually the biggest risk in this migration, not the image-build mechanics themselves.

I would consume the shared network with data sources or, if the network team publishes outputs, with terraform_remote_state. A data source is Terraform’s read-only lookup for existing infrastructure. I would prefer explicit outputs for IDs, because they are less ambiguous than searching by tags.

What I would check

• The VPC ID matches the expected CIDR, for example 10.20.0.0/16
• The subnet count is what I need, for example 2 private subnets in us-east-1a and us-east-1b
• Subnet tags match my assumptions, such as tier=private and env=prod
• All subnets belong to the same VPC

Fail loudly
I would add variable validation plus precondition checks so the plan stops before apply if the layout is wrong. For example, if I expect exactly 2 private subnets and only find 1, the module should error instead of guessing.

That approach keeps the module reusable, but still safe when the shared network changes.

Direct answer

Baking an image means pre-installing everything a server needs (OS packages, hardening, the app itself) into a reusable image with a tool like Packer, instead of configuring the server after it boots. Build the pipeline around one principle: nothing reaches production as an image that has not been baked, tested, and scanned the same way every time, and the fleet gets updated by replacing instances behind health checks and connection draining rather than patching them in place. The design has two paths through the same pipeline: the normal path (bake, test, promote through environments, canary, full rollout) and a fast path for urgent security patches that skips environment promotion but never skips the tests or the scan.

Structured elaboration

Image build and test

• CI triggers a Packer build on a base-image or application change: provision the base OS, apply hardening (CIS-style benchmarks, i.e. standardized security-configuration checklists), install the app artifact, and pull secrets via short-lived tokens rather than baking them in.
• Baked-in automated tests run as part of the same pipeline, not as a separate manual step: unit and config-validation tests during the bake, then a post-bake stage that launches the image in an isolated environment and runs integration and smoke tests against it.
• A vulnerability scan (for example Trivy or Grype against the baked image) runs in that same post-bake stage. This is a hard gate, not advisory: an image with a scan finding above the agreed severity threshold does not get published.
• On pass, the image is registered in the artifact registry tagged with its git SHA, build ID, SBOM, and the CVE baseline it passed against, so any later question of "what is actually running" and "was it scanned against what we knew at the time" has an answer.

Promotion across environments

Promotion is a pipeline gate, not a person clicking approve in a console: dev, then staging with regression tests, then a canary slice of production, each gated on the previous stage's tests and monitoring staying green.

Swapping the fleet over

• The fleet sits behind an ASG (or equivalent instance group) and a load balancer. The launch template points at the new image; an ASG instance refresh (or an equivalent rolling-replace controller) walks the fleet in batches.
• Per instance: deregister from the target group first, which starts connection draining; wait for in-flight requests to finish or the drain timeout to hit; only then terminate it. The replacement instance must pass its health check before the load balancer sends it any traffic.
• A minimum-healthy-percentage setting (for example 90%) caps how much capacity can be replacing at once, so a bad new image degrades a fraction of the fleet rather than all of it while it is still being watched.
• For workloads that carry state (a service with long-lived connections, or one with session affinity), connection draining alone is not enough: the drain window also has to respect existing session affinity, and if any part of the workload is stateful in the sense of holding data (not just connections), that has to coordinate with the data layer's own replication or failover process rather than treating the instance as freely swappable the moment its health check fails.

Fast-tracking an urgent security patch

The fast path changes how far the image travels before real traffic sees it, not whether it is tested:

• Skip the full dev-then-staging promotion chain; go straight from bake to a canary slice of production.
• Keep the bake-time tests and the vulnerability scan as hard gates; an urgent patch that has not been scanned is exactly the failure mode a patch process exists to prevent.
• Shorten, but do not remove, the canary observation window, and have the rollback path pre-verified rather than improvised, since this path is exercised under time pressure.
• Immediately backfill afterward: once the emergency patch has gone through the fast path, run it (or its base) through the normal dev and staging pipeline the following day, so the fast-tracked version does not become a permanent exception living outside the standard promotion history.

Worked example

flowchart TD
  A[Source or base image change] --> B[CI triggers Packer bake]
  B --> C[Bake time tests: hardening, vuln scan, smoke tests]
  C --> D[Publish image with SBOM and CVE tags to registry]
  D --> E[Promote through dev then staging]
  E --> F[Canary: weighted traffic on new AMI]
  F --> G{Health checks and SLOs pass?}
  G -- Yes --> H[Full fleet rollout via ASG instance refresh]
  G -- No --> I[Roll back to prior AMI, tag new image as bad]
  B -.urgent security patch.-> J[Fast path: skip dev and staging, bake plus scan only]
  J --> F

Concretely: a CVE lands in the base OS image. CI triggers a Packer bake immediately (the dashed path above). The bake produces a new image; the same automated tests and the same vulnerability scan run against it as any normal build, just without waiting for a scheduled promotion window. It goes straight to a canary slice of the fleet, monitored against the same health checks and error-rate thresholds as any other rollout, then to the full fleet via instance refresh. The following day, the same image is run through the normal dev and staging environments to confirm nothing outside the emergency scope regressed.

Trade-offs & pitfalls

• Baking images takes longer than patching in place, and that trade-off is deliberate: reproducibility and a clean rollback (revert the launch template to the previous image ID) are worth the extra build minutes.
• The most dangerous version of a "fast path" is one that quietly also skips testing or scanning under time pressure; the fast path should only ever shorten promotion, never verification.
• For stateful workloads, connection draining and health checks are necessary but not sufficient; assuming they are enough to make image replacement safe for anything holding data is a common design mistake.
• Rolling back an in-flight instance refresh needs to be a rehearsed, one-command action (point the launch template back at the previous image ID), not something improvised the first time it is needed.