Infrastructure as Code and Automation Questions

Direct answer
Sequence by dependency layer, not by application: foundational services first (network connectivity, identity, DNS, logging), shared services next, then non-prod workloads to prove the IaC modules work, and only then production, one tier at a time with a canary before full cutover. For each system, decide import versus rebuild based on whether it's a well-understood, cloud-native-mappable resource, in which case import, or a piece of accumulated config drift where reproducing it faithfully would just encode the debt, in which case rebuild.

Structured elaboration

Phase 1: Discovery and inventory
Automated discovery beats a spreadsheet built from memory: pull from the CMDB (configuration management database, the org's system-of-record inventory) if one exists and is trustworthy, cross-reference with what's actually running (Ansible facts or an SCCM/WSUS export for Windows, network scans for anything undocumented). The output is one canonical list per host/service: what it is, what it depends on, what depends on it, current backup/replication status, and an owner. Systems with no identifiable owner get flagged for a decision, migrate as-is or decommission, before they get IaC'd at all; don't automate ownership questions away.

Deciding import versus rebuild

• Import when the resource maps cleanly onto a cloud construct and its current config is trusted (a well-behaved VM that's really just a Linux box running a stateless service maps onto a cloud VM/instance with no surprises).
• Rebuild when the current config has drifted far enough from any documented baseline that reproducing it faithfully would mean writing IaC for years of undocumented manual tweaks, when a cloud-native replacement removes an entire operational burden (a hand-managed on-prem DB server becoming a managed database service), or when the security posture is bad enough that carrying it forward as-is isn't acceptable.
• The uncomfortable middle case, a system nobody fully understands but that's too risky to touch, gets isolated (its own network segment, tightly scoped IAM) and migrated last, with extra testing budget, rather than either extreme.

Sequencing to keep risk low

1. Network and identity: VPN/DirectConnect (or equivalent) between on-prem and cloud, IAM/AD federation, so hybrid connectivity exists before anything depends on it.
2. Shared services: DNS, NTP, centralized logging, backup targets, so everything migrated afterward has somewhere to report to.
3. Non-production workloads: prove the IaC modules (network, compute, IAM patterns) work end to end where a mistake is cheap.
4. Staging/canary for production-bound applications, using production-shaped data volumes and traffic patterns where feasible.
5. Stateful systems (databases, anything with data gravity), with explicit replication and cutover planning, not treated as "just another VM."
6. Production cutover, app by app, in dependency order.

Testing at each stage

• Terraform-level: validate, fmt, tflint/checkov in CI before anything runs against real infrastructure.
• Integration: apply into a sandbox, run smoke tests against the actual service and functional tests exercising real workflows, not just "did terraform apply succeed."
• Cutover rehearsal: run the actual runbook against staging at least once before running it against production, timing each step so production isn't the first execution.

The cutover runbook
An ordered checklist with explicit go/no-go gates:

1. Pre-checks: final data sync started, config drift on the source system confirmed at zero, backups verified restorable via an actual test restore, on-call notified.
2. Provision cloud infra via the pinned IaC module version, not "latest."
3. Final data sync completes, replication lag confirmed near zero.
4. Health checks and smoke tests pass against the new environment before any real traffic touches it.
5. Traffic shift via DNS/load-balancer weighting, gradual, with metrics watched between each step.
6. Go/no-go gate: if error rate or latency degrades beyond the pre-agreed threshold at any step, weight shifts back to on-prem immediately, no debugging in place first.
7. Once fully shifted and stable for the agreed validation window, decommission the on-prem system, keeping backups per the org's retention policy.

Worked example
A concrete piece of Phase 1 for one Linux host, using two purpose-built Ansible modules rather than plain gather_facts (the standard setup module never collects open ports or installed packages, referencing a fact it doesn't produce fails silently, not loudly, which is its own trap):

# discovery-playbook.yml
- hosts: legacy_estate
  gather_facts: true
  tasks:
    - name: Gather listening port facts
      community.general.listen_ports_facts:

    - name: Gather package facts
      ansible.builtin.package_facts:

    - name: Record OS, packages, and listening ports
      ansible.builtin.set_fact:
        inventory_record:
          hostname: "{{ ansible_hostname }}"
          os: "{{ ansible_distribution }}{{ ansible_distribution_version }}"
          packages: "{{ ansible_facts.packages.keys() | list if ansible_facts.packages is defined else [] }}"
          listening_ports: "{{ ansible_facts.tcp_listen | default([]) | map(attribute='port') | list }}"

    - name: Append to canonical inventory
      ansible.builtin.lineinfile:
        path: ./inventory.jsonl
        line: "{{ inventory_record | to_json }}"
        create: true

community.general.listen_ports_facts (it needs netstat or ss on the target and is Linux-only, matching this "legacy_estate" group) is what actually populates ansible_facts.tcp_listen, a list of dicts with a port field, among others, which is why the field is only meaningful after this task runs, not from gather_facts alone. ansible.builtin.package_facts auto-detects the box's package manager and populates ansible_facts.packages as a dict of package name to installed-version list. lineinfile needs create: true or it refuses to touch a file that doesn't exist yet; without it, the very first run of this playbook fails before the inventory file is ever created.

Running this across the estate produces one JSON line per host, which feeds the import-versus-rebuild decision: a host whose listening ports and packages match a known, simple service profile is a strong import candidate; one that doesn't match anything documented goes into the "investigate before deciding" bucket.

Trade-offs & pitfalls

• Treating discovery as a one-time step is a common mistake: on-prem estates drift during the months a migration takes, so re-run discovery periodically through the project, not just at kickoff.
• Rebuilding too aggressively, using the migration as an excuse to modernize everything at once, multiplies risk; separate "move" from "improve" and do the improvement afterward, once the system is stable on the new platform.
• The go/no-go gate only works if the threshold is agreed before the cutover; under pressure, teams tend to talk themselves into pushing through a borderline metric rather than rolling back.
• Stateful systems are where hybrid migrations actually go wrong; budget disproportionately more testing and rehearsal time for databases and anything with data gravity than for stateless compute.

Direct answer

Decouple the schema change from the infra rollout and sequence it in stages using the expand-contract pattern: expand first (an additive, backward-compatible schema change applied through the IaC/migration pipeline while the old application code keeps running unchanged), then migrate data, then only contract (drop or tighten the old shape) once the new application version has been fully rolled out and validated against the new schema. The fallback if something goes wrong partway through is usually to just stop and hold at whatever phase you are in and flip a feature flag off, because every step up through migrate is additive and non-destructive; nothing forces an emergency down-migration unless you jump straight to contract before it is safe.

Structured elaboration

The expand-contract pattern

• Expand: add the new column, table, or index in a way that does not require old code to change, nullable or defaulted, no rename, no drop.
• Migrate: backfill or dual-write so the new shape is populated while the old shape is still being read and written by the currently deployed application.
• Contract: once the new application code is fully rolled out and only depends on the new shape, remove the old column or constraint in a later, separate release.

Sequencing against the infra rollout

The schema migration and the infra/app rollout are two different releases, not one. The expand step ships through the IaC/migration pipeline on its own, ahead of any application change that depends on it. The application code that reads or writes the new column stays behind a feature flag even after the column exists, so the schema release and the app release are decoupled: you can flip the flag on independently of any deploy, and flip it back off instantly if something looks wrong, without touching the database again.

Comparing rollout strategies against a shared, stateful database

Long-running migrations on large tables

For a table too large to migrate in one blocking DDL, use an online schema-change approach: either the database's own online DDL if the engine and change support it, or a chunked backfill job that processes bounded batches (for example by primary-key range) with built-in throttling to bound replication lag and lock contention. Each batch should be written so it is idempotent, re-running a batch that already succeeded should be a no-op, so the job can pause and resume safely instead of needing to restart from scratch.

Worked example

Say we need to add a NOT NULL orders.shipped_at column, no default, to a 50 million row table.

1. Expand: add shipped_at as nullable, no default. This is an additive, non-blocking change with no dependency on any read path changing.
2. Backfill: run a batched UPDATE in chunks of 10,000 rows ordered by primary key, each batch scoped as WHERE shipped_at IS NULL AND id BETWEEN :start AND :end. That bounds every single transaction's lock and redo footprint to 10,000 rows regardless of total table size, and because the WHERE clause only matches unfinished rows, re-running any batch that already completed touches zero rows, so the job is safely resumable after a failure or pause. At 10,000 rows per batch, 50 million rows means roughly 5,000 batches; that number is only there to show the backfill decomposes into a bounded, resumable unit of work, it is not a timing claim.
3. Dual-write: the new application code, behind a feature flag, writes shipped_at on every new order while still tolerating it being null on read for older rows.
4. Validate: confirm there are zero unexpected NULL rows outside the ones that legitimately have not shipped yet, before proceeding.
5. Contract: only after the new application version is fully rolled out and the flag has been on and stable, add the NOT NULL constraint in its own, separate release, and only then remove any old fallback code path that tolerated null.

Trade-offs & pitfalls

• "Rollback" here mostly means flipping the feature flag off and pausing, not a destructive down-migration, because everything through the migrate step is additive. Keep an explicit rollback script for the rare case an expand step itself needs reverting, for example dropping the new column, which is only safe precisely because nothing depends on it being NOT NULL yet.
• Coordinating flag state with schema state is itself a source of bugs: flipping the flag on before the backfill has finished sends null values into code paths that were not written to expect them.
• Canarying the app version alongside a shared database only tells you about app-level regressions; canary and control traffic hit the exact same schema, so schema-level correctness has to be validated independently, during the migrate phase, not inferred from canary metrics.

I would use count or for_each for the feature flag, but I would prefer for_each for the per-item objects. count is a simple on or off switch. for_each creates one instance per stable key, which is better when the list order changes.

Pattern

• For the feature-flagged singleton, create either one instance or none
• For the repeated objects, convert the caller’s list into a map keyed by a stable ID, such as name
• Avoid indexing directly into a list, because reordering ['api', 'worker'] can cause unnecessary replacement

Example
If the caller passes ['api', 'worker'] today and ['worker', 'api'] tomorrow, keys like api and worker still point to the same resources. That keeps Terraform from churning objects just because the order changed.

Rule of thumb
Use count for a single optional resource, and for_each for anything that should survive list reordering. That makes the module much easier to maintain as the list grows.

Approach
I treat a shared Terraform module like an API. First I classify the change: additive and backward-compatible, or breaking. If it is additive, I release a new minor version, keep existing variables and outputs unchanged, and make the new capability opt-in with a default that preserves current behavior. If I must rename or remove something, I publish a new major version and keep the old one available for a transition period.

How I keep upgrades predictable

• Use semantic versioning: patch for fixes, minor for new optional features, major for breaking changes.
• Pin module versions in callers, for example ~> 1.4, so teams only receive compatible updates.
• Add tests that run old examples and new examples in CI.
• Publish a changelog with migration notes and deprecation dates.
• Announce the change early, then give teams a canary path in one workspace before broad rollout.

Concrete example
If the module currently creates an S3 bucket and I want to add optional access logging, I would add enable_access_logging = false and a new logging block. Existing consumers get the same bucket as before, while teams that want logging can opt in. After a release or two, I can deprecate any old workaround variables without breaking them immediately.

Result
That approach lets teams upgrade on their schedule, keeps state changes predictable, and makes ownership clear.

Pull resources into a module once they're reused across more than one stack or environment, or once a single-purpose group of resources, a VPC, an EKS cluster, a standard RDS setup, has grown enough internal complexity that inlining it clutters the root config. A good module boundary maps to one concern with a small, stable interface; over-modularization is wrapping something in a module because it feels tidy, not because anything actually reuses it or needs the extra indirection, and it shows up as one-resource modules, deep nesting, or modules stitched together with a pile of boolean flags to fake reuse across cases that were never really the same.

When to extract, when to leave inline

• Extract when the resource group is reused across environments, teams, or projects, or its complexity, many variables, repeated lifecycle rules, would otherwise be copy-pasted.
• Leave inline when resources are small, unique to one stack, or tightly coupled to that stack's own variables and lifecycle; wrapping them in a module just adds a layer to look through for no reuse benefit.

Layered module hierarchy
A common shape at organization scale: networking, identity, shared-services, platform, and application layers, each owning a narrow slice and exposing only what the next layer needs, networking exposes VPC and subnet IDs, identity exposes role ARNs, shared-services exposes things like a shared logging bucket, platform composes those into a cluster, application deploys onto the platform. The dependency direction only ever points one way, up: a lower layer like networking must never read or depend on a higher layer's outputs like application, or you get a circular dependency that makes independent deploys impossible. Enforce that with separate state per layer and remote-state or published-lookup reads flowing strictly upward, never the other direction.

The module contract, and encapsulation leaks
A module's real contract is its inputs and outputs, nothing else. The failure mode to design against: consumers start depending on a specific resource name or ID inside the module that it never promised to keep stable, referencing an internal resource address directly instead of a proper output, so a refactor inside the module that changes nothing about its behavior still breaks every consumer. Prevent it by never exposing internal resource addresses, only named outputs, and by treating "adding or renaming an output" with the same breaking-change discipline as changing an input.

Choosing a DRY mechanism

If the same config keeps getting copy-pasted across stacks, start with the cheapest fix, a locals map, before reaching for a full shared module; promote to a module once real behavioral variation, not just value substitution, shows up between consumers.

Trade-offs & pitfalls
Over-modularization is a common overcorrection once a team learns the reuse lesson: they start extracting one-resource modules, or nesting modules three deep, and debugging becomes tracing variable propagation through layers that never needed to exist. A practical rule: start inline, extract when reuse or complexity actually hits, and keep every module's interface small enough that its README fits on one screen.