Infrastructure as Code and Automation Questions

Direct answer

Reach for a full image rebuild-and-replace when the patch touches anything below the application layer, kernel updates, base OS packages, init system, or chains together many dependency changes, because the risk of a botched imperative patch script or accumulated drift outweighs the cost of building a new golden image. Reach for in-place patching when the change is a small, app-level or config-only update where a full rebuild is genuine overkill. Either way, roll it out as a canary against a defined set of signals before going wide, with a rollback path decided in advance, not improvised after something looks wrong.

Structured elaboration

Deciding between rebuild-and-replace and in-place

Canarying either way

• Stage 0: internal integration or smoke tests before touching any fleet member.
• Stage 1, canary: patch a small percentage of the fleet, spread across availability zones so a single zone's noise does not look like a global signal, and hold for an observation window before proceeding.
• Stage 2: ramp in defined steps (for example 10 percent, then 25, then 50, then 100), with an automated gate at each step rather than a single all-or-nothing jump.

What to watch to catch a bad patch

• Service-level: error rate and latency against baseline, not just absolute thresholds, since a canary group is a small sample and needs to be compared to the rest of the fleet running the old version concurrently.
• Host-level: boot success, health-check pass rate, process crash or restart counts, config validation failures.
• A hard rule for automatic rollback: for example, error rate on the canary group exceeding the control group's by some fixed margin, sustained across the observation window, not a single noisy data point.

Rolling it back

• Image rebuild-and-replace: repoint the load balancer/instance group at the previous image version; this is fast precisely because nothing on the bad instances needs to be fixed, they are simply replaced again.
• In-place: run the equivalent revert through the same configuration-management tool that applied the patch; if the patch is not cleanly revertible (for example it already ran a one-way data migration), that alone is a strong argument the patch should have gone out via image rebuild instead.

Worked example

Say the fleet has 200 instances and the patch is a base-OS security update, favoring image rebuild-and-replace given the criteria above. A canary of 2 percent of the fleet is 4 instances (4 divided by 200 equals 0.02), which is small enough to bound the blast radius of a bad patch to a handful of hosts, while still being enough instances that a real regression shows up as a pattern across more than one host rather than looking like noise from a single flaky instance. If those 4 instances hold steady against the defined error-rate and latency thresholds through the observation window, the rollout proceeds to the next step (for example 20 instances, 10 percent); if not, the load balancer is repointed back to the previous image version for those 4 instances immediately.

Trade-offs & pitfalls

• Image rebuild-and-replace adds real pipeline overhead, building, testing, and distributing a new golden image is slower to iterate on than a quick in-place script, so reserving it for changes that actually warrant it (rather than defaulting to it for everything) matters for how often the team can ship patches at all.
• In-place canarying only works if the patch can be cleanly reverted; a patch that has already made a one-way change (data migration, irreversible config transform) mid-canary leaves no clean rollback path, which is itself a signal that patch should have gone out as an image replacement.
• Scheduling patches to respect maintenance windows across regions and time zones adds coordination cost; a globally staggered rollout reduces blast radius per region but takes longer end to end, which is a real trade-off against how quickly a critical security patch needs to reach the whole fleet.

Direct answer

A resource block declares something Terraform owns the full lifecycle of: it will create, update, or destroy it to match what's declared. A data block reads information about something that already exists, without ever creating, changing, or destroying it. You reach for a resource when you want Terraform to own an object; you reach for a data source when you need to reference something that already exists, whether that's managed by another team, another Terraform config, or was created outside Terraform entirely.

Definitions and when to use which

• resource: Terraform issues API calls to bring the real object to the declared state, and records that object in state as something it manages.
• data source: Terraform issues read-only API calls to fetch attributes of an existing object, and does not persist it in state as a managed object.

Worked example

# resource: Terraform owns this EC2 instance's full lifecycle
resource "aws_instance" "web" {
  ami           = data.aws_ami.ubuntu.id
  instance_type = "t3.micro"
  subnet_id     = data.aws_vpc.prod.id
}

# data source: reads an existing, externally-managed AMI
data "aws_ami" "ubuntu" {
  most_recent = true
  owners      = ["099720109477"]
  filter {
    name   = "name"
    values = ["ubuntu/images/*"]
  }
}

# data source: reads a VPC this config does not manage
data "aws_vpc" "prod" {
  filter {
    name   = "tag:Name"
    values = ["prod-vpc"]
  }
}

Here aws_instance.web is the only resource, the thing this module actually provisions. The AMI and VPC are read through data sources because some other process owns them, a shared networking module owns the VPC, and AWS itself owns the published Ubuntu AMI.

Effect on plan and the dependency graph

• Plan: resources show up with create/update/destroy actions; data sources show up as reads and never propose a destructive action themselves, but their fetched values can still change what a downstream resource's plan looks like (a new AMI ID means the instance using it may show a forced replacement).
• State: resources are recorded as managed objects; data sources are not, only the values they return get used elsewhere in the config.
• Dependency graph: referencing data.aws_vpc.prod.id inside aws_instance.web creates an implicit dependency, Terraform reads the data source before it can plan the resource that consumes it. If the object a data source reads is itself managed by a different resource in the same configuration, referencing it directly (rather than through a data source) creates an explicit dependency edge instead; mixing the two for the same object, reading via data something also managed via resource in the same run, can create ordering ambiguity, since the data source might read a stale value from before that run's own changes land.

Trade-offs and pitfalls

• A data source is re-read on every plan (and refresh), so if the object it points at changes outside Terraform, your plan can shift underneath you with no corresponding resource change in your own config to explain why. This is a common source of "why did my plan suddenly change, I didn't touch anything" surprises.
• Prefer explicit remote-state outputs over a data source when you're referencing something managed by a sibling Terraform config in your own org, since outputs give you an explicit contract and version boundary; reserve data sources for things genuinely outside Terraform's control (cloud-provider-published AMIs, resources owned by another team's tooling entirely).
• Referencing a shared object (a shared VPC, a shared KMS key) via a data source from many independent configs works, but it also means none of those configs can see or coordinate around each other's dependency on it; a breaking change to that shared object has no single blast-radius list to check.

I would put both human review and automation around the plan. A Terraform plan is a preview of create, update, destroy, and replace actions. Replace is especially risky because Terraform deletes and recreates a resource.

Guardrails

• Run terraform fmt -check and terraform validate in CI
• Save the plan as an artifact and require approval before apply
• Fail the pipeline if the plan includes unexpected destroys or replacements on critical resources
• Use policy checks for things like public exposure, open security groups, or deletion of databases
• Add prevent_destroy to the few resources that should almost never be removed

Example
If the plan shows aws_db_instance.main will be replaced because of an engine change, I would force manual review from an engineer and, ideally, a service owner. If the plan only adds an autoscaling instance, that can follow the normal path.

This catches surprises before users feel them, which is the real goal.

Direct answer

Run checks in a layered pipeline ordered from cheapest to most expensive: format and lint first (seconds, no cloud access), then static policy/security scanning against the plan or template (still no cloud access), then anything that actually stands up real resources (integration tests, plan review with real credentials), and finally post-deploy verification against the live environment. Each layer is designed to catch a different class of mistake, and putting the cheap checks first means a typo never has to wait for an expensive real-resource test to fail.

The two fundamentally different kinds of check

Before mapping tools to stages, it's worth naming the split explicitly: some checks are fast, syntax- or policy-level, and need no cloud access at all; others genuinely stand up real (usually short-lived) infrastructure to prove it actually works. Confusing the two, or skipping straight to the expensive kind, is the most common mistake in a pipeline like this.

Worked example: a pipeline definition

jobs:
  fmt-and-lint:
    steps:
      - run: terraform fmt -check -recursive
      - run: tflint

  policy-scan:
    needs: fmt-and-lint
    steps:
      - run: checkov -d . --framework terraform

  plan:
    needs: policy-scan
    steps:
      - run: terraform plan -out=plan.tfplan
      - run: terraform show -json plan.tfplan > plan.json
      # a script here fails the job if plan.json contains an unreviewed delete/replace

  integration-test:
    needs: plan
    if: github.event_name == 'push' && github.ref == 'refs/heads/main'
    steps:
      - run: go test ./test/... -run TestModule -timeout 30m
      # this job is the one that actually provisions and tears down real infra

  apply:
    needs: integration-test
    environment: production   # requires manual approval
    steps:
      - run: terraform apply plan.tfplan

The ordering matters: fmt-and-lint and policy-scan run on every commit because they're free and fast; integration-test, which provisions real infrastructure, is scoped to run only on merges to main, not on every push to a feature branch, to bound cost and time.

Trade-offs and pitfalls

• Skipping straight to integration tests (or relying on them to catch what a linter would have caught in seconds) is slow and expensive for no extra safety, put the cheap checks first and let them fail fast.
• Running the real-resource layer on every commit, rather than on merge or nightly, is the single most common way teams accidentally burn cloud spend on a testing pipeline.
• Static policy scanning only catches what it has rules for; it gives false confidence if the rule set isn't kept current with new resource types the team starts using.
• A plan-review gate that only a human reads doesn't scale, encode the "never allow an unreviewed delete on a protected resource" rule as an automated check on the plan JSON, not just a habit.

I would make the module interface small, typed, and caller-driven. A module should accept inputs through variables and return only the outputs another team needs.

Inputs

• env for environment name like dev, staging, prod
• name_prefix for naming consistency
• tags as a map(string) so teams can add ownership and cost-center labels
• sizing inputs, such as instance_class or cpu_count
• network inputs like subnet_ids and vpc_id

Design rules

• Use defaults only for safe values, not for important architecture choices
• Add validation so bad values fail early, such as rejecting an empty subnet list
• Use stable names and tags inside the module, but let callers control the prefix
• Output IDs, ARNs, DNS names, or endpoints, not whole resources

Example
Dev can pass instance_class = "t3.small" and prod can pass instance_class = "m6i.large", while the module code stays unchanged. That lets each environment differ in size and naming without forking the module.