Infrastructure as Code and Automation Questions

Configuration drift is when the real state of a resource no longer matches what's declared in code, because someone or something changed it outside the normal pipeline, a console click, an autoscaler, or a break-glass fix. Left undetected it causes deploys to fail unexpectedly, opens security and compliance gaps, and eventually causes an apply that either silently reverts someone's emergency fix or fails because the tool's assumptions about current state are wrong. How you detect it scales with team size: a small team can get by with terraform plan in CI and an occasional manual look; hundreds of accounts need continuous, provider-native scanning because nobody can watch that many plans by hand.

Detection methods and their operational cost

Detecting without customer impact, at scale
Detection should always be read-only, plan, describe, and list calls, never anything that touches the resource, so scanning hundreds of accounts never risks the workload it's checking. At enterprise scale that means a central scanning identity with read-only cross-account roles rather than one credential per account to manage, aggregated results in one dashboard, and rate-limited API calls so drift scanning doesn't itself trip service quotas that page someone.

Auto-remediate or alert a human
Not all drift deserves the same response. A tag changed by hand is safe to reconcile automatically; a security group opened to the world, or a change in a hybrid Terraform-and-Ansible environment where Ansible legitimately manages day-2 configuration Terraform doesn't track, should alert a human rather than auto-revert, because auto-remediating could undo a deliberate, valid change made by the other tool. A reasonable default: auto-remediate low-risk, easily-reversible drift; alert and require review for anything security-relevant, stateful, or ambiguous about which tool owns it.

Once drift is found
Confirm it's real first, ruling out a stale cache or a provider-side eventual-consistency blip, identify who or what made the change from audit logs, then decide accept-into-code or revert, and only then act through the normal pipeline rather than another manual edit.

Trade-offs & pitfalls
Scanning more often catches drift sooner but multiplies noisy alerts and API cost; the common failure mode is a team turning off alerting after getting paged for cosmetic drift, which defeats the purpose. Cloud-native config services give the best enterprise-scale coverage but carry their own onboarding cost that a two-person team can't justify, which is exactly why the small-team answer leans on CI plan checks instead.

Direct answer

Terraform state is the record, a JSON file, local or (far more commonly) in a remote backend, of the mapping from every resource block in your configuration to the real-world object it created, plus that object's last-known attribute values. It exists because provider API calls alone can't tell Terraform "this resource address already corresponds to that cloud object" or "here's what its attributes were the last time we looked." State is what lets plan compute an actual diff instead of guessing, and what lets apply update or destroy the correct object instead of creating a duplicate.

What's actually inside the state file

• Resource mapping: resource address to real cloud resource ID, for example aws_instance.web to i-0123456789.
• Attributes and metadata: the last-known values of every attribute, including computed ones the provider filled in, plus timestamps.
• Dependency graph metadata: what depends on what, and which module each resource belongs to.
• Outputs, and any values a data source resolved.
• Terraform version and a serial number, used to detect concurrent state modifications.

What state makes possible

• Idempotent plan/apply: comparing desired config against last-known state, rather than re-describing everything from the provider on every run, lets plan compute a targeted diff and lets repeated applies against unchanged config produce a no-op.
• Resource targeting (-target): Terraform resolves a targeted address using the mapping in state; without that mapping there's nothing to reliably point -target at.
• terraform import: writes an existing resource's real ID into state under a chosen address, which is exactly what lets Terraform manage a resource it didn't originally create.

Worked example: your config declares

resource "aws_instance" "web" {
  ami           = "ami-0123456789"
  instance_type = "t3.micro"
}

After apply, state records aws_instance.web -> i-0abc123... plus every attribute AWS returned. Change instance_type to t3.small and run plan again: Terraform reads the last-known t3.micro value out of state (it doesn't need to re-describe the instance from AWS to know the old value), compares it to the new config's t3.small, and shows exactly ~ instance_type = "t3.micro" -> "t3.small" as an in-place update, a diff it could not compute correctly without a stored last-known value to compare against.

What would actually break without it

• No diffing: every apply would have to either blindly re-create everything (duplicate resources on every run) or re-describe every resource from the provider API on every operation, and many attributes (like a write-only password field) can't be losslessly reconstructed that way even if you tried.
• No safe in-place updates: Terraform wouldn't know that a config change should modify i-0abc123 in place rather than create a new instance, since there's no persisted mapping from config address to that specific real object.
• No dependency-aware destroy ordering: the stored dependency graph is what lets destroy tear things down in reverse-dependency order, and what lets Terraform correctly destroy a resource that's since been removed from config entirely, the current config's graph alone has no idea that resource ever existed.
• No detection of external drift: without a last-known snapshot to diff against, Terraform can't distinguish "this attribute changed outside of Terraform" from "this attribute has always had this value."

Trade-offs and pitfalls

State is also a common source of operational pain on its own: when it goes stale (drifts from what's actually deployed) or gets corrupted (truncated, partially written from an interrupted apply, or hand-edited), plan and apply stop being trustworthy. Recovering from that is its own topic rather than a footnote here, but the short version is diagnosing against a known-good, versioned backup of the state object and repairing with terraform state subcommands (mv, rm, import) rather than hand-editing the JSON directly.

Direct answer

The mechanics are the same regardless of what is being changed: stand up the new version alongside the old one, shift traffic to it gradually (or all at once for blue-green), watch metrics against a defined threshold before promoting further, and have an automated way to shift traffic back if those metrics degrade. What differs by change type is what "the new version" means and which metric actually tells you it is safe, so the mechanics come first below, then how they change for a few different kinds of infrastructure change.

Structured elaboration

Traffic shifting mechanics

• Canary: route a small percentage of traffic to the new version (for example 5%), hold, then ramp (25%, 50%, 100%) with a wait window at each step.
• Blue-green: fully validate the new environment out of band, then cut traffic over in one step (DNS, load balancer listener swap, or weighted routing swap to 100%). Rollback is just swapping back.

What "the new version" and "the signal to watch" are, by change type

Stateless versus stateful

For a stateless service, this is close to mechanical: deregister the old instance from the load balancer, let connections drain, terminate it, and the new instance takes over with no coordination needed between old and new.

For a service with state, for example one backed by a database or holding session or connection state, the "new version" cannot simply replace the old one at will:

• Deregistration has to wait for graceful connection draining, not just for the health check to fail, or you drop in-flight work.
• If the state itself is what is changing (the DB-class case above), the new version has to be replication-caught-up before it sees real traffic, and you need a plan for what happens to writes that land during the cutover window.
• The old and new versions may be reading or writing the same underlying data store, so canary steps need to account for that shared state, not just for the request path.

Worked example

A concrete case: shifting 5% of production traffic on an ALB to a target group fronting instances built from a new AMI, before fully cutting over.

resource "aws_lb_target_group" "green" {
  name     = "app-green"
  port     = 443
  protocol = "HTTPS"
  vpc_id   = var.vpc_id
  health_check {
    path                = "/health"
    healthy_threshold   = 3
    unhealthy_threshold = 2
    timeout             = 2
  }
}

resource "aws_lb_listener_rule" "canary" {
  listener_arn = aws_lb_listener.app.arn
  priority     = 100

  action {
    type = "forward"
    forward {
      target_group {
        arn    = aws_lb_target_group.green.arn
        weight = 5
      }
      target_group {
        arn    = aws_lb_target_group.blue.arn
        weight = 95
      }
    }
  }

  condition {
    path_pattern { values = ["/*"] }
  }
}

Watch the green target group's 5xx rate and p95 latency for a fixed window (say 15 minutes). If it stays within the same range as blue, bump the weight to 25 and repeat; if 5xx rate or latency breaches the threshold at any step, a second terraform apply (or a CI job wrapping it) sets green's weight back to 0 and blue keeps serving everything.

flowchart TD
  A[Provision green environment via Terraform] --> B[Register in target group, run health checks]
  B --> C[Shift 5 percent traffic to green]
  C --> D{Metrics within SLO?}
  D -- Yes --> E[Ramp to 25 percent, then 50 percent]
  E --> F{Still within SLO?}
  F -- Yes --> G[Shift 100 percent traffic, decommission blue]
  F -- No --> H[Roll back: weight to 0 percent, blue keeps serving]
  D -- No --> H

Trade-offs & pitfalls

• The biggest mistake is treating every infra change like an app deploy and watching only 5xx rate. A security-group rule change will not show up as a 5xx, it shows up as connections that never complete; a DB-class change will not show up in the app's error rate until replication lag has already gotten bad. Pick the signal that matches what actually breaks for that change type.
• Canary steps for stateful changes need explicit wait-for-caught-up gates, not just a timer; ramping traffic to a replica that has not finished catching up just moves the failure into user-visible latency or stale reads.
• Automating the rollback path, not just the promotion path, is what actually buys "close to zero downtime." If pulling the plug requires someone to remember the right terraform apply under pressure, the safety only exists on paper.

I would consume the shared network with data sources or, if the network team publishes outputs, with terraform_remote_state. A data source is Terraform’s read-only lookup for existing infrastructure. I would prefer explicit outputs for IDs, because they are less ambiguous than searching by tags.

What I would check

• The VPC ID matches the expected CIDR, for example 10.20.0.0/16
• The subnet count is what I need, for example 2 private subnets in us-east-1a and us-east-1b
• Subnet tags match my assumptions, such as tier=private and env=prod
• All subnets belong to the same VPC

Fail loudly
I would add variable validation plus precondition checks so the plan stops before apply if the layout is wrong. For example, if I expect exactly 2 private subnets and only find 1, the module should error instead of guessing.

That approach keeps the module reusable, but still safe when the shared network changes.

Direct answer

Pick one real automation story where you replaced a manual, error-prone operational task with IaC or configuration management, and be ready to state the before state, what you built, how you proved it was safe, and the after state, in that order. The strongest version of this answer is not "I wrote a script," it is "I removed a specific, recurring source of toil and showed I validated the change before it touched production."

Structured elaboration

Use STAR, but weight it toward what interviewers are actually probing:

• Situation: name the manual task and quantify the pain in terms a non-expert would recognize (how often it happened, how long it took, what broke because of it). Vague pain ("it was annoying") does not land; specific pain does.
• Task: state your actual scope and ownership. Were you asked to fix an incident pattern, reduce provisioning time, or both?
• Action: this is where seniority shows. Interviewers want the tool-choice reasoning (why Terraform or CloudFormation for provisioning, why Ansible or Chef for configuration, or why both), and the validation strategy before it touched anything real: automated checks against the IaC itself, a staged rollout (dev, then a canary environment, then production), and a defined rollback path.
• Result: tie the outcome directly back to the pain named in Situation. If you claimed provisioning was slow, report the new provisioning time. If you claimed incidents were frequent, report the incident count after.

Worked example

Situation: a payment-processing team was provisioning Kafka clusters and their Prometheus exporters by hand for every new environment. Manual setup took about 4 hours (240 minutes) and regularly produced misconfigured ACLs or missing exporter flags, causing 2 to 3 incidents per quarter.

Task: cut that toil and eliminate the config drift causing the incidents.

Action:

• Wrote Terraform modules for the cloud infrastructure (VPC resources, security groups, IAM roles) and Helm charts (Kubernetes' packaging and templating tool) with a standardized values.yaml (the file that sets each environment's specific settings) template for the Kafka brokers and exporters.
• Added Ansible playbooks for post-deploy validation and ACL bootstrapping.
• Built automated checks before anything touched a real environment: Terraform plan validation and linting, chart linting, and an Ansible dry run.
• Rolled out in stages: dev, then a single canary (a small-scale test rollout run before going fully live) staging cluster monitored for 48 hours against SLOs (service level objectives, the reliability targets already committed to), then production, gated behind CI so infrastructure only applied after tests passed.

Result: provisioning dropped from about 240 minutes to about 10, which is more than 20x faster (240 divided by 10 is 24). Configuration-related incidents went from 2 to 3 a quarter to zero over the following two quarters, because the class of error causing them (hand-typed ACLs and exporter flags) was now generated from the same reviewed template every time.

Trade-offs & pitfalls

• The most common weak version of this story skips the validation step entirely and jumps from "I wrote the automation" to "it worked." That is indistinguishable from luck. Naming what you tested, and what would have happened if you had shipped without testing it, is what separates a senior answer.
• Automation also changes the blast radius of a mistake: a bad manual change breaks one environment, a bad automated change can break every environment it touches in one run. Staged rollout and a canary window exist specifically to catch that before it is everywhere, and a good answer says so explicitly.
• Be honest about scope. If the automation only covered provisioning and not day-2 operations (patching, decommissioning), say so rather than implying it solved everything.