Infrastructure as Code and Automation Questions
Defining, provisioning, and automating infrastructure programmatically. Covers declarative IaC with Terraform and comparable tools like CloudFormation (resource and provider model, state management and remote backends, module design and reuse, workspaces, drift detection, and safe plan/apply workflows), plus the broader automation discipline: provisioning pipelines, golden-image and machine-image building, scripting glue, self-service platforms, and end-to-end environment stand-up. The authoring, lifecycle, and automation of infrastructure code that reduces manual toil across provisioning workflows.
Walk me through a practical testing strategy for infrastructure code: what belongs at the unit layer, what needs a real integration test, and what only shows up in an end-to-end run. What would you actually assert at each layer, and where does each one run so you're not burning cloud spend on every commit?
Sample Answer
Direct answer
Use a three-layer pyramid: unit tests validate syntax, style, and module logic with no cloud access at all; integration tests provision real (or containerized) resources in a short-lived, isolated environment and assert they actually behave correctly; end-to-end tests exercise a full ephemeral environment stand-up as one workflow. The payoff isn't code elegance, it's catching a destructive, unintended change (a replace or delete that shouldn't happen) before it ever reaches production, while keeping each layer cheap and fast enough that people don't quietly route around it because it's too slow to bother with.
The three layers, what each one asserts, and where it runs
| Layer | What you assert | Where it runs | Needs real cloud resources? |
|---|---|---|---|
| Unit | Module output types and values, required-variable defaults, template rendering, policy-as-code rules against the plan JSON | Every PR, on every commit | No |
| Integration | The resource was actually created with the right properties (tags, encryption, IAM statements), a service responds correctly on a real or containerized host | On merge to main, or nightly, in an ephemeral account | Yes, but short-lived and scoped small |
| End-to-end | The full workflow, multi-resource stand-up, service boots and passes a real health check, deploy-time automation (dashboards, alerts) actually fires | Scheduled or pre-prod gate | Yes, a full ephemeral environment |
This same shape holds regardless of whether the codebase is a single Terraform module or a shared body of infrastructure code spanning Terraform, CloudFormation, and Ansible end to end: each tool's unit layer stays fast and cloud-free, each tool's integration layer provisions real-but-short-lived resources, and the full picture only comes together in an ephemeral-account end-to-end run. Keep every layer fast and cheap enough that engineers don't route around it, because a test suite people skip protects nobody.
At platform scale (say, a platform managing on the order of 10,000 resources), the binding constraint shifts from "can I test this" to "can I test this fast enough, and cheap enough, while the resource count keeps growing." Shard integration tests by service or module boundary and run shards in parallel, and track CI feedback time as a first-class budget right alongside cloud spend, since at that scale both keep growing together and either one silently degrading pushes people back toward skipping tests.
Layer note for Ansible: the same three-way split applies to roles
- Unit-equivalent:
ansible-lint,yamllint, and Molecule's lint/syntax scenario, no hosts involved. - Integration: Molecule running the role against both a container (fast, cheap, good for most logic) and, on a slower cadence, a real cloud instance, since a container can't catch everything, cloud-init behavior and kernel-level specifics only show up on a real instance.
- Mock any external service the role calls (an internal API, a secrets endpoint) with a stub, so the test's pass/fail depends on the role, not on a live dependency being up.
- Run role test scenarios in parallel across cases to keep CI time bounded as the number of roles grows.
- Explicitly assert what happens when a task fails partway through a play, not just the happy path: does the right handler still fire, does the play halt cleanly, is it safe to re-run.
Worked example: a Terratest check and an idempotent Ansible playbook
Terratest (Go), asserting a real S3 bucket's encryption and versioning via the AWS SDK:
func TestS3BucketEncryptionAndVersioning(t *testing.T) {
opts := &terraform.Options{
TerraformDir: "../modules/secure-bucket",
Vars: map[string]interface{}{
"bucket_name": "tftest-" + strings.ToLower(random.UniqueId()),
},
}
defer terraform.Destroy(t, opts) // teardown always runs, even if an assertion fails
terraform.InitAndApply(t, opts)
bucketName := terraform.Output(t, opts, "bucket_name")
sess := session.Must(session.NewSession(&aws.Config{Region: aws.String("us-east-1")}))
s3Client := s3.New(sess)
enc, err := s3Client.GetBucketEncryption(&s3.GetBucketEncryptionInput{Bucket: aws.String(bucketName)})
require.NoError(t, err)
assert.Equal(t, "AES256", *enc.ServerSideEncryptionConfiguration.Rules[0].ApplyServerSideEncryptionByDefault.SSEAlgorithm)
ver, err := s3Client.GetBucketVersioning(&s3.GetBucketVersioningInput{Bucket: aws.String(bucketName)})
require.NoError(t, err)
assert.Equal(t, "Enabled", *ver.Status)
}
Provision the bucket for real, its encryption and versioning behavior is exactly what's under test and can't be meaningfully faked. Mock or stub anything the test depends on but isn't the subject under test (say, an internal metadata-registration call the module also makes). Random-suffix the bucket name so parallel CI runs never collide, and always defer the destroy so a failed assertion doesn't leak a real bucket.
Ansible: a playbook that only restarts a service when its config actually changes
- name: Provision deploy user with SSH access
hosts: app_servers
become: true
tasks:
- name: Ensure deploy user exists
ansible.builtin.user:
name: deploy
shell: /bin/bash
create_home: true
- name: Install deploy user's SSH public key
ansible.posix.authorized_key:
user: deploy
key: "{{ lookup('file', 'files/deploy_id_ed25519.pub') }}"
- name: Grant scoped sudo to the deploy user
ansible.builtin.copy:
dest: /etc/sudoers.d/deploy
content: "deploy ALL=(ALL) NOPASSWD:ALL\n"
mode: "0440"
validate: "visudo -cf %s"
- name: Render app config from template
ansible.builtin.template:
src: templates/app.conf.j2
dest: /etc/app/app.conf
mode: "0644"
notify: restart app
handlers:
- name: restart app
ansible.builtin.service:
name: app
state: restarted
The user, authorized_key, copy, and template modules are all naturally idempotent, a second run reports no changes unless the source content changed. The service only restarts because the template task fires the notify handler when it detects a real change, not on every run. The same only-restart-on-change pattern is how you'd validate an idempotent NTP install too: template /etc/ntp.conf, notify a restart handler only on change, and run the whole play in check mode (--check) first in CI to confirm it reports zero changes against a clean box before trusting it in production.
Trade-offs, pitfalls, and catching idempotency bugs specifically
- A playbook or script that silently isn't idempotent, it changes something on a second run that it shouldn't, is one of the most common ways infrastructure automation testing fails quietly. Catch it with a required CI gate, not a manual habit: run the same test twice and assert the second run reports zero changes (Ansible:
changed == falseon the second play; Terraform: a secondplanshows no diff). Making that double-run assertion mandatory, rather than something someone remembers to check, is what stops a non-idempotent change from merging in the first place and slipping through again later. - Over-mocking is its own failure mode: a test that stubs out everything never catches a real provider bug or a real API behavior change, so reserve mocking for genuine third-party dependencies, not the resource actually under test.
- Shared cloud resources or rate limits across parallel test runs cause flaky failures that look like test bugs but are really infrastructure contention; isolate test resources per run (unique names, separate accounts if the budget allows) rather than sharing a pool.
- Cost creep at scale is gradual and easy to miss until a monthly bill spikes; track integration-test cloud spend as its own line item, not folded into general CI cost.
Walk me through blue-green versus canary deployment from an infrastructure perspective; when would you reach for each? Think about traffic routing, resource duplication, cost, and what gets harder when a database schema change is part of the rollout.
Sample Answer
Direct answer
Blue-green gives you a full, already-validated environment and an instant, simple rollback, at the cost of running two complete copies of your infrastructure during the change; canary gives you gradual, lower-risk validation at the cost of more routing and monitoring complexity, and it is harder to reason about while it is in flight. Pick blue-green when a clean, all-or-nothing cutover is affordable and rollback speed matters most; pick canary when the blast radius of being wrong needs to stay small and you can invest in the routing and monitoring to support gradual exposure.
Structured elaboration
Comparison
| Dimension | Blue-green | Canary |
|---|---|---|
| Traffic routing | Atomic cutover: DNS, listener swap, or weight flip to 100% | Gradual: weighted routing or service-mesh rules, ramped over time |
| Resource duplication / cost | Full duplicate environment while both are live, higher but short-lived cost | Only a fraction of the fleet runs the new version at any point, lower peak cost |
| Rollback | Instant, swap back to the known-good environment | Ramp weight back down; usually just as fast, but only a subset of traffic was ever exposed |
| Blast radius if wrong | All traffic hits the new version the moment you cut over | Bounded to the canary percentage until you choose to widen it |
| Operational complexity | Lower: mostly "which environment is live" | Higher: routing rules, staged thresholds, automated promote/rollback logic |
| Database schema changes | Blue and green must both work against the same schema at the moment of cutover, since there is no gradual overlap window; this forces backward and forward compatible migrations (expand, migrate, contract) or a maintenance window | Old and new versions coexist against the same schema for longer, so the compatibility window has to hold for however long the canary runs, not just for an instant |
Other strategies worth naming
Blue-green and canary are not the only tools:
- Progressive expansion: widen the set of inputs a change applies to in stages rather than widening a traffic percentage, for example rolling a new IAM policy or network ACL out to one account, then one region, then everywhere, independent of any single request's traffic path.
- Feature flags for infra: gate exposure to a new code or config path behind a flag rather than, or in addition to, a routing change, so the change can be turned off instantly without touching load-balancer configuration at all. This is especially useful when the change is a behavior toggle inside already-deployed infrastructure rather than a new fleet.
Why this gets harder for networking or storage than for a stateless app release
Blue-green and canary are easy to describe for a stateless app release because the unit being duplicated (a fleet of identical, disposable instances) and the unit being routed (an HTTP request) line up cleanly. That stops being true for infrastructure-level changes:
- Networking: a routing or firewall change often affects an entire connection or an entire flow, not a single request, so traffic cannot always be shifted 5% at a time the way it can with an HTTP load balancer. A stateful TCP connection, a VPN tunnel, or a peering change uses the old path or the new path for its whole lifetime; the workaround is to canary at the level of an entire subnet, account, or long-lived connection cohort rather than per request, and to accept a coarser blast radius.
- Storage: there is no cheap duplicate of a stateful data store the way there is for a stateless instance. Standing up a second, fully synced database is expensive and introduces replication lag as a new failure mode. The common workaround is to validate the new storage layer with shadow traffic or dual writes before anything reads from it in production, and to use expand-then-contract schema migrations so both the old and new consumers can run against the same underlying store during the transition instead of trying to canary the store itself.
Worked example
A schema change on a Postgres-backed service being rolled out with blue-green: adding a NOT NULL column safely requires an expand-then-contract sequence rather than a single migration, because both blue and green must work against the same schema during cutover.
- Expand: add the column as nullable, deploy it (both blue and green tolerate a nullable column).
- Backfill: populate the column for existing rows.
- Cut over blue-green as normal, both versions still tolerate the nullable column.
- Contract: once green is fully promoted and blue is decommissioned, add the
NOT NULLconstraint in a separate migration, since only green's code path needs to rely on it.
This is the same three-step compatibility pattern regardless of whether the deployment strategy is blue-green or canary; what changes is only how long the "both versions must tolerate the old and new shape" window needs to hold.
Trade-offs & pitfalls
- Choosing canary by default because it "sounds safer," without the routing and monitoring maturity to support it, is a common mistake; a canary that cannot actually be measured is worse than a blue-green cutover that can be instantly reversed.
- Forgetting that duplication cost for blue-green is temporal, not permanent, leads teams to avoid it for cost reasons when the actual bill is only for the cutover window.
- Assuming the app-release playbook (weighted HTTP routing) transfers directly to networking or storage changes is the single most common failure mode; check whether the thing being changed is even divisible at the granularity being planned for the canary.
You are moving a production environment from local state to a remote backend shared by the team. What design choices would you make around locking, access control, and failure recovery so concurrent work does not corrupt the environment?
Sample Answer
I would move production to a remote backend that supports both shared access and state locking. State is Terraform’s record of what it created, and locking means only one writer can change that record at a time.
Design choices
- Use one backend per environment, so prod does not share state with dev
- Turn on encryption at rest for the backend storage
- Restrict write access to CI and a small break-glass admin group
- Give most engineers read-only access to plans and state history
- Require an exclusive lock for
apply, with a short timeout and clear failure message
Recovery
I would also enable versioning or history so I can recover a prior state file if an apply fails halfway through. If a run breaks, I would not edit state first. I would inspect the lock, compare the current cloud resources with terraform plan -refresh-only, and then decide whether to fix drift, import an orphan, or roll back.
Example
If a colleague applies from their workstation while CI is planning, the lock should stop the second write. That prevents two people from corrupting the same prod state.
You're advising a team moving from long-lived servers to baked images. What are the trade-offs against staying with configuration management on mutable servers, especially around patching, hotfixes, debugging, and deployment speed?
Sample Answer
Direct answer
Recommend immutable, baked images (images built once with everything preinstalled, then deployed as-is rather than configured after boot) as the default for anything at fleet scale, because they buy reproducibility and safe rollback; keep configuration management for runtime-only concerns (secrets injection, feature toggles, small dynamic config) and as a deliberate bridge during migration, not as the long-term patching strategy. The trade-offs mostly show up in four places: how patches ship, how urgent hotfixes get applied, how debugging works, and how deployment speed scales with fleet size. That default flips for a stateful service, which needs its own answer below.
Structured elaboration
Comparison
| Concern | Baked images (immutable) | Configuration management (mutable) |
|---|---|---|
| Patching | Rebuild the image with the patch, redeploy via rolling replace; consistent across the whole fleet, easy to roll back by pointing at the previous image | Apply in place with Ansible or Chef across the fleet; fast for a handful of hosts, but risks partial application and drift if a run fails partway |
| Hotfixes | Requires a rebuild-and-replace cycle; slower to first fix, but the fix is guaranteed to be in every future image, not just the running instances | Can be applied immediately to running hosts; faster time to fix, but the fix has to be separately folded back into the source of truth or it silently disappears on the next rebuild |
| Debugging | The exact environment is reproducible from the image ID and version, so "what is actually running" is never in doubt; live debugging still needs logging and tracing rather than SSH-and-poke | Easy to SSH in and iterate directly on a running host, which can be faster in the moment, but every such change is an undocumented, untracked snowflake until someone remembers to encode it |
| Deployment speed at scale | Slower for a single one-off change (build, test, replace), faster in aggregate as fleet size grows, because every instance is guaranteed identical and rollouts are automatable | Faster for a handful of hosts, but velocity degrades as fleet size grows because keeping every host consistent becomes a manual, error-prone effort |
When each makes sense
- Prefer immutable images for stateless, autoscaled fleets: the automation investment (image pipeline, CI, registry) pays for itself once more than a handful of instances are being managed.
- Keep configuration management for genuinely dynamic, runtime concerns: injecting secrets at boot, toggling a feature flag, or configuring something that legitimately needs to change without a full rebuild.
- During a migration specifically, a hybrid approach is normal and reasonable: bake the OS and runtime into the image, and use lightweight, idempotent configuration at boot for the parts that still need to vary per environment. That is not indecision, it is how most teams actually get from mutable to immutable without a big-bang cutover.
The stateful-service case
For a stateful service, for example a database node or a broker holding local queue data on disk, naive image-replacement breaks the pattern above. Terminating an old instance and swapping in a fresh one from a new image throws away whatever state lived only on that instance; the deregister-drain-terminate flow that makes stateless replacement safe does not, by itself, move that state anywhere. There is also a handoff problem a stateless fleet does not have: something has to own transferring or re-syncing the state (or explicitly failing it over) before the old instance goes away, not just draining its connections.
Two realistic patterns:
- Separate state from compute where possible. Move the data itself onto something outside the instance's lifecycle, an external volume that reattaches to the new instance, or a managed data store, so the compute layer becomes stateless again and can be image-replaced exactly like the fleet case above. This is the preferred pattern whenever the workload allows it, because it lets the stateful service inherit the same rollback and testing story as everything else.
- In-place with strict drift controls when separation is not possible. If state genuinely has to live on the box, or the migration to external state has not happened yet, stay with configuration management for that host, but treat convergence checks as a required gate after every run rather than an optional audit, since a failed run on a stateful host is far more costly to miss than on a stateless one.
The same three axes from the comparison above still drive the decision, but they cut differently here:
- Rollback complexity: rolling back a baked image is instant for stateless compute, but rolling back a stateful host means rolling back its data too, not just its binaries; an image-based rollback plan for a stateful service has to include a data restore or failover step, not just a launch-template revert.
- Configuration drift: drift is more dangerous to tolerate on a stateful host, because a missed convergence run there does not just leave a config mismatch, it can leave that host holding data no other instance has, which makes it unsafe to simply terminate and replace.
- Testing requirements: testing a stateful change means testing against a realistic copy of the data (a snapshot or a replica), not just a fresh boot; a stateless image can be smoke-tested cold, a stateful one usually cannot.
Worked example
A team migrating a fleet of 40 application servers from Chef-managed mutable hosts to baked AMIs, mid-migration: 15 servers have already moved to immutable images, 25 are still Chef-managed.
- An OS-level CVE needs patching across all 40. For the 15 immutable servers: rebuild the AMI, run it through the standard test and canary gates, and replace those instances via a rolling instance refresh, with no manual steps on any host.
- For the 25 still-mutable servers: run the existing Chef cookbook update across them directly, which is faster to start for that subset but needs a fleet-wide Chef run to confirm every one of the 25 actually converged, since a failed run on any single host leaves it unpatched and undetected until the next audit.
- This is exactly the operational cost mutable configuration carries at scale: the same patch requires two different verification strategies depending on which half of the fleet a given host is in, and the mutable half needs an explicit convergence check the immutable half gets for free from "the image ID changed."
Trade-offs & pitfalls
- The most common mistake is presenting this as all or nothing. A hybrid, migration-aware answer is the stronger answer; a purist "always immutable" answer that ignores legitimate runtime-config needs is not.
- Debugging discipline has to change with immutability: if engineers keep SSHing into production and hand-patching "just this once," the fleet quietly becomes as inconsistent as the mutable model it was supposed to replace, just with extra image-pipeline overhead on top.
- Do not understate the upfront cost of the image pipeline (CI, registry, automated tests, rolling-replace tooling); it is real engineering investment that has to be justified against the ongoing toil of drift remediation and manual patch campaigns it replaces.
- For a stateful service, do not default to "just bake it anyway": image-replacement that ignores where the state lives is not a safer version of the stateless pattern, it is a data-loss risk wearing the same rollout choreography.
Outline a CI/CD pipeline for Terraform: what runs when someone opens a PR, what needs a human approval before a production apply, and how non-production changes get applied automatically. Where would you keep the credentials it needs?
Sample Answer
A safe Terraform CI/CD pipeline treats every pull request as a proposal and every merge as an execution decision gated by blast radius (how much could break if this specific change goes wrong). On PR open, CI runs formatting, validation, and a plan, then posts the plan as a PR comment so reviewers see the actual infrastructure diff, not just the code diff. Merges that only touch non-production branches or workspaces auto-apply immediately once the PR is approved and merged. Merges that touch production require a separate, explicit human approval gate before anything runs against production state, and the pipeline authenticates to the cloud provider with short-lived, per-environment credentials issued via OIDC federation rather than static keys sitting in CI secrets.
Pipeline stages
- PR opened: checkout,
terraform fmt -check,terraform validate,terraform plan -out=plan.tfplanagainst the target workspace's (Terraform's mechanism for keeping one environment's state separate from another's) remote state, a policy check (a tool such as OPA or Sentinel that automatically evaluates the plan against rules) for guardrails such as no public buckets or required tags, then post the plan output as a PR comment (a plan-formatting GitHub Action, or Atlantis/Terraform Cloud's native PR integration). - Non-production merge: on merge to a
dev/stagingbranch, or when the changed path maps to a non-prod workspace, the pipeline re-plans against current state and auto-applies if the plan is clean. - Production merge: on merge to
main, or when the path maps to a prod workspace, the pipeline re-plans and stops. Apply requires a required reviewer on a protected environment, or a human confirming in Terraform Cloud/Atlantis. The apply step re-plans immediately before applying, since time has passed since the last plan and drift could have crept in. - Remote state and locking: state lives in a backend with native locking (S3 with a DynamoDB lock table, Terraform Cloud, or a Postgres backend with locking) so two runs can never apply concurrently against the same workspace.
flowchart LR
A[PR opened] --> B[CI: fmt + validate + plan]
B --> C[Post plan as PR comment]
C --> D{Merge target}
D -->|non-prod branch| E[Auto-apply on merge]
D -->|main / prod| F[Required reviewer approval]
F --> G[Protected environment gate]
G --> H[Manual apply trigger]
E --> I[(Remote state + lock)]
H --> I
I --> J[OIDC short-lived cloud role]
Where credentials live
- The pipeline should never hold long-lived cloud access keys. Use OIDC federation: the CI system exchanges a short-lived OIDC token for a cloud IAM role scoped to exactly one environment, for example a
terraform-dev-deployrole for non-prod and a separate, more restrictedterraform-prod-deployrole for prod. - The role used to plan can be narrower (read-only, broad enough to describe resources) than the role used to apply; some teams even use different identities for plan (safe to run on any PR) versus apply (only runs on trusted, merged code).
- Secrets the Terraform code itself needs, like database passwords or provider API keys, come from a secrets manager (Vault, or a cloud-native secrets store) referenced at runtime, never hardcoded in
.tfvarscommitted to the repo.
Worked example
A concrete two-branch layout: the dev branch maps to workspace dev, and main maps to workspace prod.
- An engineer opens a PR from
feature/add-cacheintodev. CI plans using the read-scopedterraform-dev-deployrole and posts the diff as a PR comment. - The PR is approved and merged into
dev. A merge-triggered workflow re-plans and applies automatically using the apply-scopedterraform-dev-deployrole, no human in the loop. - Once validated in
dev, a second PR mergesdevintomain. CI plans against theprodworkspace and posts that diff too. - Because the target is
main, the apply job sits behind a protected "production" environment requiring one designated approver. The approver reviews the plan output itself, not just the code diff, approves, and only then does the apply job run usingterraform-prod-deploy.
Trade-offs & pitfalls
Re-plan immediately before apply instead of reusing the PR-time plan: a long-lived PR or a busy shared environment means the world can move between plan and apply, and applying a stale plan is a common source of unpleasant surprises. Terraform Cloud or Atlantis reduce how much pipeline glue you write yourself (state, locking, the plan-comment UI, approvals) at the cost of another system to operate; rolling it in plain CI with an S3 backend gives full control but you own more of the plumbing. OIDC federation removes the need to rotate static secrets but requires upfront IAM trust-policy setup per environment; skipping it for "just get it working" static keys is the most common regression back to a real security gap. Auto-apply on non-prod is only safe if "non-prod" genuinely means low blast radius; a shared staging environment other teams demo against deserves the same gate as prod.
Unlock Full Question Bank
Get access to all Infrastructure as Code and Automation interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.