Infrastructure as Code and Automation Questions
Defining, provisioning, and automating infrastructure programmatically. Covers declarative IaC with Terraform and comparable tools like CloudFormation (resource and provider model, state management and remote backends, module design and reuse, workspaces, drift detection, and safe plan/apply workflows), plus the broader automation discipline: provisioning pipelines, golden-image and machine-image building, scripting glue, self-service platforms, and end-to-end environment stand-up. The authoring, lifecycle, and automation of infrastructure code that reduces manual toil across provisioning workflows.
Outline a CI/CD pipeline for Terraform: what runs when someone opens a PR, what needs a human approval before a production apply, and how non-production changes get applied automatically. Where would you keep the credentials it needs?
Sample Answer
A safe Terraform CI/CD pipeline treats every pull request as a proposal and every merge as an execution decision gated by blast radius (how much could break if this specific change goes wrong). On PR open, CI runs formatting, validation, and a plan, then posts the plan as a PR comment so reviewers see the actual infrastructure diff, not just the code diff. Merges that only touch non-production branches or workspaces auto-apply immediately once the PR is approved and merged. Merges that touch production require a separate, explicit human approval gate before anything runs against production state, and the pipeline authenticates to the cloud provider with short-lived, per-environment credentials issued via OIDC federation rather than static keys sitting in CI secrets.
Pipeline stages
- PR opened: checkout,
terraform fmt -check,terraform validate,terraform plan -out=plan.tfplanagainst the target workspace's (Terraform's mechanism for keeping one environment's state separate from another's) remote state, a policy check (a tool such as OPA or Sentinel that automatically evaluates the plan against rules) for guardrails such as no public buckets or required tags, then post the plan output as a PR comment (a plan-formatting GitHub Action, or Atlantis/Terraform Cloud's native PR integration). - Non-production merge: on merge to a
dev/stagingbranch, or when the changed path maps to a non-prod workspace, the pipeline re-plans against current state and auto-applies if the plan is clean. - Production merge: on merge to
main, or when the path maps to a prod workspace, the pipeline re-plans and stops. Apply requires a required reviewer on a protected environment, or a human confirming in Terraform Cloud/Atlantis. The apply step re-plans immediately before applying, since time has passed since the last plan and drift could have crept in. - Remote state and locking: state lives in a backend with native locking (S3 with a DynamoDB lock table, Terraform Cloud, or a Postgres backend with locking) so two runs can never apply concurrently against the same workspace.
flowchart LR
A[PR opened] --> B[CI: fmt + validate + plan]
B --> C[Post plan as PR comment]
C --> D{Merge target}
D -->|non-prod branch| E[Auto-apply on merge]
D -->|main / prod| F[Required reviewer approval]
F --> G[Protected environment gate]
G --> H[Manual apply trigger]
E --> I[(Remote state + lock)]
H --> I
I --> J[OIDC short-lived cloud role]
Where credentials live
- The pipeline should never hold long-lived cloud access keys. Use OIDC federation: the CI system exchanges a short-lived OIDC token for a cloud IAM role scoped to exactly one environment, for example a
terraform-dev-deployrole for non-prod and a separate, more restrictedterraform-prod-deployrole for prod. - The role used to plan can be narrower (read-only, broad enough to describe resources) than the role used to apply; some teams even use different identities for plan (safe to run on any PR) versus apply (only runs on trusted, merged code).
- Secrets the Terraform code itself needs, like database passwords or provider API keys, come from a secrets manager (Vault, or a cloud-native secrets store) referenced at runtime, never hardcoded in
.tfvarscommitted to the repo.
Worked example
A concrete two-branch layout: the dev branch maps to workspace dev, and main maps to workspace prod.
- An engineer opens a PR from
feature/add-cacheintodev. CI plans using the read-scopedterraform-dev-deployrole and posts the diff as a PR comment. - The PR is approved and merged into
dev. A merge-triggered workflow re-plans and applies automatically using the apply-scopedterraform-dev-deployrole, no human in the loop. - Once validated in
dev, a second PR mergesdevintomain. CI plans against theprodworkspace and posts that diff too. - Because the target is
main, the apply job sits behind a protected "production" environment requiring one designated approver. The approver reviews the plan output itself, not just the code diff, approves, and only then does the apply job run usingterraform-prod-deploy.
Trade-offs & pitfalls
Re-plan immediately before apply instead of reusing the PR-time plan: a long-lived PR or a busy shared environment means the world can move between plan and apply, and applying a stale plan is a common source of unpleasant surprises. Terraform Cloud or Atlantis reduce how much pipeline glue you write yourself (state, locking, the plan-comment UI, approvals) at the cost of another system to operate; rolling it in plain CI with an S3 backend gives full control but you own more of the plumbing. OIDC federation removes the need to rotate static secrets but requires upfront IAM trust-policy setup per environment; skipping it for "just get it working" static keys is the most common regression back to a real security gap. Auto-apply on non-prod is only safe if "non-prod" genuinely means low blast radius; a shared staging environment other teams demo against deserves the same gate as prod.
A CloudFormation stack update partially succeeds, some resources get replaced, others just updated, and now the stack is in an inconsistent state that includes a stateful resource like a database. How do you reconcile that stack and get back to a known-good state without taking the database down?
Sample Answer
Direct answer
Stop making further changes, protect the stateful resource immediately with a fresh snapshot (or a replica), and then use a CloudFormation change set, never a blind update, to see exactly what any reconciling update would do before it happens. Keep the database itself out of that reconciliation entirely by setting DeletionPolicy/UpdateReplacePolicy to Retain and, if the template and the live resource have drifted apart, adopting the existing database into the stack via resource import rather than letting CloudFormation replace it. Only cut traffic over to anything new once it is validated, so the database is never touched by an apply that could replace it.
Structured elaboration
Assess before touching anything further
- Check stack events (
aws cloudformation describe-stack-events) to see exactly which resources were replaced, which were updated in place, and which are in a failed or rollback state. - Run drift detection (
aws cloudformation detect-stack-driftthendescribe-stack-resource-drifts) to see how far the live resources have diverged from the template's understanding of them.
Protect the stateful resource first
- Take an immediate manual snapshot:
aws rds create-db-snapshot --db-instance-identifier mydb --db-snapshot-identifier pre-reconcile-YYYYMMDD. - If the engine supports it, stand up a read replica as a warm standby before making any further changes to the stack.
- Confirm automated backups and the earliest available restore point, in case a snapshot restore ends up being the fallback.
Prepare a non-destructive plan via a change set
- Generate a change set against the current template rather than applying directly:
aws cloudformation create-change-set --stack-name my-stack --template-body file://new.yml --change-set-name reconcile-cs --capabilities CAPABILITY_NAMED_IAM. - Inspect it specifically for any
Replaceaction on the database. If the change set would replace it, stop and rework the template rather than proceeding, this is the single most important check in the whole process.
Reconciling the stack's model of reality with what is actually running
| Approach | When it fits | What it costs you |
|---|---|---|
DeletionPolicy/UpdateReplacePolicy: Retain + adjust template | The database itself is healthy but the stack wants to replace it due to a property change | Requires getting the template's declared properties to match the live resource closely enough that CloudFormation stops proposing a replace |
| Resource import | Physical resource exists and is healthy, but the stack has lost track of owning it, or a related resource was replaced around it | Import is strict about matching properties exactly; a mismatch means CloudFormation still thinks something needs to change |
| Snapshot restore into a new instance, then adopt | The existing instance itself is compromised or a required change (e.g. major version upgrade) forces replacement anyway | Real downtime/cutover risk unless paired with a replica or blue-green cutover of the stateless tier pointing at the new instance |
Cutover for anything that does need to move
Where the database tier genuinely has to change (a forced replacement, a major version upgrade), prefer promoting a replica or restoring a snapshot into a new instance, validating it (schema checks, smoke tests) in parallel, and only then repointing the application via its endpoint during a short, controlled cutover, rather than letting a stack update do it as a side effect of resolving the CloudFormation inconsistency.
Worked example
Given a stack where an EC2 Auto Scaling Group was replaced successfully but the RDS instance update rolled back partway, leaving the stack in UPDATE_ROLLBACK_FAILED: first, snapshot the RDS instance immediately. Second, run drift detection to confirm the RDS instance's live properties versus what the template currently declares. Third, generate a change set against a corrected template that sets DeletionPolicy: Retain on the RDS resource and matches its declared properties to the live instance's actual configuration, confirm the change set shows zero actions against the RDS resource. Fourth, apply that change set, since it only touches the already-replaced ASG's dependent resources (for example, security group references) and leaves the database untouched. Fifth, once the stack is back to a clean state, address the original intended change (whatever caused the RDS property to want replacing) as its own follow-up change set, reviewed specifically for that risk.
Trade-offs & pitfalls
DeletionPolicy: Retainprotects against accidental replacement, but if a team forgets to set it before a risky update, that protection simply is not there when it is needed; it has to be a template default for stateful resources, not something added reactively.- Change sets add real process overhead compared to a direct update, but that overhead is the entire point for a stateful resource; skipping it to move faster is how partial-failure states like this one happen in the first place.
- Resource import is strict: even a small mismatch between the template's declared properties and the live resource's actual configuration means CloudFormation still believes a change is needed, which can silently reintroduce the same class of risk this whole process is meant to avoid.
Explain GitOps as a workflow for infrastructure changes: what does the branch, PR, review, merge, automated-apply flow actually look like, and what does having a reconciler constantly watching git buy you over people applying changes by hand?
Sample Answer
Direct answer
GitOps treats the Git repository as the single source of truth for infrastructure: an engineer opens a PR against the IaC (Terraform, CloudFormation, or similar), CI renders a plan and posts it on the PR for review, a human approves and merges, and only then does something apply the change, either a CI job triggered by the merge, or a reconciler that is continuously watching the repo. What a continuously watching reconciler buys you over ad hoc manual applies is that it also converges any out-of-band drift back to what is in git, not just deploy new commits, so git stays true to what is actually running, not just to what someone last remembered to push.
Structured elaboration
The flow
- Developer creates a branch with the IaC change and opens a PR.
- CI runs a plan (for example
terraform plan) and posts the diff on the PR, along with lint, static checks, and any policy gates. - The team reviews the plan output, not just the code diff, since that is what will actually happen to live infrastructure.
- On approval, the PR merges to the main branch.
- Apply happens one of two ways: a CI job triggered directly by the merge event, or a reconciler that is separately, continuously polling or watching the repo (via webhook) and converging live state toward whatever the latest merged commit declares.
Push-triggered apply versus a continuously watching reconciler
- Push-triggered: an apply only runs in response to a merge event. Nothing happens between merges, so if the live environment drifts out of band (someone changes something through the console), nothing corrects it until the next merge happens to touch that resource.
- Continuously watching reconciler: it does not just react to new commits, it repeatedly compares live state to the declared state and re-converges even when nothing new has merged, which is what catches and fixes drift, not only new intent.
What this buys you over manual applies
- Git becomes the single, versioned record of every change, who proposed it, who approved it, and what actually got applied, which is a real audit trail instead of tribal memory.
- Reviewed plans catch mistakes before they touch production, instead of relying on whoever is running
applyfrom their laptop to notice something is off. - No one needs standing, long-lived cloud credentials on a local machine; only the pipeline or reconciler's service identity needs apply-level access.
Worked example
Say the change is adding a lifecycle rule to an existing S3 bucket to expire objects after 90 days. The engineer edits the bucket module, opens a PR, and CI's plan output shows exactly one resource changing in place, the lifecycle configuration, zero creates, zero destroys. A teammate reviews that plan (not just the HCL diff) and approves. On merge, either the CI apply job runs terraform apply against that one change, or, if the team uses a reconciler pattern instead, the reconciler notices the new commit on its next poll or webhook trigger and converges the bucket to match. If someone had manually added a different lifecycle rule through the console the week before, a reconciler's next pass would have already flagged or reverted that drift independently of this PR; a push-only pipeline would not have caught it until some unrelated change happened to touch that same bucket.
Trade-offs & pitfalls
- Secrets management: nothing sensitive should live in the git-tracked IaC itself. The reconciler still needs a way to pull secrets from a vault or secrets manager at apply time, referencing an ARN or path rather than a value, and the state file itself needs to be treated as sensitive since resolved values can still land there.
- Drift windows between commits: there is always a gap between when something changes out of band and when the reconciler's next poll or webhook fires and corrects it. The length of that window is a direct function of your reconciliation interval, and during it the live state and the declared state genuinely disagree, which matters for anyone debugging an incident in that window.
- Reconciler loops: a reconciler can get stuck reapplying the same "diff" every cycle when a resource attribute is not stably convergent, for example a provider that normalizes a value differently than it stores it, so every pass reports a change and reapplies it. This looks like the reconciler doing its job but wastes API calls and can trip provider rate limits; the fix belongs in the module or resource definition, not in disabling reconciliation.
Design a GitOps setup for infrastructure changes that has to work for 100+ teams. What would repo layout, merge controls, and environment promotion look like, and how do you recover if the control plane itself goes down?
Sample Answer
Direct answer
This design assumes the target platform is Kubernetes, since Argo CD and Flux are Kubernetes-native GitOps operators. Use a hybrid repo layout: a small shared platform repo for cluster bootstrap and org-wide policy, plus one repo per team for their application infra, tied together by a GitOps operator (Argo CD or Flux) that enforces merge controls and promotion through PRs, not manual kubectl or console access. The control plane's own disaster recovery works because git, not the operator's internal state, is the source of truth: losing the operator means redeploying it and letting it re-reconcile from the same repos, not reconstructing anything by hand.
Architecture
flowchart TD
A[Team Repo: app infra] -->|PR plus CI checks| B[GitOps Operator: Argo CD or Flux]
C[Platform Repo: bootstrap and policy] --> B
D[Shared Module Registry] -->|version bump PR| A
B --> E[Dev Cluster]
E -->|promote PR| F[Staging Cluster]
F -->|promote PR| G[Prod Cluster]
B --> H[Audit Log and Git History]
I[Git Mirror: DR] -.-> B
Repository layout
- Shared platform repo: cluster bootstrap manifests, platform operators, common libraries, and org-wide policy. Changes here are rare and reviewed by the platform team.
- Per-team repos: application infra and each team's own dev/staging/prod definitions. This keeps each team's blast radius small and lets 100+ teams move independently instead of contending for one repo's review queue.
- Shared module registry: versioned Terraform/Helm modules a team's repo depends on by version pin, not by copy-paste.
Shared module publishing flow
When the platform team publishes a new version of a shared module, that is itself a GitOps event, not a side channel: the module repo tags a release, and a bot opens a version-bump PR against every consuming team's repo. That PR runs the same automated checks and approvals the team's own changes would (plan, policy-as-code, required review) before the version bump can merge and apply, so a shared module update reaches production the same audited way a team's own change would, and no team is silently running a stale or unvetted version.
Merge controls
- Branch protection: required PR review, required CI checks (plan, policy-as-code, lint), signed commits.
- CODEOWNERS mapped per repo so the right reviewers are auto-requested without a manual routing step.
- The GitOps operator's own RBAC, driven by SSO group mapping, decides who can approve a promotion into staging or prod; only the operator's service account actually applies changes to a cluster, no human ever runs
kubectl applydirectly against a managed cluster.
Environment promotion
Promotion is itself a PR: dev to staging to prod is a change to which git ref (or image tag) an environment's manifest points to, opened automatically once the lower environment's tests pass, and gated by human approval and stricter policy checks the higher the environment. This makes promotion auditable by construction, since "what's in prod" is always just "whatever the prod branch/ref currently says," recoverable from git history.
Recovering the control plane
- Git is the durable source of truth; mirror the platform and module repos across providers or regions so a single git host outage does not strand the fleet.
- The GitOps operator itself runs highly available across zones, but if it is lost entirely, redeploying it from the platform repo's own bootstrap manifests and pointing it at the existing repos re-establishes state through normal reconciliation. no manual replay of "what changes happened" is needed, because git already has that history.
- Secrets do not live in git: an external secret store (Vault, or a KMS-backed sealed-secrets approach) is restored from its own backup, and the operator picks the references back up on reconciliation.
- Test this recovery path deliberately (a scheduled game day that rebuilds the control plane from scratch), not just assume it works because the design implies it should.
Trade-offs & pitfalls
- A hybrid layout balances team autonomy against discoverability; a full mono-repo simplifies org-wide changes but becomes a review bottleneck at 100+ teams, while a fully per-team layout maximizes autonomy but makes org-wide policy changes require touching every repo.
- Strict pre-merge checks (required reviews, full policy-as-code) protect prod but add real friction to every change; the mitigation is investing in good starter templates and fast CI, not loosening the checks.
- Treating the GitOps operator's own state as anything other than disposable is the most common design mistake here: if recovering it requires more than "redeploy and let it reconcile from git," the design has quietly made the operator a second source of truth without meaning to.
Design the machine-image pipeline for a fleet of stateless instances behind a load balancer: how images get built and tested, how you promote an image across environments, and how you actually swap the fleet over to a new image with health checks and connection draining so nothing gets dropped. How would this change if you also needed to fast-track an urgent security patch?
Sample Answer
Direct answer
Baking an image means pre-installing everything a server needs (OS packages, hardening, the app itself) into a reusable image with a tool like Packer, instead of configuring the server after it boots. Build the pipeline around one principle: nothing reaches production as an image that has not been baked, tested, and scanned the same way every time, and the fleet gets updated by replacing instances behind health checks and connection draining rather than patching them in place. The design has two paths through the same pipeline: the normal path (bake, test, promote through environments, canary, full rollout) and a fast path for urgent security patches that skips environment promotion but never skips the tests or the scan.
Structured elaboration
Image build and test
- CI triggers a Packer build on a base-image or application change: provision the base OS, apply hardening (CIS-style benchmarks, i.e. standardized security-configuration checklists), install the app artifact, and pull secrets via short-lived tokens rather than baking them in.
- Baked-in automated tests run as part of the same pipeline, not as a separate manual step: unit and config-validation tests during the bake, then a post-bake stage that launches the image in an isolated environment and runs integration and smoke tests against it.
- A vulnerability scan (for example Trivy or Grype against the baked image) runs in that same post-bake stage. This is a hard gate, not advisory: an image with a scan finding above the agreed severity threshold does not get published.
- On pass, the image is registered in the artifact registry tagged with its git SHA, build ID, SBOM, and the CVE baseline it passed against, so any later question of "what is actually running" and "was it scanned against what we knew at the time" has an answer.
Promotion across environments
Promotion is a pipeline gate, not a person clicking approve in a console: dev, then staging with regression tests, then a canary slice of production, each gated on the previous stage's tests and monitoring staying green.
Swapping the fleet over
- The fleet sits behind an ASG (or equivalent instance group) and a load balancer. The launch template points at the new image; an ASG instance refresh (or an equivalent rolling-replace controller) walks the fleet in batches.
- Per instance: deregister from the target group first, which starts connection draining; wait for in-flight requests to finish or the drain timeout to hit; only then terminate it. The replacement instance must pass its health check before the load balancer sends it any traffic.
- A minimum-healthy-percentage setting (for example 90%) caps how much capacity can be replacing at once, so a bad new image degrades a fraction of the fleet rather than all of it while it is still being watched.
- For workloads that carry state (a service with long-lived connections, or one with session affinity), connection draining alone is not enough: the drain window also has to respect existing session affinity, and if any part of the workload is stateful in the sense of holding data (not just connections), that has to coordinate with the data layer's own replication or failover process rather than treating the instance as freely swappable the moment its health check fails.
Fast-tracking an urgent security patch
The fast path changes how far the image travels before real traffic sees it, not whether it is tested:
- Skip the full dev-then-staging promotion chain; go straight from bake to a canary slice of production.
- Keep the bake-time tests and the vulnerability scan as hard gates; an urgent patch that has not been scanned is exactly the failure mode a patch process exists to prevent.
- Shorten, but do not remove, the canary observation window, and have the rollback path pre-verified rather than improvised, since this path is exercised under time pressure.
- Immediately backfill afterward: once the emergency patch has gone through the fast path, run it (or its base) through the normal dev and staging pipeline the following day, so the fast-tracked version does not become a permanent exception living outside the standard promotion history.
Worked example
flowchart TD
A[Source or base image change] --> B[CI triggers Packer bake]
B --> C[Bake time tests: hardening, vuln scan, smoke tests]
C --> D[Publish image with SBOM and CVE tags to registry]
D --> E[Promote through dev then staging]
E --> F[Canary: weighted traffic on new AMI]
F --> G{Health checks and SLOs pass?}
G -- Yes --> H[Full fleet rollout via ASG instance refresh]
G -- No --> I[Roll back to prior AMI, tag new image as bad]
B -.urgent security patch.-> J[Fast path: skip dev and staging, bake plus scan only]
J --> F
Concretely: a CVE lands in the base OS image. CI triggers a Packer bake immediately (the dashed path above). The bake produces a new image; the same automated tests and the same vulnerability scan run against it as any normal build, just without waiting for a scheduled promotion window. It goes straight to a canary slice of the fleet, monitored against the same health checks and error-rate thresholds as any other rollout, then to the full fleet via instance refresh. The following day, the same image is run through the normal dev and staging environments to confirm nothing outside the emergency scope regressed.
Trade-offs & pitfalls
- Baking images takes longer than patching in place, and that trade-off is deliberate: reproducibility and a clean rollback (revert the launch template to the previous image ID) are worth the extra build minutes.
- The most dangerous version of a "fast path" is one that quietly also skips testing or scanning under time pressure; the fast path should only ever shorten promotion, never verification.
- For stateful workloads, connection draining and health checks are necessary but not sufficient; assuming they are enough to make image replacement safe for anything holding data is a common design mistake.
- Rolling back an in-flight instance refresh needs to be a rehearsed, one-command action (point the launch template back at the previous image ID), not something improvised the first time it is needed.
Unlock Full Question Bank
Get access to all Infrastructure as Code and Automation interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.