Infrastructure as Code and Automation Questions
Defining, provisioning, and automating infrastructure programmatically. Covers declarative IaC with Terraform and comparable tools like CloudFormation (resource and provider model, state management and remote backends, module design and reuse, workspaces, drift detection, and safe plan/apply workflows), plus the broader automation discipline: provisioning pipelines, golden-image and machine-image building, scripting glue, self-service platforms, and end-to-end environment stand-up. The authoring, lifecycle, and automation of infrastructure code that reduces manual toil across provisioning workflows.
A teammate hands you a Terraform file that provisions a new environment from scratch. Before it can be applied to a shared cloud account, what would you review in the provider configuration, resource definitions, variables, and outputs to make sure the configuration is readable, parameterized, and not hiding risky assumptions?
Sample Answer
I would review it from top to bottom with a bias toward clarity and safety.
Provider config
- Pin the provider version so behavior is reproducible
- Confirm the region and account are explicit, not inherited by accident
- Check whether aliases are used correctly for multi-account access
Resources
- Look for hard-coded IDs, names, or CIDRs that should be variables
- Check that dangerous resources have sane lifecycle settings
- Make sure dependencies are clear and not hidden in implicit assumptions
Variables
- Give each variable a type and description
- Add validation for things like CIDR shape, allowed instance sizes, or non-empty lists
- Mark secrets as sensitive, but remember that only hides output, not state
Outputs
- Expose only what other modules need, such as IDs, endpoints, or ARNs
- Do not output passwords or full secret payloads
Example
If I see a hard-coded 10.0.0.0/16 and a shared-account provider with no explicit role assumption, I would flag both. That makes the config less portable and hides an assumption that could break a later environment.
One of your engineers discovers that a database password ended up committed in a Terraform state file that was pushed to a shared repository. Walk me through what you'd do right away, and what you'd change afterward so it can't happen again.
Sample Answer
Direct answer
Rotate and revoke the credential immediately, everywhere it's used, before doing anything else: that's the only step that actually stops the bleeding, and forensics, cleanup, and policy changes all happen after containment, not instead of it. Separately, because this keeps happening across the industry, fix the two structural causes: encrypt the state backend so a leaked state file isn't plaintext-readable, and prefer ephemeral, short-lived credentials so secrets stop landing in state to begin with, rather than relying on catching every leak after the fact.
Structured elaboration
Immediate containment, before root-causing anything
- Revoke or rotate the exposed credential at the source (disable the IAM user/key, rotate the DB password) before investigating how it leaked. Every minute it stays valid is exposure.
- Remove the credential's blast radius: rotating a DB password and updating the app's connection string via the secret manager, not a manual edit, locks out anyone who copied it.
- Pull repository access temporarily if the state file is still sitting in a shared, readable location, so no one can pull a fresh copy while cleanup is in progress.
Investigate blast radius
Was the credential actually used by someone who shouldn't have had it? Check the target system's own access logs (DB connection logs, cloud provider audit logs) for activity from unfamiliar IPs or unusual times in the window since the state file was pushed. This determines whether it's "credential exposed, no evidence of misuse" or "confirmed unauthorized access," which changes the notification obligations.
Two different remediation paths depending on how long it's been exposed
A state file pushed an hour ago and one that's been sitting in git history for months are different problems.
- Fresh push, hours old: rotate, remove the offending commit before it's widely pulled, and a normal coordinated
git push --forceplus history cleanup is enough, because the exposure window is small and you likely know who has pulled it. - Buried in history for months: assume it's been exposed the entire time, not just since discovery, because you can't retroactively know who cloned, forked, or cached the repo, including CI runner caches and backup snapshots, during that window. This needs a full history rewrite (
git filter-repoto strip the file/commit from every ref), coordinated re-clone by every developer and CI system, since a plain force-push doesn't fix clones that already exist, and treating the credential as compromised regardless of what the access logs show, since log retention may not even cover the full exposure window.
Structural fix 1: encrypt the state backend
Even after rotation, the old, now-invalid but still informative, value sitting in a plaintext state file is a problem for anyone auditing what leaked. Use a state backend with encryption at rest by default (S3 with SSE-KMS and a bucket policy denying unencrypted puts, Terraform Cloud/Enterprise's built-in encryption, or a Vault-backed backend) so a copy of the state file alone isn't enough to read secrets even if it leaks again.
Structural fix 2: prefer ephemeral credentials so nothing long-lived is ever in state
The deeper issue is that a static, long-lived credential value, a real DB password, not a reference to one, was ever an attribute in the resource's config to begin with. The fix is architectural, not just procedural: use dynamic, short-TTL credentials issued at apply time (Vault, or a cloud-native equivalent) instead of static passwords, so the value that ever touches Terraform is valid for minutes, not indefinitely. Even that has a gap worth knowing: a value fetched via a data source still gets written into the state file's plaintext by default, because Terraform's state format persists every attribute of every resource and data source it manages, dynamic or not. This is exactly the gap Terraform 1.10+ addressed with ephemeral resources and write-only arguments: values marked ephemeral are used during the apply but are deliberately never written to state or plan files at all, a stronger guarantee than "the credential expires quickly" on its own.
Structural fix 3: a pre-commit/CI scanning gate
Rotation and history rewriting fix one incident; a scanning gate stops the next one from merging in the first place. Run a secret-scanning tool (gitleaks, truffleHog, or similar) both as a pre-commit hook, for fast author feedback, and as a required CI check on every PR, the actual enforcement point since pre-commit hooks can be skipped locally. Scan the Terraform plan/state output specifically, not just source files, since state and plan JSON are exactly where a secret shows up even when the .tf source never had it hardcoded.
Notification and postmortem
If the credential's compromise could have exposed customer data, loop in compliance/legal early, since this determines regulatory notification timelines, often measured in days. Regardless of scale, write a postmortem with a timeline, what was exposed, what was rotated, and concrete owners/dates for the structural fixes above, not just "we rotated it and moved on."
Worked example
Concrete before/after for structural fix 2, showing the actual problem, a static password as a plain resource argument, versus the pattern that avoids it:
# Before: static secret is a plain attribute, gets written to state in plaintext
resource "aws_db_instance" "app" {
identifier = "app-prod"
password = var.db_password # a real, long-lived value, now permanently in state history
}
# After: value is marked ephemeral (Terraform 1.10+), used during apply,
# never persisted to state or plan output
ephemeral "random_password" "db" {
length = 24
}
resource "aws_db_instance" "app" {
identifier = "app-prod"
password_wo = ephemeral.random_password.db.result
password_wo_version = 1
}
The practical difference: if this state file leaks again after the fix, there's no password in it to rotate in a panic, because the value was never written there.
Trade-offs & pitfalls
- Rotating before investigating feels backwards to people who want to preserve evidence first, but a live credential someone else might be actively using is a bigger risk than losing a few minutes of forensic cleanliness; contain first.
- Treating "we rotated it" as the end of the incident misses that the old value is still sitting in every git clone and CI cache that pulled it before rotation; a fresh push and a months-old buried secret need genuinely different remediation, not the same checklist applied at different speeds.
- Ephemeral values and write-only arguments require the specific resource/provider to support them; not every provider has adopted the pattern yet, so this is a direction to move toward for new resources, not a guarantee that already covers the whole estate.
- A pre-commit hook alone isn't enforcement, since
--no-verifyskips it; the CI check on the PR is what actually blocks a merge, and that's the one to treat as required rather than advisory.
A Terraform run needs to hand a newly provisioned service a database credential or similar application secret. How would you get that credential to the resource without it ever being a long-lived static value baked into your configuration or left sitting in state, and what has to happen when that credential eventually needs to be renewed?
Sample Answer
Direct answer
Don't have Terraform mint or hold the actual database credential at all. Have it provision the plumbing of a secrets engine (Vault, HashiCorp's tool for storing and minting credentials on demand instead of them being fixed values anywhere, has a database secrets engine for this; a cloud-native equivalent like AWS Secrets Manager with rotation configured works too) that mints a short-lived, per-consumer credential on demand, and have both Terraform's own auth to that engine and the application's auth use identity-based, short-lived methods (AppRole, a role-ID-plus-secret-ID login pattern explained below; Kubernetes service-account auth; cloud IAM/OIDC) instead of a static token. Renewal then becomes the secrets engine's job (lease renewal or a scheduled rotation function), not a manual re-apply of a static value.
Structured elaboration
Why baking the secret in is unsafe
A credential written into a resource argument or interpolated into a data source ends up in the Terraform state file (state is plaintext JSON unless you add extra tooling around it) and can appear in plan/apply CLI output and CI logs. It also never expires on its own: a static value is a permanent liability until someone remembers to rotate it by hand.
The pattern: Terraform provisions the engine, not the secret
Terraform's job is to create the database secrets engine mount, the connection, and a role that defines what SQL a dynamic credential is allowed to run and for how long. The application (or the config-management tool bootstrapping it) asks the engine for a credential at the moment it needs one, not before.
Authenticating the secrets engine itself
Two named patterns cover most cases: AppRole-style auth, where a role_id (non-secret, can be baked into config) is paired with a secret_id (short-lived, delivered out of band) and exchanged for a token; and cloud-IAM-based auth, where the caller's existing cloud identity (an EC2 instance profile, a GCP service account, an Azure managed identity) is validated directly by the secrets engine without any bootstrap secret at all, for example Vault's aws auth method validates a signed STS GetCallerIdentity request from the instance's own credentials.
Bootstrapping brand-new hosts and CI runners
A host or CI runner that has never run before has no identity to present yet. Three concrete answers: deliver a short-lived, single-use wrapped AppRole secret_id via cloud-init/user-data at boot; use Kubernetes' projected service-account token with Vault's kubernetes auth method, so a pod authenticates using the token Kubernetes already gives it, no separate secret to distribute; or use cloud-IAM-based auth directly (an EC2 instance profile, or GitHub Actions OIDC federated into a cloud IAM role via AssumeRoleWithWebIdentity) so the runner's platform-issued identity is the credential.
provider "vault" {
address = var.vault_addr
auth_login {
path = "auth/approle/login"
parameters = {
role_id = var.approle_role_id
secret_id = var.approle_secret_id
}
}
}
# Kubernetes-native bootstrap: the pod's projected service-account token
# is the credential, nothing extra to distribute.
provider "vault" {
auth_login {
path = "auth/kubernetes/login"
parameters = {
role = "ci-runner"
jwt = file("/var/run/secrets/kubernetes.io/serviceaccount/token")
}
}
}
resource "vault_database_secret_backend_role" "app_role" {
backend = "database"
name = "my-app-role"
db_name = "my-db"
creation_statements = ["CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT ON app_table TO \"{{name}}\";"]
default_ttl = "1h"
max_ttl = "24h"
}
Extending the pattern beyond Terraform
The same idea applies to whatever converges the host after Terraform hands off: Ansible, Chef, or Puppet runs should also fetch secrets at converge time from the same engine using the same identity-based auth, rather than templating a static credential into a playbook variable or a Chef data bag. Terraform's role in that flow stops at "the engine and its access policy exist," it does not hand the config-management tool a credential to pass along.
Leasing and rotation, operationally
Every dynamic credential carries a default_ttl and max_ttl. Short-lived processes just request a fresh credential each run. Long-running processes (an app server holding a DB connection pool, a persistent CI agent) need to renew the lease before it expires, typically via Vault Agent running in auto-auth plus auto-renew mode, or the cloud secrets manager's built-in rotation Lambda on a schedule shorter than the credential's expiry.
The dual-side rotation problem
This is the case that trips people up: a database credential that is used both by a Terraform-managed resource (say, an aws_db_instance master password, or a Vault-issued credential a live application pool is actively holding open connections with) has to be rotated without dropping those connections. You cannot just swap the value in one step. Sequence it:
- Create the new credential alongside the old one, so both are valid simultaneously (a second DB user, or a fresh lease from the secrets engine).
- Push the new credential to the application's config/secret store and let the app pick it up via a rolling restart or a hot-reload of its connection pool, while the old credential still works for any connection that hasn't cycled yet.
- Confirm nothing is still authenticating with the old credential (check the database's connection/audit log for the old username, or the secrets engine's lease usage).
- Only then revoke or rotate out the old credential.
The fallback if it fails partway: if step 2 fails (the app can't reach the new secret, or the rolling restart stalls), do not revoke the old credential. Roll the app config back to the old credential and retry, keeping both valid until a rotation attempt actually completes end to end. Revoking the old credential before confirming cutover is what causes an outage, not the rotation itself.
Keeping Terraform out of the runtime fetch path
Avoid reading dynamic secrets with a Terraform data source at plan or apply time if you can help it; if you truly must, mark the value sensitive = true on any output or local (this only redacts CLI output, it does not encrypt state, so pair it with a remote encrypted backend and restricted ACLs). For long-running CI pipelines, re-authenticate before each terraform invocation with a fresh ephemeral token rather than caching one that could go stale or leak.
Trade-offs & pitfalls
Running a dynamic-secrets engine is real operational overhead (an HA service you now own or pay for) compared to a static secret in a vault-of-passwords tool; that overhead buys you automatic expiry and no long-lived value to leak. sensitive = true is frequently mistaken for encryption, it only hides a value from CLI output, the value is still sitting in state in the clear unless the backend itself encrypts it. The dual-side rotation sequence only works if the application can actually pick up a new credential without a full restart (hot-reload a connection pool); if it can't, budget for a rolling deploy as part of every rotation, not just a config push. Setting default_ttl too aggressively short for a high-traffic service creates a thundering-herd of renewal requests against the secrets engine.
Your CI pipeline needs to authenticate to your cloud provider to run Terraform, but you don't want a long-lived access key sitting in a repo secret or environment variable. How would you design that authentication so the pipeline gets valid credentials only when it actually needs them, and how would you make sure a pull request from an untrusted fork can't get at anything sensitive?
Sample Answer
Direct answer
Give the pipeline no long-lived key at all: configure your cloud provider's identity federation (AWS IAM OIDC provider, GCP Workload Identity Federation, Azure federated credentials) to trust GitHub's OIDC token issuer (OIDC: a standard that lets GitHub hand over a signed, short-lived proof-of-identity token instead of a stored secret), scope the trust policy to a specific repo/branch/environment claim, and have the pipeline exchange that short-lived, per-run token for temporary cloud credentials only at the moment it needs them; those credentials expire on their own, so there's nothing long-lived sitting in a repo secret to leak. Untrusted fork PRs are handled by GitHub itself before your trust policy even matters: a workflow triggered by the pull_request event on a fork does not receive repository secrets and does not get the id-token: write permission needed to mint an OIDC token, unless you deliberately opt in with pull_request_target, which should be avoided for anything that checks out and runs fork-provided code.
Design
IAM trust policy (OIDC federation)
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::123456789012:oidc-provider/token.actions.githubusercontent.com"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"token.actions.githubusercontent.com:aud": "sts.amazonaws.com"
},
"StringLike": {
"token.actions.githubusercontent.com:sub": "repo:your-org/your-repo:ref:refs/heads/main"
}
}
}
]
}
The sub condition pins which repo and branch (or environment) can assume this role, this is the actual boundary that matters, a role trusting repo:your-org/* for any branch is far weaker than one pinned to main or a protected environment.
Workflow configuration
name: terraform
on:
push:
branches: [main]
permissions:
id-token: write
contents: read
jobs:
apply:
runs-on: ubuntu-latest
environment: production
steps:
- uses: actions/checkout@v4
- uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: arn:aws:iam::123456789012:role/terraform-ci
aws-region: us-east-1
- run: terraform init
- run: terraform apply -auto-approve tfplan
The official aws-actions/configure-aws-credentials action handles requesting the OIDC token and exchanging it for temporary AWS credentials; nothing gets echoed to a secret variable. Triggering only on push to main also means a fork PR can never reach this job in the first place, a separate, unprivileged plan-only workflow can run on pull_request for visibility, but should not carry id-token: write or any cloud credentials at all.
Fork PR isolation
A workflow triggered by pull_request from a fork runs with a read-only GITHUB_TOKEN and does not receive repository secrets or id-token: write, GitHub enforces this at the platform level regardless of what the fork's copy of the workflow file requests. The one way to defeat that protection is switching the trigger to pull_request_target, which runs with the base repository's permissions and secrets against the PR's code, without adding a required manual approval gate (a GitHub Environment with protection rules) that is a well-documented misconfiguration and exactly the setup to avoid here.
Least privilege and log hygiene
- Scope the IAM role's attached policy to only the actions and ARNs that environment's apply actually needs, and use a separate role per environment so a dev-scoped OIDC subject can never assume the prod role.
- Never enable verbose/debug SDK or Terraform logging (
TF_LOG=trace,set -x) in a step that touches plan or apply output, and if any custom script parsesterraform show -jsonto build a summary, apply the same sensitive-field redaction (after_sensitive/before_sensitive) needed anywhere else that output is consumed.
Auditing and rotating the CI identity itself
There's no static credential to rotate with OIDC, the whole point is that nothing long-lived exists, but the trust relationship itself is the credential now and needs periodic review: who can push to the branches or trigger the environments named in the sub condition, does the role's attached policy still match least privilege, and are AssumeRoleWithWebIdentity events in CloudTrail reviewed for anomalies (an unexpected source repo, an unusual time of day).
Worked example
An attacker forks the repo and opens a PR with a modified .tf file that would, if it had credentials, try to exfiltrate values. Because that PR triggers the pull_request event, not push, GitHub does not expose repository secrets to that run and does not grant id-token: write even though the forked copy of the workflow file might ask for it, the platform enforces this regardless of the fork's own YAML. The run gets a short-lived, read-only GITHUB_TOKEN and nothing else; there is no path in this setup for a fork PR to reach the terraform apply job at all, since that job only runs on push to main.
Trade-offs and pitfalls
- OIDC removes the leaked-long-lived-key failure mode but shifts the risk to the trust policy configuration; an overly broad
subcondition effectively grants the role to any branch matching the pattern, including a feature branch anyone can push to. - There's nothing to rotate on a schedule the way you would a static key, so the audit discipline has to shift to reviewing who can satisfy the trust condition and what the role's policy actually grants, that review is easy to let lapse precisely because there's no expiring secret forcing anyone to revisit it.
- Logs still need discipline: OIDC solves the credential-storage problem, not the "did we accidentally print a secret to a log or a PR comment" problem, that's a separate, ongoing concern.
Terraform shows a live resource as missing, but the cloud console says it still exists. You suspect the object was originally created outside Terraform and later partially brought under management. How would you decide whether to import it, recreate it, or leave it alone?
Sample Answer
Decision framework
I first determine whether Terraform should own the object. A resource should be imported only if the team wants Terraform to manage its lifecycle and we can make the current cloud object match code with low risk. If ownership is unclear or another team runs it, I leave it alone and document the boundary.
How I decide
- Import when the resource already exists, is needed long term, and its current settings are close enough to model in code.
- Recreate when the live object is too drifted, unsafe, or impossible to express cleanly, and downtime or cutover is acceptable.
- Leave alone when it is intentionally outside this stack, or the business risk of taking ownership is higher than the benefit.
Concrete example
Say an existing prod-logs bucket was created manually. If the app depends on it and the bucket policy, versioning, and encryption are acceptable, I would import it, then update the module so the code matches the real settings. If the bucket has years of data and a clean import is possible, import is safest. If the bucket name, policy, or region is wrong and fixing it in place would be risky, I would create a new bucket, migrate traffic or data, then delete the old one after validation.
What I check first
I compare state, config, and live attributes, then run a plan to see whether the desired change is truly just drift or a bigger ownership problem.
Unlock Full Question Bank
Get access to all 18 Infrastructure as Code and Automation interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.