Infrastructure Automation and Provisioning Questions

Covers designing, implementing, and operating automated infrastructure provisioning and configuration using Infrastructure as Code practices and complementary automation patterns. Candidates should be able to select and author declarative infrastructure definitions with tools such as Terraform, CloudFormation, and Azure Resource Manager templates, and discuss configuration management tools such as Ansible, Puppet, or Chef. Core skills include modular and reusable code organization for multiple environments, variable and output management, remote state management and locking, idempotency and atomicity of operations, and version control integration for infrastructure artifacts. Candidates should understand testing and validation practices including linting, plan or dry run validation, unit and integration testing of infrastructure changes, and drift detection and remediation. The topic includes strategies for safe changes and rollbacks, change coordination, error handling and recovery, and deployment patterns such as canary and blue green where applicable. It also encompasses automation and orchestration patterns, immutable infrastructure and self healing practices, autoscaling and scaling policies, automated patching and updates, secrets handling patterns using secret managers, and integrating observability and monitoring into automated workflows. Finally, candidates should be able to reason about trade offs between imperative and declarative approaches, scaling Infrastructure as Code across large projects and teams, and security and compliance considerations for automated provisioning.

HardTechnical

66 practiced

Problem-solving: Your team must upgrade a major Terraform provider or Terraform core that contains breaking changes. Outline a safe upgrade process across many repositories: pinning strategies, automated detection of incompatible usages, staging upgrades, and rollback plans if the upgrade causes regressions.

Sample Answer

**Situation & goal**I’d run a controlled, low-risk rollout so many repos using Terraform core/provider with breaking changes keep working and can be rolled back quickly.

**Pinning strategy**- Immediately pin current working versions in modules and root modules (required_providers / terraform block).- Use a central policy (registry module or shared module) that exposes allowed provider versions.- Example pin in repo:

hcl

terraform {
  required_version = ">= 1.3.0, < 1.5.0"
  required_providers {
    aws = { source = "hashicorp/aws", version = "~> 4.60.0" }
  }
}

**Automated detection**- Add static checks in CI: tflint, checkov, terraform validate with provider plugin versions, and a custom script to scan for unpinned ranges.- Run terraform init -upgrade --check-only in a sandbox to surface incompatibilities.- Add a nightly job across repos to detect drift vs pinned matrix and report failures to a dashboard/Slack.

**Staging & rollout**- Create a test matrix: local modules → dev repos → canary prod repos (low-risk services) → full prod.- For each stage: run plan, automated policy (Sentinel/Opa), and human review. Promote only when no diffs or approved diffs.

**Rollback & recovery**- Keep previous provider/core versions installable in CI by caching plugin binaries.- Record safe state: snapshot tfstate (encrypted), Terraform Cloud/State lock snapshots.- Rollback steps: restore state snapshot if mutated; revert version pins and reapply; run targeted taint/refresh if needed.- Prepare runbooks: immediate revert PR template, CI rollback job, contact list, and postmortem checklist.

**Observability & safeguards**- Monitor apply duration, resource changes, CPU/network anomalies after upgrade.- Use feature flags (module-level) to gate new provider features.

This plan minimizes blast radius, automates detection, stages rollout, and provides clear rollback/runbook for rapid recovery.

MediumTechnical

53 practiced

Problem-solving: Describe, step-by-step, how to configure Terraform remote state in AWS using an S3 backend with DynamoDB for state locking. Include the IAM permissions required for the state bucket and table, bucket policy considerations, and safe migration steps from a local state to this remote backend.

Sample Answer

**Approach (high level)** Step-by-step tasks: create S3 bucket + DynamoDB table, set IAM, add bucket policy, configure Terraform backend, test locking, migrate local state, verify.

**Steps**

1. Create resources- S3 bucket (versioning enabled, encryption AES256/KMS) and DynamoDB table with primary key `LockID` (string).- Example Terraform backend stanza to add later:

hcl

terraform {
  backend "s3" {
    bucket         = "my-tf-state-bucket"
    key            = "envs/prod/terraform.tfstate"
    region         = "us-east-1"
    encrypt        = true
    dynamodb_table = "terraform-locks"
  }
}

2. IAM permissions (principle of least privilege)- For users/CI that run terraform: s3:GetObject, s3:PutObject, s3:DeleteObject, s3:ListBucket (on the state prefix), s3:GetBucketVersioning, s3:PutObjectAcl (if needed), s3:GetBucketLocation; kms:Encrypt/Decrypt if using KMS.- For DynamoDB locking: dynamodb:PutItem, dynamodb:GetItem, dynamodb:DeleteItem, dynamodb:UpdateItem, dynamodb:Query.

Example policy (JSON):

json

{
  "Version":"2012-10-17",
  "Statement":[
    {"Effect":"Allow","Action":["s3:GetObject","s3:PutObject","s3:DeleteObject"],"Resource":["arn:aws:s3:::my-tf-state-bucket/*"]},
    {"Effect":"Allow","Action":["s3:ListBucket"],"Resource":["arn:aws:s3:::my-tf-state-bucket"]},
    {"Effect":"Allow","Action":["dynamodb:PutItem","dynamodb:GetItem","dynamodb:DeleteItem","dynamodb:UpdateItem","dynamodb:Query"],"Resource":["arn:aws:dynamodb:us-east-1:123456789012:table/terraform-locks"]}
  ]
}

3. Bucket policy considerations- Restrict access to known AWS principals, VPC endpoints, or require TLS. Prevent public access; block public ACLs and policies. If using KMS, add KMS key policy allowing encrypt/decrypt for IAM role.

4. Safe migration from local to remote- Backup local state: copy `terraform.tfstate` to safe location/version control (sensitive data redacted).- Initialize backend in a non-destructive way: - Add backend block to config. - Run `terraform init` — Terraform will prompt to copy local state to remote. Choose “yes”. - Alternatively, run `terraform init -migrate-state`.- Verify: run `terraform plan` to ensure no changes. Check S3 has the state object and DynamoDB is used for locks by attempting two concurrent applies (one should block).- Rollback plan: keep local backup; if migration fails, restore local state file and remove backend block.

**Checks & best practices**- Enable bucket versioning and server-side encryption.- Use least-privilege IAM roles for CI agents; prefer instance/role-based creds over long-lived keys.- Test locking by running concurrent operations.- Monitor S3 access logs and DynamoDB CloudWatch metrics.

EasyTechnical

61 practiced

Describe what remote state is in Terraform and why teams store state remotely. Explain benefits of using a remote backend with locking (for example S3 plus DynamoDB), and list common failure modes that require intervention when remote state is misconfigured.

HardTechnical

65 practiced

Leadership/system-design: Evaluate the trade-offs between monorepo and multi-repo strategies for large-scale IaC. Consider module reuse, release cadence, ownership boundaries, CI scalability, and permissions. Which model would you recommend and how would you migrate teams to it?

HardTechnical

54 practiced

Problem-solving: You applied an infrastructure change via Terraform which caused a production incident. Design an automated rollback strategy that supports quick remediation: detection, reverting infrastructure to the previous known-good state, reapplying configuration management, and coordinating any stateful rollbacks safely.

Sample Answer

**Situation & goal**I’d design an automated rollback pipeline that detects a bad Terraform change, safely reverts infra to the last known-good state, re-applies config management, and coordinates stateful rollbacks (databases, queues) with human validation where needed.

**Detection**- Monitor deployment health: Prometheus/Synthetic checks, alerting, and CloudWatch alarms triggered by SLA breaches.- Deploy pipeline emits a “failed” event when playbook tests or health checks fail within a short window post-deploy.

**Automated rollback flow (high level)**1. Lock Terraform backend (prevent concurrent writes).2. Identify last known-good commit/tag in Git and corresponding Terraform state version (S3/GCS versioning).3. Run an automated plan/apply to revert to that commit; run smoke tests.4. Re-run configuration management (Ansible/Puppet/Chef) against restored infra.5. If stateful components changed, trigger coordinated DB/app rollback steps (see below).6. Unlock backend, notify stakeholders, create incident ticket.

**Concrete automation snippet**

bash

# lock backend (example: DynamoDB lock for S3 backend)
export TF_WORKSPACE=prod
git checkout tags/last-known-good
terraform init -backend-config="bucket=infra-state" 
# restore state by using backend version or by importing resources as needed
terraform plan -out=rollback.plan
terraform apply "rollback.plan"

**Stateful rollback coordination**- Always snapshot DBs and volumes before any infra change. On rollback: - Restore DB from pre-deploy snapshot to a read-replica, run schema compatibility checks. - Use feature flags to pause writes or route traffic to read-only mode while restoring. - Run data reconciliation jobs, and only promote restored DB after integrity checks and a manual gate if needed.

**Safety & governance**- Require automated canary first (partial region/instance).- Multi-step with automated smoke tests and an approval gate before full rollback of stateful data.- Maintain immutable runbooks with exact commands, timeouts, and escalation contacts.

**Post-incident**- Capture root cause, add additional pre-deploy tests, shorten detection windows, and tag known-good commits automatically after successful canary.

This approach balances speed (automated revert of infra) with safety (snapshots, canaries, gates) and operational clarity (lock/state lineage and runbooks).

Unlock Full Question Bank

Get access to hundreds of Infrastructure Automation and Provisioning interview questions and detailed answers.

Join thousands of developers preparing for their dream job.