Infrastructure as Code Tool Proficiency (Terraform/CloudFormation/Ansible) Questions

Deep proficiency in at least one IaC tool. For Terraform: understand resources, data sources, variables, outputs, local values, modules, state management, state locking, backend configuration (S3, Terraform Cloud), and best practices (remote state, sensitive variables, module organization). For CloudFormation: understand templates (YAML/JSON), stacks, parameters, conditions, mappings, resources, outputs, and intrinsic functions. For Ansible: understand playbooks, roles, inventory, variables, handlers, and idempotency. Write reusable, maintainable code: modules for Terraform, roles for Ansible. Understand code organization, naming conventions, and team collaboration practices.

HardSystem Design

31 practiced

Design an enterprise-scale Terraform architecture for a multi-region, multi-account AWS environment with thousands of engineers. Cover module registry approach, remote state isolation, cross-account role assumptions, CI/CD pipeline separation, drift prevention, secrets, and governance (policy-as-code). Provide a high-level diagrammatic description of components and responsibilities.

Sample Answer

Requirements & constraints:- Multi-account (org units: org-infra, prod, non-prod, tools), multi-region, thousands of engineers, strict security & audit, high availability.

High-level architecture (ASCII):Registry/API/CD → Module Registry (private) → Git repos | v CI/CD: Infra Pipelines (centralized) <--> Remote State (S3 + Dynamo) | | App Pipelines (separate, limited infra changes) v | State per-account-per-region (encrypted, KMS) v Cross-account roles (assume-role) ----> Target AWS Accounts

Components & responsibilities:- Module registry: Private Terraform Module Registry (registry.terraform.io-compatible) or artifactory; modules owned by platform team, semantic versioning, automated tests, policy checks before publish.- Remote state: S3 buckets per account-region with server-side encryption (KMS CMKs per account), DynamoDB for locking; state stored per-stack (env/account/region/stack) to isolate blast radius.- Cross-account access: Central CI service assumes a short-lived IAM role in target account using AWS STS. Roles limited by IAM permissions and condition keys (source ARN, MFA). No long-lived keys in pipelines.- CI/CD separation: Two pipeline families — Platform/infra pipelines (run Terraform apply, require peer review, run in hardened runners with OIDC or STS) and App pipelines (consume platform outputs, no infra apply rights). Use separate IAM identities and approval gates for prod.- Drift prevention & detection: Prevent by policy-as-code blocking out-of-band modifications (Service Control Policies + IAM permissions); detect using periodic "terraform plan" or drift scanner (config rules, AWS Config, periodic Terraform runs) and alert/auto-remediate via orchestrated runs or change requests.- Secrets: Central Vault (HashiCorp Vault with auto-unseal via KMS or AWS Secrets Manager for low-sensitivity). Terraform gets dynamic short-lived secrets via Vault/SSM parameter store and least-privilege roles; never store secrets in state (use state encryption and sensitive flag).- Governance / Policy-as-code: Implement OPA/Conftest or Terraform Cloud/Enterprise policy sets (Sentinel) enforcing tagging, allowed instance types, network boundaries, and module provenance. Integrate policy checks in PRs and CI pre-apply gates. Audit logs to centralized SIEM.- Observability & Audit: Central logging of Terraform runs, AWS CloudTrail, S3 access logs, DynamoDB, and pipeline artifacts. Metrics & alerts for failed plans, drift, unauthorized changes.- Developer UX & guardrails: Self-service portal backed by modules, example stacks, scaffolding templates, automated module upgrades, and clear contributor process.

Trade-offs & rationale:- Centralized infra pipeline reduces risk; per-account state isolates failures.- Module registry + versioning enables scale and reuse while preventing uncontrolled drift.- Short-lived cross-account roles + Vault minimize secret exposure.- Policy-as-code enforces compliance early; periodic drift detection catches manual changes.

Example minimal flow:Developer updates module -> CI tests & policy checks -> Publish module -> App repo updates version -> App pipeline runs terraform plan against its state (S3 in its account/region) -> Infra pipeline with assume-role performs apply after approvals.

This design balances scale, security, and velocity for thousands of engineers while keeping blast radius small and governance automated.

EasyTechnical

31 practiced

Describe approaches to manage secrets used by Terraform and Ansible: where to store secrets, how to avoid committing secrets into tfstate or playbooks, and patterns to inject secrets into runs securely (examples: HashiCorp Vault, AWS Secrets Manager, Ansible Vault, environment variables).

MediumTechnical

33 practiced

Discuss trade-offs between Terraform provisioners, CloudFormation cfn-init/user-data, and using Ansible for instance configuration. Recommend a pattern for bootstrapping application servers in a cloud-native environment and justify the recommendation.

Sample Answer

Situation: As an SRE you must bootstrap reliable, repeatable application servers at scale. Below I compare three approaches and give a recommended pattern.

Trade-offs- Terraform provisioners - Pros: quick for one-off commands during provisioning. - Cons: brittle, runs from client machine (or remote-exec), mixes infra and config, poor retrying/observability, hard to scale or idempotently re-run. Not recommended for production configuration.- CloudFormation cfn-init / user-data (cloud-init) - Pros: runs at first boot, good for lightweight bootstrapping (installing agents, pulling config), idempotent if scripted, integrates with AWS metadata/signaling (cfn-signal). - Cons: complex scripts get unwieldy, testing is harder, platform-specific details leak into templates.- Ansible - Pros: expressive, idempotent, testable, great for ongoing config drift remediation and complex orchestration. Works agentless or with an agent. - Cons: slower for initial boot at scale unless run from a controller; requires network/credentials; not ideal as the single source for first-boot immutability.

Recommended pattern (cloud-native, scalable)1. Build immutable images (AMIs / container images) with Packer: bake application and most dependencies into the image → fast, predictable boot time, fewer runtime failures.2. Use cloud-init/user-data for minimal, deterministic bootstrapping: register instance with service discovery, fetch secrets/credentials from a vault, run health-check signaling (cfn-signal / cloud provider API), start container runtime or systemd service.3. Use orchestration/CI for deployments: deploy containers via Kubernetes or autoscaling groups with baked images; use pipelines to promote images.4. Use Ansible (or configuration management) for non-time-critical tasks, emergency fixes, periodic audits, or runbooks. For long-lived VMs that require live drift management, run Ansible Tower/Automation Controller on a schedule.5. Avoid Terraform provisioners for in-instance config; keep Terraform focused on cloud resource lifecycle only.

Justification- Immutable images + minimal cloud-init gives repeatability, speed, and simpler troubleshooting (image = single source of truth). Cloud-init handles provider-specific bootstrapping cleanly. Ansible complements by managing drift and complex operational workflows without coupling infra provisioning tools to runtime configuration. This combination maximizes reliability, testability, and safety for an SRE-run platform.

HardTechnical

29 practiced

Write a test strategy for Ansible roles using Molecule in CI for both containerized and cloud-instance testing. Include how to mock external services, run tests in parallel, run testinfra/unit tests, and perform failure injection to validate error handling.

Sample Answer

Approach: Define a Molecule-driven test strategy that runs fast container-based scenarios for PRs and fuller cloud-instance scenarios in CI (canary / nightly). Combine unit tests for Ansible modules/filters, testinfra for integration checks, mocking for external services, parallel execution, and deliberate failure-injection to validate error paths.

Molecule config (excerpt) — multiple scenarios:

yaml

# molecule.yml (simplified)
platforms:
  - name: instance
    image: geerlingguy/docker-ubuntu2004-ansible:latest   # container scenario
    pre_build_image: true

provisioner:
  name: ansible
  lint:
    name: ansible-lint

scenario:
  test_sequence:
    - lint
    - dependency
    - create
    - converge
    - verify
    - destroy

CI orchestration:- PR pipeline: run molecule test --scenario-name=container in parallel across roles using GitLab CI/GitHub Actions matrix; limit concurrency to avoid host exhaustion.- Merge/nightly: trigger cloud scenario using Molecule with delegated driver (ec2/azure/gce). Use ephemeral VPCs and instance tagging; tear down on failure.

Mocking external services:- Use local WireMock or moto_server for AWS APIs inside container scenario. Start mocks in molecule prepare/playbooks (or use Docker Compose to run mock services). In cloud runs, use recorded responses (VCR-like) or inject test doubles by pointing role variables to mock endpoints.- Example task to start WireMock in prepare:

yaml

- name: start wiremock
  docker_container:
    name: wiremock
    image: rodolpheche/wiremock
    state: started
    published_ports: "8080:8080"

Testinfra & unit tests:- Place testinfra tests under molecule/default/tests/test_*.py and run via pytest in verify step.- Unit tests: ansible-test, pytest for custom modules/filters; run in CI before Molecule.

Parallelization:- Use CI job matrix to run independent Molecule scenarios/roles in parallel.- For within-host parallelism, run pytest -n auto (pytest-xdist) for testinfra tests.

Failure injection:- Create dedicated "fault" scenarios that manipulate variables to trigger failures (bad creds, unreachable services).- Use Chaos tasks: temporarily block ports via iptables, inject disk full (fallocate), or kill services using systemd during converge/verify.- Example verify test that ensures role handles service restart failure:

python

def test_idempotent_and_error_handling(host):
    svc = host.service("myservice")
    # simulate failure
    host.run("sudo systemctl stop myservice")
    # run handler (invoke role handler or re-run converge)
    result = host.run("ansible-playbook /tmp/recover.yml")
    assert result.rc == 0
    assert svc.is_running

Observability and metrics:- Capture Molecule logs/artifacts (stdout, fired events). Upload VM logs and pytest JUnit XML to CI.- Define SLO-related assertions as tests (startup latency, health endpoints).

Security and cost controls:- Limit cloud run TTLs, use small instance types, and require manual approval for large-scale cloud tests.

Trade-offs:- Containers = fast, cheap; cloud = realistic (kernel, cloud metadata). Use both: PRs run containers, nightly/cloud validates real infra.

This strategy ensures fast feedback, realistic end-to-end validation, robust error-path coverage, and scalable CI execution.

HardTechnical

26 practiced

A CloudFormation stack update partially succeeded and left resources in inconsistent states (some replaced, some updated). You must reconcile the stack and minimize downtime, especially for stateful resources like RDS. Walk through a safe reconciliation and rollback strategy, including snapshots, change sets, and manual steps.

Sample Answer

Situation: A CloudFormation update partially succeeded—some resources were replaced, others updated—leaving the environment inconsistent. Priority: minimize downtime and protect stateful data (RDS), then reconcile the stack to desired template.

Strategy (high-level):1. Stop and assess2. Protect data (snapshots / replicas)3. Create and review change-set(s)4. Execute controlled reconciliation or manual fix5. Validate and harden for future updates

Detailed safe plan (SRE-focused):

1) Immediate assessment- Check CloudFormation stack events and resource statuses (aws cloudformation describe-stack-events).- Identify which resources were replaced vs updated and whether replacements are complete/rolled back.- Note any resources in FAILED/ROLLBACK_COMPLETE states.- Run CloudFormation drift detection on the stack to see differences.

2) Protect stateful resources first (RDS)- Take an immediate manual snapshot of any affected RDS instances: aws rds create-db-snapshot --db-instance-identifier mydb --db-snapshot-identifier pre-reconcile-YYYYMMDD- If possible, create an RDS read-replica (for MySQL/Postgres) to use as a warm standby: aws rds create-db-instance-read-replica ...- If Multi-AZ is enabled, prefer a controlled failover to the standby for quick recovery: aws rds reboot-db-instance --db-instance-identifier mydb --force-failover- Ensure automated backups are intact and identify earliest restore points.

3) Prepare a non-destructive CloudFormation plan- Never execute ad-hoc updates on production without a change set. Generate a change set from the current cloudformation template and the intended template: aws cloudformation create-change-set --stack-name my-stack --template-body file://new.yml --change-set-name reconcile-cs --capabilities CAPABILITY_NAMED_IAM- Inspect the change set carefully for any Replace actions on RDS (Modify vs Replace). If it would replace RDS, stop and rework the template (use UpdateReplacePolicy/DeletionPolicy: Retain, or modify properties to avoid replacement).- If the drift shows that resource logical IDs changed in the template, adjust template to match physical resources (use stack imports if necessary).

4) If CloudFormation cannot safely reconcile without replacement- Use an in-place manual reconciliation: - For RDS: keep the current instance (avoid CFN-triggered replacement). Set DeletionPolicy: Retain and UpdateReplacePolicy: Retain in the template, then update stack to adopt desired properties that don’t require replacement (or use modify-db-instance via RDS API for safe changes). - For EC2/EBS/ALBs: reattach or reconcile using resource physical IDs or exported outputs. Consider creating a small temporary CloudFormation nested stack mapping to existing physical resources (resource import).- If a resource was replaced successfully and the new resource is healthy, update the template to reference the new physical resource (export/import or update mappings) so CFN’s desired state matches reality.

5) Controlled rollback if needed- If change set shows dangerous replacements and manual reconciliation is not feasible, roll back to last known-good template and stack state: - Use CloudFormation Stack rollback if still in progress. If not, create a change set that will revert to previous template and parameters. - For stateful resources, instead of letting CFN delete/replace, set DeletionPolicy: Retain, import the existing snapshot/instance into the stack, or restore from snapshot to a new instance and adopt it into CFN via resource import.

6) Testing & cutover- Use blue/green-style approach where possible: provision new resources in parallel, switch traffic (Route53/ELB) when healthy.- For RDS, promote a read replica, or restore snapshot into a new instance, run schema checks, then point application to the new endpoint during a short maintenance window.- Validate backups, metrics, application health checks, and run smoke tests.

7) Post-reconciliation hardening- Add safeguards to templates: DeletionPolicy/UpdateReplacePolicy: Retain for stateful resources; enable RollingUpdate/UpdatePolicy for autoscaling groups; use stack policies to block replacements for critical resources.- Add pre-update runbook: automated pre-update snapshots, automated change-set creation and approval, drift detection and preflight tests in a staging environment.- Improve CI: run CloudFormation diffs and test change-sets on a duplicate environment.

Example decisions and commands you’ll use:- Snapshot: aws rds create-db-snapshot ...- Create change set: aws cloudformation create-change-set ...- Drift detection: aws cloudformation detect-stack-drift && describe-stack-resource-drifts- Failover (multi-AZ): aws rds reboot-db-instance --force-failover

Why this works:- Prioritizes data safety (snapshots/replicas) before any destructive actions.- Uses change sets and drift detection to avoid surprises and see planned replacements.- Prefers manual, controlled fixes or blue/green cutovers for stateful resources to minimize downtime.- Adds template protections to reduce recurrence.

Common edge cases:- Resource physical IDs changed and CFN “owns” none: use resource import or update templates to adopt new IDs.- Replacement required for RDS-major version: restore snapshot to new instance, test, then cut traffic.- IAM/permission failures during update: grant CloudFormation proper roles or split IAM changes into separate, reviewed deployments.

This approach balances safety (snapshots, retaining state) with minimizing downtime (replicas, parallel provisioning, controlled failover), and uses CFN change-sets and drift detection to guide safe, auditable actions.

Unlock Full Question Bank

Get access to hundreds of Infrastructure as Code Tool Proficiency (Terraform/CloudFormation/Ansible) interview questions and detailed answers.

Join thousands of developers preparing for their dream job.