Infrastructure as Code and Configuration Management Questions

Infrastructure as Code and Configuration Management covers designing, implementing, and operating infrastructure defined as code and the practices that keep that infrastructure consistent and auditable. Topics include Infrastructure as Code principles and patterns, declarative versus imperative approaches, idempotency, state management, module and template design for reusable infrastructure, version control integration, testing and validation of infrastructure code, drift detection and remediation, safe rollout and rollback strategies, policy as code and Git operations for infrastructure, and how to scan and remediate infrastructure misconfigurations. This topic also encompasses integration points with system configuration tools and considerations for managing secrets and secure defaults within infrastructure definitions.

EasyTechnical

29 practiced

Write a minimal CloudFormation (YAML) template that creates an S3 bucket with versioning enabled and default server-side encryption (AES256). Explain how CloudFormation ensures idempotency and describe what happens when the logical resource name or BucketName changes between deployments.

Sample Answer

Here's a minimal CloudFormation YAML template that creates an S3 bucket with versioning enabled and default server-side encryption using AES256:

yaml

AWSTemplateFormatVersion: '2010-09-09'
Description: Minimal S3 bucket with versioning and SSE-S3 (AES256)

Resources:
  MyBucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: !Sub '${AWS::StackName}-logs-bucket'  # optional; omit to let CFN generate a unique name
      VersioningConfiguration:
        Status: Enabled
      BucketEncryption:
        ServerSideEncryptionConfiguration:
          - ServerSideEncryptionByDefault:
              SSEAlgorithm: AES256

How CloudFormation ensures idempotency- CloudFormation treats the template as the desired state and compares it to the current state in the stack. Applying the same template repeatedly results in no-op updates if resources already match the desired properties.- It uses logical resource identifiers (e.g., MyBucket) plus resource type to track resources in the stack. For mutable properties, CFN issues API calls to change configuration; if nothing changed, no API call is made.

What happens if the logical resource name changes- Changing the logical name (e.g., MyBucket → MyNewBucket) makes CloudFormation consider the old resource deleted and a new resource created. CFN will attempt to delete the original bucket (which can fail if not empty) and create a new one. This is effectively a replacement and can cause data loss/unavailability unless you use Retain deletion policy or handle migration.

What happens if BucketName changes- If you explicitly set BucketName and change it, CloudFormation will replace the bucket (create new bucket with new name, then delete old one). Because S3 bucket names are globally unique, creation can fail if the name is taken. If you omit BucketName, CFN generates a stable physical name tied to the logical resource; changing logical name still triggers replacement.

Recommendations (SRE perspective)- Avoid changing logical names or explicit BucketName for production buckets.- Use DeletionPolicy: Retain when you must preserve data.- For zero-downtime migrations, create a new bucket and copy objects (sync), update references, then delete old bucket after verification.

HardTechnical

35 practiced

During a high-severity incident you discover a Terraform change deleted critical subnets causing service outages. As the SRE on-call, outline your immediate incident response actions, recovery steps including rollback vs manual repair, communication plan for stakeholders, and post-incident improvements to prevent recurrence.

Sample Answer

Immediate actions (first 0–15 minutes)- Stop the bleeding: revoke further Terraform runs by locking the state (e.g., disable CI job, set a maintenance flag) and remove any automation that could apply changes.- Triage impact: identify which subnets were deleted, list affected resources (instances, load balancers, route tables) and dependent services using cloud console/CLI and inventory (tags, IaC state).- Short-term mitigation: if DNS/ingress impacted, failover traffic to healthy regions/standby pools or put a redirect/maintenance page. Open an incident channel, assign roles (Incident Commander, Communications, Recovery lead, SREs).

Recovery steps- Decide rollback vs manual repair: - Prefer automated rollback if available and tested: restore Terraform state from recent backup and reapply a controlled run in plan-only mode to confirm resources to be recreated. This minimizes drift and preserves IaC as source-of-truth. - If rollback unsafe or slow, perform manual repair for critical pieces (recreate subnets, attach route tables, reassign IPs) following runbooks, then reconcile IaC afterwards.- Validate: confirm networking connectivity, health checks, service restoration, and run end-to-end smoke tests. Monitor metrics and error rates.

Communication plan- Immediate: notify on-call rotation, product and platform leads, and affected customer-facing teams with current impact, scope, and ETA for next update (every 15–30 minutes).- Executive: provide concise impact summary and customer-facing risk within 30–60 minutes.- Post-restoration: inform stakeholders when service restored and when full reconciliation completed. Prepare customer communication if SLAs/SLOs impacted.

Post-incident improvements- Root cause: conduct blameless postmortem within 48–72 hours documenting timeline, contributing factors (missing safeguards, PR review gaps).- Short fixes: add Terraform safeguards (prevent destroy on critical modules, require approval, state locking, run-only in CI with manual promotion), stricter code reviews, and pre-apply plan checks.- Long-term: enforce owner approvals, implement policy-as-code (OPA/Gatekeeper), automated integration tests for infra changes, more granular monitoring/alerting for networking deletions, and rehearsed runbooks + regular chaos/restore drills.- Metrics: track incident MTTR, number of infra rollbacks, and policy violations to measure improvement.

HardTechnical

33 practiced

Design an automated drift remediation system that detects resource drift and remediates non-critical drift automatically while alerting for critical resources. Include detection methods, reconciliation frequency, throttling/backoff, safety gates to avoid thrash, and audit/logging requirements an SRE would enforce.

Sample Answer

Requirements & goals:- Automatically remediate non-critical drift to reduce toil and keep desired-state.- Surface critical or high-risk drift for human review.- Ensure safety (no thrash), observability, and auditable changes.

Design summary:1) Detection methods- Continuous: AWS Config rules + custom rules (Lambda) for resource schema/permission drift.- Periodic: Cloud Custodian policies run on schedule for broader checks.- Agent / GitOps: Reconciliation controller (Kubernetes-style operator) comparing live vs Git-declared state.- Event-driven: CloudTrail / EventBridge to trigger immediate re-check on config-change events.

2) Reconciliation frequency- Near-real-time for sensitive resources: event-triggered immediate check + hourly scans.- Regular resources: scheduled full-scan daily; incremental scan every 15–60 minutes depending on resource criticality and churn.

3) Remediation policy & safety gates- Categorize resources: Critical (prod DB, IAM, network), Important, Non-critical.- Automatic remediation allowed only for Non-critical and some Important types (stateless infra, tags, autoscaling settings).- Critical drift generates high-priority alert + automated “containment” like revert-of-risky setting to safe-value only if explicit approved runbook exists.- Staged remediation: dry-run -> preflight validation -> canary remediation on subset -> full remediation.- Idempotent, declarative remediations stored as code (CloudFormation/SAM/Terraform apply or Cloud Custodian actions).

4) Throttling & backoff- Rate-limit remediations per resource-type and account (e.g., 5 changes/min per resource-type).- Exponential backoff on repeated failures: retry delays 1m, 5m, 20m, 1h; stop after N attempts and alert.- Circuit breaker: if >X failures in Y minutes for a resource-type, pause automated remediation and escalate.

5) Safety to avoid thrash- Change provenance check: ignore reconciliation if last change was caused by our remediation controller within a TTL (e.g., 10 minutes).- Cooldown window after remediation before re-evaluating resource.- Compare desired state source-of-truth (Git commit hash); do not auto-fix if desired-state is ambiguous.- Manual lock/maintenance windows supported via tagged annotations.

6) Audit, logging, and observability- Immutable audit events for every detection and action: timestamp, resource, drift diff, who/what triggered, remediation plan, pre/post snapshots, runbook link.- Store logs in centralized, tamper-evident store (e.g., S3 with object-lock + AWS CloudTrail + AWS Config snapshots); ship to SIEM (Splunk/ELK) and long-term cold storage.- Correlate with incident/alert system (PagerDuty) for critical drift.- Expose metrics: drift-rate, time-to-detect, time-to-remediate, remediation-failure-rate, throttling events. Define SLOs (e.g., 95% non-critical drifts auto-remediated within 1 hour).- Provide human-readable audit UI showing diffs, attempts, runbook, and ability to rollback.

Example workflow (non-critical):EventBridge detects tag removal -> invoke Lambda to evaluate -> Cloud Custodian runs dry-run -> policy allows auto-fix -> controller applies fix via API -> record pre/post snapshots and emit audit event -> metric incremented; if apply fails, retry with backoff; after max retries, create ticket and alert.

Tools & implementation notes:- Use AWS Config + Cloud Custodian + Terraform + a reconciliation controller (Python/Go).- Secure credentials using IAM roles, least privilege, and KMS-encrypted secrets.- Unit & integration tests for remediation scripts; chaos testing for safety.

This design balances automation and safety, reduces toil for benign drift, and ensures humans are alerted for high-risk changes with full auditability.

HardSystem Design

31 practiced

You inherited a monolithic Terraform repo that manages tens of thousands of resources. Propose a plan to restructure the codebase and states to improve team autonomy, reduce plan/apply times, and prevent accidental global changes. Discuss state sharding, repo organization, CI changes, and measures to prevent resource collisions.

Sample Answer

Requirements & constraints- Improve team autonomy, reduce plan/apply time, and eliminate accidental global changes while preserving reliability and auditability.- Minimize downtime and allow incremental migration of tens of thousands of resources.

High-level plan (phased)1. Discover & classify- Inventory resources (terraform state list, cloud API) and map to logical domains: infra, networking, platform services, apps, env (prod/stage), and owners.- Identify strong coupling (shared VPCs, IAM, DNS) and rank by risk.

2. Define sharding strategy (state sharding)- Shard by logical ownership + blast radius: e.g., env/service/component: prod-networking, prod-database, stage-app-*, infra-shared.- Keep truly global resources in a tiny “global core” shard (DNS zones, org-wide IAM) with strict controls.- Use remote backends with locking (Terraform Cloud/Enterprise workspaces or S3+DynamoDB locks) per shard.

3. Repo organization- Move from single monorepo root to multi-folder mono-repo or multiple repos per team depending on org: /modules, /platform/{networking,db,monitoring}, /services/{svc-a,svc-b} , /env/{prod,stage}- Extract reusable modules into a private registry (/modules) versioned semantically.- Keep module code separate from environment deployments.

4. CI/CD changes- Adopt per-shard CI pipelines: - PR lint & fmt (terraform fmt, tflint, checkov) - Run terraform init/plan only for affected shards (change detection by path and dependency graph) - Use parallel pipeline runs for independent shards; serialize for shards with shared dependencies. - Use plan artifacts and require human approval for applies to sensitive shards (global core, prod). - Use targeted applies and canary deploys for big changes.

5. Migration steps (safe, incremental)- For each shard: create new workspace/backend, import or terraform state mv resources into new state: - Create empty workspace with same resource definitions (or modules). - Use terraform state mv to move resources from monolith to shard; validate plan = no-op. - Run plan/apply in new workspace; verify. - Repeat with least risky shards first.- Maintain read-only archived monolith until migration completes.

6. Preventing resource collisions & accidental global changes- Enforce naming and tagging conventions (owner, shard, environment) via modules and pre-commit hooks.- Implement policy-as-code (Sentinel/Oso/OPA) in CI to block cross-shard resource creation, wildcard IAM, or changes outside owner boundaries.- RBAC: restrict who can approve applies to global core; use separate service accounts for CI per shard.- State locking and encryption (SSE/KMS, IAM policies).- Use terragrunt or build tooling to manage dependency graph and prevent implicit remote references.- Protect sensitive resources: require manual approval, multi-signature, or runbook.

7. Observability and validation- Add drift detection (scheduled plans) per shard and alert on divergence.- Track plan/apply times and state sizes; set thresholds (e.g., state file < X MB).- CI metrics: time-to-plan, failed-plans, apply frequency, change size.

Trade-offs and notes- More shards => faster plans but more operational overhead and potential duplication of code; mitigate with shared modules and automation.- Global core is a single point of control — keep it minimal and guarded.- Migration is manual and requires careful state mv operations and testing; plan rollback and backups.

Outcome- Faster, parallelizable plans; smaller state files; clear ownership boundaries; reduced chance of accidental global changes; auditable history and safer change control.

MediumTechnical

31 practiced

Compare Ansible and Terraform with respect to provisioning OS-level configuration (packages, services) and cloud resources. For common SRE tasks such as package installs, service configuration, and provisioning cloud infrastructure, explain when to use each tool and how to integrate them without causing conflicts.

Sample Answer

Short answer: Use Terraform to declare and manage cloud resources (networks, VMs, load balancers, managed DBs). Use Ansible for OS-level configuration inside those resources (package installs, services, runtime configs). Keep a clear separation of concerns, integrate via orchestration steps, and avoid race conditions by using each tool’s strengths.

Why:- Terraform is a desired-state, immutable-infrastructure tool with strong lifecycle and dependency management for cloud APIs. It tracks state, plans changes, and is ideal for provisioning resources.- Ansible is an agentless configuration-management/orchestration tool that excels at idempotent changes on running machines (apt/yum, systemd, config files, templates).

When to use which (examples):- Provision EC2/GCE instances, VPCs, IAM, managed RDS: Terraform.- Install nginx, configure /etc/nginx/nginx.conf, enable systemd service, deploy app artifacts: Ansible.- For cloud-init or image baking (Packer): consider baking common packages so Ansible surface area is smaller.

Integration pattern (recommended):1. Terraform apply: create network, instances, security groups, and output connection details (IPs, instance IDs, SSH keys).2. Inventory generation: use Terraform outputs to feed dynamic inventory (Terraform file, inventory script, or Terraform + local_file to produce Ansible inventory).3. Ansible playbooks: configure packages/services, deploy app, run health checks.4. Use orchestration tooling (CI/CD) to run steps sequentially and gated by Terraform plan/apply success.

Avoiding conflicts:- Single source of truth: don’t have both tools manage the same resource. E.g., let Terraform set instance user-data and tags; don’t have Terraform also manage package lists that Ansible changes.- Immutable attributes in Terraform: treat resources created by Terraform as “owned” — avoid external mutation that Terraform can’t see.- Use lifecycle ignore_changes for attributes you want Ansible to manage (e.g., metadata or tags) so Terraform won’t attempt to revert Ansible changes.- Idempotency & retries: ensure Ansible playbooks are idempotent and safe to re-run; use handlers for service restarts.- Locking and CI: use remote Terraform state with locking (S3+DynamoDB, GCS) and run Terraform in CI to prevent concurrent applies. Similarly, coordinate Ansible runs via orchestration (e.g., Ansible Tower/AWX).

Advanced patterns:- Use Terraform provisioners only sparingly (remote-exec, local-exec) for bootstrapping; prefer Ansible for full configuration.- Consider image baking (Packer + Terraform) to bake as much configuration as possible to reduce run-time drift.- For config drift detection, have periodic Ansible convergence jobs and Terraform plan diffs as part of CI.

This separation leverages Terraform for lifecycle of cloud infra and Ansible for mutable, detailed OS-level configuration while preventing ownership overlap and drift.

Unlock Full Question Bank

Get access to hundreds of Infrastructure as Code and Configuration Management interview questions and detailed answers.

Join thousands of developers preparing for their dream job.