Infrastructure Automation and Provisioning Questions

Covers designing, implementing, and operating automated infrastructure provisioning and configuration using Infrastructure as Code practices and complementary automation patterns. Candidates should be able to select and author declarative infrastructure definitions with tools such as Terraform, CloudFormation, and Azure Resource Manager templates, and discuss configuration management tools such as Ansible, Puppet, or Chef. Core skills include modular and reusable code organization for multiple environments, variable and output management, remote state management and locking, idempotency and atomicity of operations, and version control integration for infrastructure artifacts. Candidates should understand testing and validation practices including linting, plan or dry run validation, unit and integration testing of infrastructure changes, and drift detection and remediation. The topic includes strategies for safe changes and rollbacks, change coordination, error handling and recovery, and deployment patterns such as canary and blue green where applicable. It also encompasses automation and orchestration patterns, immutable infrastructure and self healing practices, autoscaling and scaling policies, automated patching and updates, secrets handling patterns using secret managers, and integrating observability and monitoring into automated workflows. Finally, candidates should be able to reason about trade offs between imperative and declarative approaches, scaling Infrastructure as Code across large projects and teams, and security and compliance considerations for automated provisioning.

HardSystem Design

65 practiced

Design observability for infrastructure automation: what metrics, logs, and traces should IaC pipelines and provisioning tools emit? Propose dashboards and alerts to detect failed applies, unusually long plans, sudden growth in state file size, and trends that indicate drift or misconfiguration across environments.

Sample Answer

Requirements & goals:- Detect failed applies, long plans, sudden state growth, cross-env drift/misconfig.- Low-latency alerts (<5m), explainable context, correlate pipeline events with infra state.- Sources: IaC pipeline (CI/CD), provisioning tools (Terraform, CloudFormation, Pulumi), state backend (S3/GCS/DB), orchestration logs, audit logs, cloud provider metrics.

Key telemetry to emit- Metrics (time-series): - pipeline.plan.duration, pipeline.apply.duration (ms) - pipeline.plan.size_changes (bytes), statefile.size.bytes - pipeline.step.success_count / failure_count, apply.error_rate - resources.created, updated, deleted per run - drift.detected.count, drift.last_scan_duration - concurrency.count (parallel applies), lock.wait_time- Logs (structured JSON): - plan/apply start/end events with pipeline_id, commit, user, workspace, target, diff summary (counts and top N resource types) - detailed error logs with Terraform error codes, stack, provider errors - state-change events: file path, object version, ETag, size- Traces: - Distributed trace for a pipeline run: CI job → planner → state backend → provider API calls (timestamps for each provider request), to locate slow components.

Dashboards (examples)- Run Overview (per env): success rate, avg plan/apply durations, last 30 runs with status, top failing modules.- Plan vs Apply latency heatmap: distribution by module and user.- Statefile Health: size history per workspace, growth rate, object versions, top contributors (modules causing size growth).- Drift/Config Health: drift detections by env, % of resources drifted, recovery time after drift.- Provider call latency: 95/99p latencies to AWS/Azure/GCP per operation.

Alerts & thresholds (tunable, paged by severity)- P1 — Failed apply spike: apply.error_rate > 5% over 10 runs OR any production apply failure → paging with run id, user, error excerpt, rollback link.- P1 — Long-running apply: pipeline.apply.duration > SLO (e.g., 30m) OR > 3× baseline for that module → paging with trace and slow provider calls.- P2 — Unusually long plan: pipeline.plan.duration > 5× median for last 30 runs OR > 5m for small modules → notify.- P2 — Statefile sudden growth: statefile.size increases > 30% in one run or growth rate > 10% weekly → notify with diff of resource counts.- P3 — Drift trend: drift.detected.count increasing > threshold over 24–72h or same resource drifting repeatedly → warning to infra team.- P3 — Lock contention: lock.wait_time > threshold → warn.

Alert context & runbook- Each alert payload includes: pipeline_id, commit, user, workspace, diff summary, links (pipeline run, statefile object, apply plan, trace), recent logs, suggested actions (taint/rollback/re-run/inspect provider quotas).- Automated mitigations: for transient apply failures, auto-retry with backoff; for statefile corruption, promote previous version and create incident runbook.

Implementation notes & best practices- Instrument IaC tools via wrappers/hooks emitting structured events to a telemetry pipeline (Prometheus metrics, OpenTelemetry traces, ELK/Cloud logging).- Capture plan diffs as summaries only (counts and top types) to avoid leaking sensitive data.- Retain full apply logs and statefile versions with strict access controls and retention policies.- Baseline using rolling medians per module/workspace to avoid static thresholds.- Correlate with cloud provider quotas/errors and CI worker health to reduce noise.- Define SLOs: e.g., apply success rate ≥ 99% for non-prod, ≥ 99.9% for prod; 95th plan duration < X. Use error budgets to guide automation.

Example alert rule (pseudo):- Trigger if rate(pipeline_apply_failed{env="prod"}[1h]) > 0.01 and count_over_time(pipeline_apply_total{env="prod"}[1h]) >= 10- Include labels: workspace, module, last_error

This design gives SREs quick detection, rich context for triage, and measurable SLOs to reduce drift and failures.

HardSystem Design

54 practiced

Design an automated system that provisions ephemeral environments per feature branch in CI for a complex microservices app. Requirements: network isolation, per-branch DNS, short TTL secrets, cost controls, fast spin-up and teardown, and automated cleanup after 24 hours unless extended. Describe architecture, IaC templating approach, secret handling, and test/data seeding strategy.

Sample Answer

Requirements clarification: per-branch isolated network & DNS, short-lived secrets, cost limits, fast lifecycle (spin-up/teardown), automated cleanup after 24h (extendable). I'll describe a Kubernetes-centric SRE solution with Terraform + Helm/Kustomize and Vault-based secret management.

Architecture (high level)- CI triggers (GitHub/GitLab) call a provisioning service (serverless webhook) which writes a BranchEnvironment CR to a controller.- A Kubernetes controller (custom or Crossplane) reconciles: creates a Namespaced stack per branch in a shared cluster or a lightweight cluster per group.- Isolation: one Kubernetes Namespace per branch + NetworkPolicies + per-namespace Istio/Linkerd routing or sidecar mTLS to isolate east-west traffic.- Per-branch DNS: ExternalDNS + ingress controller assigns branch.<env>.company.com; cert-manager issues short-lived certs.- Data: use ephemeral databases (RDS/Cloud SQL) with one-db-per-branch via fast snapshot/clone or schema-per-branch; or use in-cluster Postgres operator (pg-operator) for very fast clones.

IaC & templating- Use Terraform modules for cloud infra (VPC subnets, DB snapshots, load balancer), parameterized by branch name and commit hash.- Use Helm charts + values templating (Helmfile or Flux HelmRelease) for app components; Kustomize overlays per-branch to patch image tags and replica counts.- Keep modules idempotent; tag resources with branch metadata for easy teardown.- Use a provisioning controller (Crossplane or custom operator) to compose Terraform + Helm reconciliation and record lifecycle metadata.

Secret handling- Central Vault (or cloud secret manager) issues dynamic, short TTL credentials: - DB: Vault database secrets engine issues DB user with 1h TTL, renew only for active sessions. - Cloud IAM: use STS/OIDC to mint ephemeral tokens for cloud APIs.- Kubernetes External Secrets or Vault CSI driver injects secrets into pods at runtime; secrets not stored in Git.- CI passes a short-lived provisioning token to create the environment; controller exchanges it for scoped credentials.

Cost control & fast spin-up- Resource quotas and LimitRanges per namespace; HorizontalPodAutoscaler and Cluster Autoscaler tuned for ephemeral burst capacity.- Default low replicas (1) and small instance types; use image caching and Warm pools (e.g., GKE node pools with pre-pulled images) for faster start.- Use DB snapshot cloning or schema cloning to avoid full restores (fast).

Lifecycle & cleanup- Controller adds TTL annotation with expiry; a reconciliation loop or Kubernetes TTL Controller deletes namespace/resources after 24h.- Extension API: allow branch owner to POST an extend action (with audit) to add time; require approvals for >72h.- On delete: run a finalizer that runs data scrub jobs (if required) then deletes cloud resources via Terraform destroy.

Test & data seeding- Seed via Kubernetes Job that runs after app readiness; seeds masked/anonymized synthetic dataset or a reduced production snapshot (obfuscated).- Use feature-flagged test data toggles and unique prefixes to avoid cross-branch collisions.- Integration tests run as CI stage against branch DNS; tests include smoke, contract, and performance quick checks.- For DB: use read-only snapshot + writeable schema clone for tests; run migration job once per environment.

Observability & safety- Auto-attach per-branch logs/metrics to ephemeral dashboards (Grafana with templated namespace variable).- Alerts on runaway cost (billing pipeline) and quota exhaustion; circuit breaker to refuse new envs if cost or resource pressure high.

Trade-offs- Single shared cluster lowers infra cost and speeds spin-up but needs strong network/quota isolation; per-branch clusters are stronger isolation but higher cost/latency.- Dynamic credentials add complexity but greatly reduce secret exposure.

This design prioritizes fast, secure, cost-controlled ephemeral environments with full automation for creation, runtime secrets, testing, and guaranteed cleanup.

EasyTechnical

61 practiced

What are common pitfalls around provider version changes in Terraform, and how do you pin provider and module versions safely? Explain how you would test provider upgrades in a staging environment and coordinate upgrades across team-owned modules to avoid unexpected resource replacements in production.

Sample Answer

Common pitfalls- Unpinned providers/modules cause automatic upgrades on terraform init/upgrade that can change resource schema and trigger replacements.- Minor vs major version semantics: some providers don’t follow semver strictly; a minor bump can be breaking.- Lockfile (terraform.lock.hcl) drift across machines/CI if not committed.- Transitive module dependency upgrades can change resources unexpectedly.

How to pin safely- In provider blocks pin constraints conservatively, prefer ~> for safe minor upgrades or = for critical infra:

hcl

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 4.20"    # allow non-breaking 4.x.y updates
    }
  }
}

- Commit terraform.lock.hcl to repo so every run uses the same provider checksums.- For modules, use explicit version tags or registry versions:

hcl

module "vpc" {
  source  = "git::https://...//modules/vpc.git?ref=v1.3.2"
}

- Use signed tags and semantic versioning; prefer patch upgrades automatically, review minor/major.

Testing provider upgrades in staging- Create a staging workspace that mirrors prod state (or use a copy of prod state in a sandbox).- Update provider version constraint and run terraform init -upgrade, then terraform plan and apply in staging.- Run smoke tests, integration tests, and service-level checks (health checks, connectivity).- Use drift detection and automated acceptance tests to catch replacements.

Coordinating team-owned module upgrades- Establish module release process: changelogs, migration notes, and explicit semver.- Use a central registry or internal module repo; require PRs that include plan output against a staging state.- For changes that may replace resources, add opt-in flags (create_before_destroy, lifecycle rules) and migration helpers.- Communicate upgrade windows, require cross-team review for major bumps, and stage rollout (canary environment → non-critical prod → full prod).- Automate alerts: block merges to prod if terraform plan shows destructive changes without owner sign-off.

This approach minimizes surprises: lock versions, test upgrades in a staging copy of prod, and coordinate module releases with clear migration steps and approvals.

MediumTechnical

80 practiced

Explain approaches to testing Terraform modules at unit and integration levels. Include terraform validate, plan-based checks, static analysis (tflint, tfsec), and integration tests using terratest (Go) or kitchen-terraform. Describe how you'd structure tests in CI to catch logical and security issues before an apply.

Sample Answer

Unit-level testing (fast, local, logic-focused)- terraform validate: always run to catch syntax/schema errors and provider blocks.- Static analysis: run tflint for style and provider-specific issues; run tfsec for common security misconfigurations (e.g., public S3, open security groups).- Plan-based checks (unit-ish): run terraform plan -out=plan.tfplan and terraform show -json plan.tfplan, then assert expected resources/attributes with small scripts (jq/Python). This catches logic errors without applying.

Integration-level testing (environment/behavior)- Terratest (Go): spin up real resources in a disposable test AWS/GCP account, verify end-to-end properties, then destroy. Example minimal test:

package test

import (
  "testing"
  "github.com/gruntwork-io/terratest/modules/terraform"
)

func TestTerraformModule(t *testing.T) {
  opts := &terraform.Options{TerraformDir: "../examples/simple"}
  defer terraform.Destroy(t, opts)
  terraform.InitAndApply(t, opts)
  output := terraform.Output(t, opts, "endpoint")
  if output == "" { t.Fatal("expected endpoint") }
}

- kitchen-terraform: useful if you prefer Ruby/Test-Kitchen for convergence tests or Ansible/inspec checks.

CI structure to catch logical & security issues before apply1. Lint stage: run terraform fmt -check, tflint, tfsec in parallel.2. Validate stage: terraform init + terraform validate.3. Plan stage: terraform plan -out=plan.tfplan; publish plan as artifact and run automated plan assertions (json) to flag unexpected changes (e.g., DB replacement).4. Integration stage (nightly or PR with tagging): run terratest/kitchen in ephemeral accounts or isolated workspaces. Mark long-running tests as gated for merge to protected branches.5. Manual approval (or policy enforcement): require PR approvals, policy checks (OPA/Conftest) and, where possible, run Sentinel/Pre-commit hooks.Operational best practices- Use isolated test accounts/projects, ephemeral state backends, and short-lived credentials.- Mock external dependencies for unit tests; reserve real cloud resources for integration tests.- Fail fast: break the pipeline on lint/validate/plan assertion failures; report detailed diffs to the PR.- Track test flakiness and parallelize tests by resource isolation to reduce cost/time.

HardTechnical

55 practiced

Compare declarative IaC tools (Terraform, CloudFormation) versus imperative provisioning (ad-hoc scripts, custom APIs) for large enterprise infrastructure. Discuss trade-offs in auditability, drift exposure, readability, operator skill requirements, debugging, day-2 operations, and suitability for compliance/regulatory needs.

Sample Answer

Situation: As an SRE evaluating tooling for large-enterprise infrastructure, I compare declarative IaC (Terraform, CloudFormation) versus imperative provisioning (ad-hoc scripts, custom APIs) across auditability, drift, readability, operator skills, debugging, day‑2 ops, and compliance.

Summary judgment: Declarative IaC is generally superior for auditability, drift control, reproducibility, and compliance at enterprise scale. Imperative approaches can be useful for one-off tasks, complex procedural workflows, or where provider APIs lack declarative support—but they increase operational risk and maintenance overhead.

Comparison by dimension:

- Auditability - Declarative: State files + plan/apply lifecycle provide a clear, reviewable delta; change history is explicit (VCS + state). Good for PR-based approvals and automated gating. - Imperative: Logs of script runs are the primary record; intent is not explicit—harder to reconstruct why a change occurred.

- Drift exposure - Declarative: Detectable via plan vs. state; drift can be automated and remediated. Still requires disciplined applies to avoid manual changes. - Imperative: Higher drift risk because idempotency and convergence are harder to guarantee; drift detection must be custom-built.

- Readability - Declarative: High-level, resource-oriented; easier for reviewers to reason about desired end state. Complex logic (loops, conditionals) can reduce readability. - Imperative: Procedural flow can be clear for operations engineers but becomes opaque as scripts grow; side effects make reasoning about final state harder.

- Operator skill requirements - Declarative: Requires understanding of resource models, provider semantics, state management, and lifecycle hooks. - Imperative: Requires programming/scripting expertise, deep knowledge of APIs, and discipline to make idempotent, observable tools.

- Debugging - Declarative: Failures map to resource operations; logs and plan help but root-cause may require provider knowledge. Debugging change order or dependency resolution can be tricky. - Imperative: Easier to add verbose diagnostics and step-through logic; but tracing distributed side effects across systems is more effort.

- Day‑2 operations - Declarative: Good for routine updates, scaling, and recoveries via predictable applies. State and drift tooling support ongoing ops. - Imperative: Better for complex migrations or workflows that must run procedural steps, but harder to maintain at scale.

- Compliance/regulatory suitability - Declarative: Strong fit—VCS-backed manifests, auditable plans, automated policy (e.g., Sentinel, OPA) enforcement, and reproducible environments. - Imperative: Harder to automate policy checks and ensure reproducibility; more manual controls required.

Trade-offs and recommendations:- Prefer declarative IaC for baseline infrastructure, security controls, and anything subject to audits. Enforce PR reviews, automated plans, and policy-as-code.- Use imperative scripts sparingly: wrap them in well-tested, versioned tooling, capture detailed runbooks/logs, and translate stable behaviors into declarative templates where possible.- For hybrid needs, adopt a pattern: declarative for resource definition + small, idempotent imperative operators (e.g., bootstrapping) with strict observability and automated drift checks.This balances safety, auditability, and the flexibility needed for complex enterprise environments.

Unlock Full Question Bank

Get access to hundreds of Infrastructure Automation and Provisioning interview questions and detailed answers.

Join thousands of developers preparing for their dream job.