Infrastructure Automation and Provisioning Questions

Covers designing, implementing, and operating automated infrastructure provisioning and configuration using Infrastructure as Code practices and complementary automation patterns. Candidates should be able to select and author declarative infrastructure definitions with tools such as Terraform, CloudFormation, and Azure Resource Manager templates, and discuss configuration management tools such as Ansible, Puppet, or Chef. Core skills include modular and reusable code organization for multiple environments, variable and output management, remote state management and locking, idempotency and atomicity of operations, and version control integration for infrastructure artifacts. Candidates should understand testing and validation practices including linting, plan or dry run validation, unit and integration testing of infrastructure changes, and drift detection and remediation. The topic includes strategies for safe changes and rollbacks, change coordination, error handling and recovery, and deployment patterns such as canary and blue green where applicable. It also encompasses automation and orchestration patterns, immutable infrastructure and self healing practices, autoscaling and scaling policies, automated patching and updates, secrets handling patterns using secret managers, and integrating observability and monitoring into automated workflows. Finally, candidates should be able to reason about trade offs between imperative and declarative approaches, scaling Infrastructure as Code across large projects and teams, and security and compliance considerations for automated provisioning.

HardTechnical

47 practiced

In a GitOps model where manifests or IaC are stored in git, explain secure patterns to manage secrets across clusters and environments. Compare sealed-secrets, SOPS-encrypted files, external secret operators, and fetching from an external vault at runtime. Evaluate trade-offs in developer experience, auditability, and operational security.

Sample Answer

**Summary / approach**I’d evaluate options on three axes: developer experience (DX), auditability, and operational security (secrets at rest/in transit, key management, blast radius). Below I compare sealed-secrets, SOPS (git-encrypted), external-secret operators, and runtime Vault fetches, then give recommended patterns.

**1) Sealed‑Secrets (controller + git)**- DX: Good — developers commit sealed secrets like normal manifests.- Auditability: High — encrypted blobs stored in git with history.- Ops security: Moderate — controller holds cluster private key; compromise of controller or cluster private key can decrypt all. Key rotation is possible but operationally heavy.- Best for: cluster-scoped secrets where offline git sync is needed.

**2) SOPS-encrypted files in git**- DX: Requires tooling/keys per dev/CI (age/GPG/KMS). Slightly higher friction but scriptable.- Auditability: Excellent — file history in git; encryption metadata visible.- Ops security: Strong when using cloud KMS (no cluster private key); encryption keys live outside cluster.- Best for multi-environment secrets and DR-friendly workflows.

**3) External Secret Operators (e.g., ExternalSecrets)**- DX: Developers reference secret manifests; operator reconciles from Vault/KMS/SecretsManager.- Auditability: Medium — git shows references but not content; access logs live in vault/cloud provider.- Ops security: Good if operator uses short-lived credentials and RBAC; operator compromise risks secret retrieval.- Best for central secret stores with cross-cluster propagation.

**4) Fetching from external Vault at runtime (sidecar/init or app fetch)**- DX: Higher developer burden (integrate SDKs or sidecars) but promotes least-privilege patterns.- Auditability: Excellent via Vault audit logs; git contains no secret data.- Ops security: Best — dynamic secrets, leases, renewals, short TTLs, strong MFA/KV access controls. More complex to operate.- Best for high-security workloads and dynamic credentials.

**Trade-offs & recommendations**- If you need simple GitOps with git-only workflow: SOPS (with KMS) is my preference — strong key isolation, auditability, and scriptable CI decryption.- For cluster-local workflows that favor developer UX: sealed-secrets is acceptable but enforce strict key rotation, backups, and limit controller privileges.- For central secret governance and dynamic secrets: ExternalSecret operator + Vault backend or runtime Vault fetch for highest operational security.- Operational controls to apply across all: CI-based decryption with short-lived tokens, RBAC least privilege, key rotation policies, automated audit alerting, and documented incident playbooks.

I’d choose SOPS + ExternalSecrets (Vault backend) as a balanced pattern: git-stored encrypted values for auditability and human workflows, plus runtime retrieval for dynamic or high-risk secrets.

MediumTechnical

49 practiced

Describe strategies and tooling you would implement to detect and remediate configuration drift across cloud infrastructure. Include scheduled scans, drift detection via 'terraform plan', cloud-native tools (AWS Config), and remediation workflows that balance automated fixes and human approvals.

EasyTechnical

64 practiced

Explain idempotency and atomicity in the context of infrastructure automation. Provide concrete examples of non-idempotent actions and describe techniques you would apply in modules or playbooks to ensure idempotent behavior across repeated runs.

Sample Answer

**Definition — idempotency vs atomicity**

- Idempotency: running the same automation multiple times yields the same end state and only makes changes when needed (desired-state). Example: creating a user if it doesn't exist; repeated runs are no-ops.- Atomicity: a series of changes either all succeed or the system is left unchanged/clean (all-or-nothing). Important for multi-step workflows (DB + app + network).

**Concrete non‑idempotent actions**- Appending lines to a file without checking (duplicates on every run).- Running schema migration scripts that re-apply changes each run.- Using remote-exec/provisioners to install packages without checking state (restarts or repeated installs).- Generating new secrets on every run and overwriting previous ones.

**Techniques to ensure idempotency**- Use declarative tools (Terraform, Ansible modules) instead of ad-hoc scripts.- In Ansible: - Prefer built-in modules (user, file, apt) which are idempotent. - Use creates / removes / args that make a task conditional. - Set changed_when / check_mode for custom commands. - Use handlers for restarts so they run only when triggered. - Use stat/lookup to guard file edits and templating (avoid appending blindly).- In Terraform: - Model resources declaratively; avoid local-exec for stateful changes. - Use remote state locking (backend like S3+DynamoDB) to avoid concurrent applies. - Rely on plan -> apply; avoid recreating secrets each apply (use data sources or external state). - Use lifecycle rules (prevent_destroy) when appropriate.

**Techniques for atomicity**- Group related resources into a single module or transaction boundary; perform orchestration so failure triggers rollbacks or compensating actions.- Use blue/green or canary deployments to ensure safe cutovers.- Implement retries with backoff and idempotent retryable operations.- For multi-step critical flows, use orchestration service (Step Functions, workflows) that can rollback/compensate on failure.

These practices keep runs safe, repeatable, and predictable for reliable infrastructure automation.

HardSystem Design

78 practiced

Design a compliance and auditing framework for automated infrastructure provisioning. Explain how you would capture identity (who initiated change), evidence (plan outputs, test results), policy enforcement (policy-as-code), immutable logs for audits, storage and retention policies, and integration with SIEM or GRC tools. Describe how automated enforcement differs from reporting-only approaches.

Sample Answer

**Clarify requirements & constraints**- Enforce compliance for automated infra provisioning (IaC pipelines), capture provable evidence, retain immutable audit trails, integrate with SIEM/GRC, support both report-only and automated-enforce modes, meet retention/crypto requirements.

**High-level architecture**- IaC repo (Terraform/CloudFormation) → CI/CD pipeline (lint, plan, test) → Policy-as-Code gate (OPA/Rego or Sentinel) → Orchestrator (e.g., Spacelift/ArgoCD) → Cloud provider APIs.- Evidence store & WORM ledger → Long-term object store (S3 with Object Lock/Glacier Deep Archive) + append-only ledger (blockchain-like or write-once DB like DynamoDB with cryptographic anchoring).- SIEM/GRC bus — events pushed to Kafka/Elastic/Splunk and to GRC via connectors.

**Core components & responsibilities**- Identity capture: integrate pipeline with centralized auth (OIDC + IAM roles). Every pipeline run links to user or service identity and ephemeral token. Record principal, session id, MFA context.- Evidence capture: persist plan outputs (terraform plan), unit/integration test results, policy evaluation logs, PR diffs, approved approvals. Store signed artifacts (sign with pipeline private key).- Policy enforcement: Policy-as-code (OPA/Sentinel) executed in pipeline and as admission controller in runtime (Gatekeeper for Kubernetes). Policies return allow/deny and remediation suggestions.- Immutable logs: write event stream to append-only ledger. Hash each event and anchor periodic Merkle root to external notarization (e.g., blockchain or third-party timestamping).- Storage & retention: short-term: encrypted object store with lifecycle rules; long-term: WORM-enabled S3 with server-side encryption and MFA-delete, lifecycle to Glacier. Retention policies codified per-regulation.- Integration: forward structured events (CEF/JSON) to SIEM, tag with correlation IDs; expose evidence via GRC API for audits, generate attestations/reports automatically.

**Data flow summary**1. Dev triggers PR → CI runs tests + terraform plan.2. Policy-as-code evaluates; if enforce mode and deny → pipeline fails; if report-only → log decision and notify.3. Approved plans signed, apply executed by orchestrator using ephemeral role.4. Pipeline emits signed artifacts + events to evidence store and ledger; SIEM ingests events.

**Automated enforcement vs reporting-only**- Automated enforcement blocks undesired changes before they reach runtime—prevents drift and reduces blast radius. Requires high-confidence policies and fast remediation paths.- Reporting-only provides visibility but relies on human intervention; useful for onboarding new policies and low-risk controls.- Trade-offs: enforcement increases velocity friction and false-positive risk; mitigate with progressive rollout (monitor → warn → enforce), human-in-loop approvals for high-risk changes, and clear rollback playbooks.

**Key operational considerations**- Key management: HSM for signing keys.- Availability: replicate evidence store across regions.- Audits: provide auditors with signed bundles and verification tooling.- Metrics: policy deny rates, time-to-remediate, evidence integrity checks.

This design delivers end-to-end verifiable provenance, enforceable controls, immutable audit trails, and practical integration with SIEM/GRC while allowing measured adoption of enforcement.

MediumTechnical

66 practiced

Ansible playbooks sometimes include non-idempotent tasks when using raw scripts or improper module flags. Describe patterns and techniques to ensure idempotency in Ansible playbooks, how to detect non-idempotent tasks during testing, and how to enforce idempotency as part of CI validation.

Unlock Full Question Bank

Get access to hundreds of Infrastructure Automation and Provisioning interview questions and detailed answers.

Join thousands of developers preparing for their dream job.