Infrastructure Automation and Provisioning Questions
Covers designing, implementing, and operating automated infrastructure provisioning and configuration using Infrastructure as Code practices and complementary automation patterns. Candidates should be able to select and author declarative infrastructure definitions with tools such as Terraform, CloudFormation, and Azure Resource Manager templates, and discuss configuration management tools such as Ansible, Puppet, or Chef. Core skills include modular and reusable code organization for multiple environments, variable and output management, remote state management and locking, idempotency and atomicity of operations, and version control integration for infrastructure artifacts. Candidates should understand testing and validation practices including linting, plan or dry run validation, unit and integration testing of infrastructure changes, and drift detection and remediation. The topic includes strategies for safe changes and rollbacks, change coordination, error handling and recovery, and deployment patterns such as canary and blue green where applicable. It also encompasses automation and orchestration patterns, immutable infrastructure and self healing practices, autoscaling and scaling policies, automated patching and updates, secrets handling patterns using secret managers, and integrating observability and monitoring into automated workflows. Finally, candidates should be able to reason about trade offs between imperative and declarative approaches, scaling Infrastructure as Code across large projects and teams, and security and compliance considerations for automated provisioning.
HardTechnical
48 practiced
Explain how Infrastructure as Code can be used to enforce cost-optimization strategies: rightsizing, scheduled shutdowns for non-prod, spot instances/Preemptible VMs, and autoscaling policies. As a Solutions Architect, describe policies, tagging, automated enforcement, and reporting you would implement to control cost across multiple teams and measure ROI.
Sample Answer
Requirements & scope:- Enforce rightsizing, scheduled shutdowns for non‑prod, use of spot/preemptible VMs where appropriate, and autoscaling policies across multiple teams while measuring cost ROI.Policy & guardrails:- Define organizational Guardrails in IaC: AWS Organizations SCPs + AWS Config rules, Azure Policy, or GCP Organization Policies to ban oversized instance families in dev, require autoscaling groups for stateless services, and restrict use of on‑demand where spot is safe.- Implement policy-as-code with Terraform + Sentinel/OPA or CloudFormation + Config Rules to evaluate drift and block non‑compliant deployments.Tagging & metadata:- Enforce a mandatory tagging schema in IaC modules: cost_center, owner, environment (prod/nonprod), lifecycle (ephemeral/persistent), workload_tier, business_unit, sla. Validate tags at plan/apply time using pre‑commit hooks and policy checks.- Use tags to drive automation (e.g., identify nonprod VMs for scheduled shutdown).Automated enforcement & remediation:- Rightsizing: Schedule periodic rightsize assessments via IaC-driven jobs: run CloudWatch/Cost Explorer or GCP Recommender data -> generate Terraform plan updates or PRs suggesting smaller sizes. Optionally auto‑apply non‑breaking rightsizes for low-risk workloads with approval workflow.- Scheduled shutdowns: Deploy IaC-managed start/stop lambdas/Cloud Functions or instance schedules driven by tags (environment=nonprod && lifecycle=ephemeral) and enforced via policy to prevent manual override.- Spot/Preemptible adoption: Provide Terraform modules that default to spot instances with fallback on capacity-optimized strategies and include lifecycle hooks for graceful termination; policy flags workloads that opt-out.- Autoscaling: Provide opinionated autoscaling modules (target tracking, step scaling) with sane defaults and circuit breakers; enforce via IaC so every stateless service is deployed into an autoscaling group.Monitoring, alerts & reporting:- Centralize telemetry: push billing metrics, rightsizing recommendations, spot interruption events, and schedule compliance into a cost data lake (S3/BigQuery) and a dashboard (QuickSight/Looker/Grafana).- Implement automated alerts: budget thresholds, unexpectedly high hours for nonprod instances, repeated scale‑up patterns that indicate undersizing.- Regular reports: per‑team weekly cost variance, unused/underutilized resources, spot savings, and automated remediation actions taken.Governance & workflow:- GitOps for infra changes; PR templates require cost impact assessment. Use approval gates for exceptions (e.g., prod workloads needing on‑demand).- Cost owners receive monthly statements with chargebacks/showbacks; integrate tagging to attribute costs automatically.KPIs & ROI measurement:- Track: % of compute on spot, average CPU/RAM utilization, monthly nonprod instance-hours saved, number of rightsizing actions, budget variance, and mean time-to-remediate noncompliance.- Calculate ROI: (baseline monthly cost - optimized monthly cost + reclaimed credits)/automation implementation cost. Report payback period and annualized savings per team.Example pragmatic rollout:1. Enforce tags and policy-as-code.2. Deploy scheduled shutdowns for nonprod (tag-based).3. Automate rightsizing reports → pilot auto-apply on low-risk services.4. Convert batch and dev to spot/preemptible with fallback.5. Standardize autoscaling modules for new services.6. Iterate with dashboards and chargeback to drive adoption.This approach combines preventative guardrails, automated corrective actions implemented via IaC, and transparent reporting to control cost while measuring business impact.
MediumTechnical
46 practiced
A customer reports production instances have drifted from the desired Terraform state because of several manual emergency hotfixes. Propose a step-by-step remediation and prevention plan that minimizes downtime: detection, impact assessment, reconciliation path, communication with stakeholders, and automation to prevent recurrence.
Sample Answer
Situation summary: production resources were changed manually for emergency fixes, causing Terraform state drift. Remediation must restore a single source of truth, avoid user-visible downtime, and prevent recurrence.Step-by-step remediation1. Detection & Triage- Run read-only discovery: terraform plan (against current state file) + terraform refresh OR use drift tools (driftctl, tfsec) to enumerate differences.- Snapshot state: terraform state pull → store immutable backup. Export cloud audit logs for recent manual changes.2. Impact assessment- Categorize drift by risk: config-only (tags, metadata), infra-impacting (instance types, security groups), and data-affecting (DB schema, disk changes).- Map drifted resources to owners and dependent services; identify changes that must remain (approved hotfixes) vs. revert.3. Reconciliation path (minimize downtime)- Preferred: reconcile Terraform to match desired (declare desired = source of truth) if hotfixes were mistakes. - Create a safe plan: run terraform plan in a staging replica with plan output files. - For resources that cannot be recreated without downtime, use terraform import to align state with real resources first, then apply minimal diffs. - For destructive changes, perform rolling updates/canaries: use instance replacements with health checks and LB draining to avoid outage.- Alternative: if hotfixes are required to stay, update IaC to incorporate the emergency change, run plan/apply in controlled pipeline.- Always run terraform apply with -parallelism tuned and maintenance windows for risky ops.4. Communication & coordination- Immediately notify stakeholders: incident owner, SRE, product, and customers if SLAs affected. Provide: - What changed, potential impact, planned reconciliation steps, and ETA.- Use runbook template: detection → assessment → action → rollback criteria.- Schedule synchronous window for high-risk changes with paging and rollback contact.5. Automation & prevention- Enforce GitOps: all infra changes via PRs, CI pipeline with terraform fmt, validate, plan, and automated apply with approvals.- Prevent manual changes: revoke broad console rights, require just-in-time elevated access, enable MFA and session recording.- Continuous drift detection: schedule driftctl/terraform plan in CI and alert on drift.- Policy as code: OPA/Sentinel checks in pipeline to block risky changes.- Post-incident: run blameless postmortem, update runbooks, and add automated tests and canary deployments.Key trade-offs- Importing state is low-risk but can leave incorrect IaC; editing IaC preserves IaC-first discipline but may require temporary state alignment and careful testing.- Prioritize availability: prefer non-destructive imports + incremental Terraform updates and canary rollouts over immediate full reconciliation that risks downtime.This plan restores a single source of truth while minimizing customer impact and hardening processes to prevent future drift.
HardTechnical
51 practiced
Design a policy enforcement and remediation system for IaC to block insecure configurations (public S3 buckets, overly permissive IAM). Compare OPA/Gatekeeper, Terraform Sentinel, and Cloud Custodian for placement (pre-commit, CI, runtime), scalability, policy lifecycle management, and how you'd maintain and test a shared policy library across teams.
Sample Answer
Situation / goal: Build an IaC policy enforcement + automated remediation platform to prevent insecure configs (public S3, overly-permissive IAM) across the SDLC and runtime for large multi-team orgs.High-level design:- Multi-layer enforcement: pre-commit hooks + local CLI checks → CI pipeline policy checks (block PRs) → pre-apply gate (terraform plan checks) → runtime drift detection and auto-remediation.- Central Policy Service: single source-of-truth Git repo (policy-as-code), CI for policy linting/tests, policy registry with versioning, RBAC for edits, and a webhook/agent layer to enforce policies in CI and runtime.- Automated remediation: safe remediations via Cloud API (Cloud Custodian) for runtime drift; non-destructive suggestions in CI with remediation playbooks and one-click fixes.Compare technologies (placement / scalability / lifecycle):- OPA + Gatekeeper - Placement: Kubernetes admission (runtime), CI via conftest/opa for pre-apply and pipelines. - Scalability: Lightweight, decoupled; Gatekeeper scales with k8s control plane; OPA bundles for CI are horizontally scalable. - Policy lifecycle: Policies in Rego in Git, strong unit testing with opa test; good for cluster-level controls and fine-grained RBAC. - Best fit: Kubernetes-native enforcement and CI checks; less focused on cloud resource remediation.- Terraform Sentinel - Placement: Embedded in Terraform Cloud/Enterprise; primarily pre-apply and policy checks in plan stage. - Scalability: Scales with Terraform Cloud; good for org-wide Terraform workflows but tied to Terraform ecosystem. - Policy lifecycle: Central policies, versioned, integrated into runs; language is proprietary, testing support exists but less open ecosystem. - Best fit: Organizations standardizing on Terraform Cloud/Enterprise and needing blocking policy at plan/apply.- Cloud Custodian - Placement: Runtime (cloud account) — scheduled or event-driven enforcement; can be used in CI to validate templates via c7n policies. - Scalability: Scales across accounts via serverless runners (Lambda) or containers, multi-account support via automation. - Policy lifecycle: YAML policies in Git, unit/integration tests with custodian run; strong remediation and reporting capabilities. - Best fit: Continuous enforcement and automated remediation across cloud accounts.Operational pattern / recommended hybrid:- CI / Pre-commit: conftest (OPA) + pre-commit hooks to catch obvious issues early.- CI / Plan-stage: Terraform Sentinel for orgs on Terraform Cloud; otherwise use terraform-compliance or OPA policy checks against plan JSON.- Runtime: Cloud Custodian for detection + automated remediation, plus Gatekeeper for k8s workloads.- Policy registry: Git monorepo with modules per policy domain (iam, storage, network), semantic versioning, PR review workflow, policy metadata (severity, remediation playbooks, owners).- Testing & maintenance: - Unit tests: Rego unit tests for OPA, Sentinel tests for Sentinel policies, pytest/integration harness for custodian. - CI pipeline: On PR, run static lint, unit tests, sample infra runs (plan-level emulation), and mutation tests (introduce insecure config to ensure policy triggers). - Staging rollout: Canary policies (monitor-only) in a staging account/team, collect false positives for 2-4 weeks, then promote to deny-mode. - Change control: Policy owners, RFC process, automated changelogs, and impact analysis reports (which resources will be blocked/changed). - Shared library maintenance: Use modular policy templates, strict code review, scheduled policy audits, automated dependency and drift alerts, and cross-team working group to manage exceptions and new rules.Metrics and SLA:- Key metrics: policy coverage, false-positive rate, mean-time-to-remediate, number of blocked PRs, runtime violations per day.- SLOs: e.g., remediation within 1 hour for high severity; policy test pass rate >99% in CI.Trade-offs:- Sentinel ties you to Terraform Cloud but provides strong plan-time blocking; OPA is flexible and cloud-agnostic but needs integration work for multi-cloud remediation; Cloud Custodian excels at remediation but is not a plan-time blocker.This hybrid architecture balances early prevention, centralized policy governance, and automated runtime remediation while enabling scale and cross-team collaboration.
EasyTechnical
58 practiced
Describe the core components and considerations needed to implement autoscaling for a stateless web service in AWS or Azure. Include launch/scale configuration, health checks, metrics to drive scaling decisions, cooldown windows, and SLO/latency trade-offs you would use to set thresholds.
Sample Answer
Requirements/goal: automatically maintain SLOs for a stateless web service (HTTP API) while minimizing cost and blast radius.Core components- Launch/scale configuration: - Immutable image (AMI/Managed Image) or container image + startup script, baked with app and monitoring agent. - AWS: Auto Scaling Group (ASG) + Launch Template/Configuration; attach to ALB and target group. Azure: Virtual Machine Scale Set (VMSS) or AKS with Horizontal Pod Autoscaler. - Instance sizing, bootstrapping, IAM role, VPC/subnet, security groups. - Graceful shutdown hooks and lifecycle hooks to drain connections before termination.- Health checks: - Layered checks: platform (EC2/VM) + load balancer target health + application liveness and readiness endpoints (HTTP 200, dependency checks). - Configure ELB/ALB health check intervals and failure thresholds to avoid flapping.- Metrics to drive scaling: - Primary: request latency (p95/p99), request rate (RPS), and active concurrency. - Secondary: CPU, memory, and error rate (5xx). - Prefer application-level metrics (latency/concurrency) or a derived metric (RPS per instance) for target tracking. - Use aggregated percentiles (p95) rather than averages.- Scaling policies & cooldowns: - Prefer target-tracking policy (e.g., keep average latency <= target or CPU at target utilization). Use step-scaling for bursty patterns. - Cooldown / stabilization: scale-up fast but include short warm-up (60–120s) for instances to register; scale-down conservative with longer cooldown (5–10 minutes) to avoid oscillation. - Use lifecycle hooks to wait for warm-up and health checks before routing traffic.- SLO / latency trade-offs and thresholds: - Define SLO (e.g., 95% requests < 200 ms). Set scaling trigger at a safety margin (e.g., p95 > 0.8 * SLO or p95 > SLO * 1.1) so scaling begins before violation. - For cost-sensitive customers, tolerate higher tail latency (e.g., scale at p95 > 1.2*SLO) but alert sooner. - Use request-rate per healthy instance to compute desired capacity: desired = ceil(RPS_total / target_RPS_per_instance).- Additional considerations: - Warm pools/standby instances to reduce cold-starts; use containers for faster scale. - Autoscaling limit guards and budget controls to avoid runaway cost. - Observability: dashboards, alerts on SLO burn rate, and automated rollback if health worsens post-scale. - Test chaos/failover and scale scenarios under load to validate thresholds.This design balances rapid response to load (scale-up) with conservative scale-down, relies on application-level latency as the primary signal, and uses lifecycle/warm-up strategies to prevent oscillation and protect SLOs.
MediumTechnical
61 practiced
Explain strategies to version and lock remote modules and provider versions in Terraform to ensure stable deployments across teams. Describe how you'd enforce module version constraints in CI, how to run compatibility tests before upgrades, and how to roll out module upgrades across many stacks safely.
Sample Answer
Start by distinguishing provider lockfiles (checksum for binaries) from module version pinning (source & semantic version). Concrete strategies:- Lock providers: - Use required_providers in Terraform blocks and commit .terraform.lock.hcl (Terraform generates it on init). Enforce in CI using terraform init -lockfile=readonly to fail builds if lockfile is out-of-date. - For private providers use a provider mirror or private registry so binaries are stable.- Pin modules: - Prefer registry modules with explicit version = "x.y.z" and use semantic versioning for releases. - For git sources include ?ref=tags/v1.2.3 to pin to an exact tag.- Enforce in CI: - CI pipeline steps: git checkout → terraform init -lockfile=readonly → terraform validate → terraform plan. Fail if init requires lockfile changes or module sources differ. - Add policy checks (Sentinel, OPA/Gatekeeper, or a custom script) that reject ranges like "~> 1" and require exact/minimum versions, or check CHANGELOG presence for upgrades. - Use automated PR bots (Renovate/tfupdate) to propose controlled version bumps; CI runs full plan and test suites on those PRs.- Compatibility testing before upgrade: - Create ephemeral test workspaces/environments (short-lived accounts or isolated prefixes). On a version-bump PR, CI applies the new module/provider there and runs smoke and integration tests (API checks, service health, infra drift). - Run terraform plan against representative stacks to detect breaking changes (e.g., resource replacements). Include negative tests for drift and provider behavior.- Safe rollout across many stacks: - Automate propagation: tooling that opens per-stack PRs with the new module version (Renovate, custom script). Each PR runs CI tests and a plan; require human approval for apply. - Phased rollout: canary a small subset of non-critical stacks, monitor metrics and alerts, then progressively widen. - Use state-aware strategies: if module upgrade requires resource replacement, schedule maintenance windows and use blue/green or parallel resources where possible. - Keep rollback plans: tag previous module versions, keep tested rollback PRs, and document manual revert steps if apply fails.Trade-offs and governance:- Exact pins maximize stability but slow security updates — schedule periodic controlled upgrades.- Rely on automation + policy to balance safety and agility.
Unlock Full Question Bank
Get access to hundreds of Infrastructure Automation and Provisioning interview questions and detailed answers.
