Infrastructure Automation and Provisioning Questions

Covers designing, implementing, and operating automated infrastructure provisioning and configuration using Infrastructure as Code practices and complementary automation patterns. Candidates should be able to select and author declarative infrastructure definitions with tools such as Terraform, CloudFormation, and Azure Resource Manager templates, and discuss configuration management tools such as Ansible, Puppet, or Chef. Core skills include modular and reusable code organization for multiple environments, variable and output management, remote state management and locking, idempotency and atomicity of operations, and version control integration for infrastructure artifacts. Candidates should understand testing and validation practices including linting, plan or dry run validation, unit and integration testing of infrastructure changes, and drift detection and remediation. The topic includes strategies for safe changes and rollbacks, change coordination, error handling and recovery, and deployment patterns such as canary and blue green where applicable. It also encompasses automation and orchestration patterns, immutable infrastructure and self healing practices, autoscaling and scaling policies, automated patching and updates, secrets handling patterns using secret managers, and integrating observability and monitoring into automated workflows. Finally, candidates should be able to reason about trade offs between imperative and declarative approaches, scaling Infrastructure as Code across large projects and teams, and security and compliance considerations for automated provisioning.

HardTechnical

64 practiced

At scale, autoscaling policies can oscillate or fail to meet SLOs. Describe how you would design, test, and validate autoscaling policies for a microservices fleet. Include load testing approaches, synthetic traffic generation, warm-up and cooldown considerations, metric selection, and how you would surface misbehaving scaling actions in observability tooling.

Sample Answer

**High-level approach (goal):** I’d design autoscaling so it meets SLOs with minimal oscillation and cost. That means predictable, resilient policies validated by realistic testing and tight observability that ties scaling actions to user-impact.

**Design principles**- Use layered scaling: fast-reacting metrics (request queue / concurrency) + stable metrics (p95/p99 latency, error rate, CPU) and business signals (orders/sec).- Prefer step/predictive scaling for known patterns, target tracking (e.g., CPU/concurrency) for steady load, and add cooldown/stabilization windows to prevent oscillation.- Support warm pools / provisioned concurrency for latency-sensitive services.

**Metric selection**- Primary: p95 and p99 latency, error rate, saturation (queue length, connection count).- Secondary: CPU, memory, instance startup time, custom throughput (transactions/sec).- Use composite rules: only scale down if latency and error rate are below thresholds for N minutes AND CPU low.

**Warm-up and cooldown**- Warm-up: pre-warm containers/instances or use lifecycle hooks to mark readiness only after health and JVM/SQL caches warmed.- Cooldown/stabilization: enforce minimum time between scale events (e.g., 5–10m) and require sustained metric violation before scaling; use hysteresis (different thresholds for scale-up vs scale-down).

**Testing & validation**- Staging with production-like dataset and network characteristics; use canary rollouts for policy changes.- Load tests: gradual ramp, spike, sawtooth (oscillation-inducing), and soak tests. Tools: k6, JMeter, Locust, or cloud load generators (GCP PerfKit, AWS Distributed Load Testing).- Synthetic traffic: replay production traces or generate realistic traffic mixes (thinktime, user journeys, background jobs); include error injection and dependency latency simulation.- Chaos & failure modes: simulate instance startup failure, ASG quota limits, noisy neighbor, degraded downstreams to observe scaling behavior and SLO impact.- Dry-run & simulation: run policy simulators on historical telemetry to predict action frequency and cost.

**Observability & surfacing misbehavior**- Dashboards correlating scaling events with latency/error/throughput and instance counts; include timelines of scaling API calls.- Audit trail: log all scaling decisions (policy, metric value, decision reason) to a searchable store (Elasticsearch/CloudWatch Logs/Stackdriver).- Alerting: behavioral alerts for - failed scale attempts, - frequent oscillations (N scale events in M minutes), - sustained SLO violation despite scaling.- Automated anomaly detection: use rate-of-change or ML-based anomaly detectors to flag abnormal scaling patterns.- Tracing: link requests to backend instances to show if new instances improved latency.- Runbooks & automated mitigation: auto-roll back policy, increase stabilization window, or trigger manual review with pre-populated diagnostics.

**Validation metrics**- SLO compliance under test scenarios, time-to-recover, false-positive scale events, cost per request.- Iterate: tune thresholds, cooldowns, and warm pools based on incidents and postmortems.

This combination ensures policies are safe, observable, and validated against realistic loads so scaling helps — not hurts — SLOs.

MediumSystem Design

59 practiced

Design a CI/CD pipeline for Infrastructure-as-Code stored in Git. The pipeline must include linting, plan/dry-run, automated tests (unit and integration), PR-based review with human approval for production, secure secrets handling, and an auditable apply step. Describe pipeline stages, tooling choices, branching strategy, and how to support deployments to multiple cloud accounts or subscriptions.

Sample Answer

**Overview & goals**Design CI/CD for IaC (Terraform) in Git that runs linting, plan/dry-run, unit/integration tests, PR review with human prod approval, secure secrets, auditable apply, and multi-account deployment.

**Branching strategy**- Trunk-based: main (production), develop (staging), feature/* for work.- PRs from feature -> develop; release PR from develop -> main requires manual prod approval.

**Pipeline stages**1. Pre-commit / PR checks (fast) - terraform fmt, terraform validate, tflint, checkov (security)2. Plan & Unit Tests (CI runner) - Run `terraform plan -out=plan.tfplan` against a dry backend (or remote state snapshot) - Unit tests with mocks: terratest Go unit tests or kitchen-terraform for resources3. Integration Tests (isolated env) - Create disposable environment in test account subscription - Apply, run integration tests (terratest), then destroy4. Plan Review (artifact) - Store signed plan artifact and diffs in pipeline; post plan to PR for reviewer review5. Manual Approval (for main) - Require human approval, with checklist and link to plan artifact6. Auditable Apply - Apply executed by a CI service account with MFA/just-in-time approval; record logs, plan artifact, run-id in centralized audit store (CloudTrail / Azure Activity Logs + pipeline logs)7. Post-deploy validation & drift detection - Automated smoke tests, periodic drift scans

**Tooling choices**- IaC: Terraform + optional Terragrunt- Lint/security: terraform fmt/validate, tflint, checkov, tfsec- Tests: terratest (Go) + kitchen-terraform or pytest-terraform- CI: GitHub Actions / GitLab CI / Jenkins X. Use self-hosted runners with cloud IAM roles.- Secrets: Vault (HashiCorp) or cloud KMS/Secrets Manager; CI reads secrets via short-lived tokens (OIDC) — avoid static secrets in Git.- Remote state: Terraform Cloud/Enterprise or S3+DynamoDB (AWS) / Azure Storage with state locking- Audit: Terraform Cloud runs or centralized logging (CloudTrail / Azure Monitor) + immutable plan artifacts in artifact store (S3/GCS)

**Multi-account deployment**- Use separate workspaces per account/region or use backend config per workspace.- Pattern: modules in repo; per-account environment folders with provider configs using assume-role (AWS STS) or service principals (Azure) via OIDC.- CI obtains short-lived credentials by OIDC federated identity to assume the target account and execute apply.- Parameterize via variables/TF_VAR files stored encrypted in Vault per account.

**Security & compliance**- Enforce branch protection, required checks, signed commits for release PRs.- Secrets never in Git; use OIDC or Vault; encrypt state at rest and restrict access.- Least-privilege CI service accounts; require MFA for approval step.- Retain plan artifacts, pipeline logs, and cloud audit logs for N days for compliance.

**Why this works**- Fast feedback on PRs, safe test playgrounds, human gate for prod, auditable artifacts, secure secrets, scalable multi-account support through workspaces and short-lived credentials.

EasyTechnical

66 practiced

In Terraform HCL, write a minimal, well-structured module that creates an AWS S3 bucket with versioning enabled and server-side encryption using an AWS-managed KMS key. Use input variables for bucket name and tags, and expose the bucket ARN as an output. Do not include provider configuration; focus on module inputs/outputs and resources.

Sample Answer

**Approach**- Minimal, reusable Terraform module that: creates an S3 bucket, enables versioning, creates an AWS-managed KMS key for server-side encryption, and applies bucket encryption and tags. Exposes bucket ARN. Inputs: bucket name and tags.

**Code (module files)**

hcl

// variables.tf
variable "bucket_name" {
  description = "Name of the S3 bucket"
  type        = string
}

variable "tags" {
  description = "Map of tags to apply to resources"
  type        = map(string)
  default     = {}
}

hcl

// main.tf
resource "aws_kms_key" "s3" {
  description             = "KMS key for S3 server-side encryption"
  deletion_window_in_days = 30
  tags                    = var.tags
}

resource "aws_s3_bucket" "this" {
  bucket = var.bucket_name
  tags   = var.tags
}

resource "aws_s3_bucket_versioning" "this" {
  bucket = aws_s3_bucket.this.id

  versioning_configuration {
    status = "Enabled"
  }
}

resource "aws_s3_bucket_server_side_encryption_configuration" "this" {
  bucket = aws_s3_bucket.this.id

  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm     = "aws:kms"
      kms_master_key_id = aws_kms_key.s3.arn
    }
  }
}

hcl

// outputs.tf
output "bucket_arn" {
  description = "ARN of the created S3 bucket"
  value       = aws_s3_bucket.this.arn
}

**Notes / Best practices**- Module avoids provider config; caller supplies provider and region.- Consider adding lifecycle rules, public access block, or optional KMS key reuse via an input for production.- Handle name collisions: add random/id suffix if you want non-global-unique names.

HardSystem Design

66 practiced

Design an immutable infrastructure pipeline using Packer, a golden image lifecycle, and automated instance refresh for a fleet behind a load balancer. Describe how to build and test images, promote images across environments, perform instance replacement with health checks and draining, and ensure minimal disruption during OS and security patch rollouts.

Sample Answer

**Overview & goals**Build an immutable pipeline that bakes golden images with Packer, promotes tested images across dev→staging→prod, and performs automated, low-disruption instance refreshes behind a load balancer with health checks, draining, and rollback.

**Image build & test**- CI triggers (GitLab/Jenkins) on infra/app changes → run Packer to create AMI/VM image. Packer steps: - Provision base OS, security hardening (CIS/benchmarks), app artifacts, secrets via temporary vault tokens. - Bake automated tests (unit, smoke, config validation).- Post-bake tests: - Functional smoke VM: launch instance in isolated VPC, run integration tests, run vulnerability scans (Aqua/Trivy). - On pass, register image in artifact registry (e.g., AWS AMI shared account) with metadata: git sha, build id, SBOM, CVE baseline.

**Promotion workflow**- Promote via immutable tags/permissions: dev→staging→prod controlled by CI/CD pipeline & approvals.- Use semantic image tags and manifest in S3 or image registry. Automated gating: regression tests in staging, canary deploy in prod.

**Instance replacement & LB integration**- Use Auto Scaling Groups (ASG) or instance groups with Launch Template referencing image id.- Perform rolling replace using ASG Instance Refresh or custom process: - Set min healthy percentage (e.g., 90%) and batch size. - For each instance: - Deregister from Load Balancer (Target Group) to trigger connection draining. - Wait for in-flight requests to drain and for LB health check to fail. - Terminate instance → ASG launches new instance with new image. - New instance passes health checks (app + ELB) before traffic routed. - Monitor metrics (5xx, latency) and abort/rollback if thresholds exceeded.- For zero-downtime, prefer blue-green / canary: - Launch small fleet with new image; shift small % traffic via weighted ALB rules; monitor; increase if healthy.

**Patching & security updates**- Prefer building new golden images with OS/security patches instead of in-place patching.- Schedule image-builder pipelines (daily/weekly) for critical patches. For emergency patches, trigger high-priority bake + canary.- For long-lived stateful workloads, use strategies: - Draining + session affinity awareness. - Database replicas promotion where needed.- Maintain a tested rollback image and automation to revert ASG Launch Template to previous image id.

**Observability & safety**- Pre-flight: automated pre-deploy canaries, chaos testing in staging.- During rollout: alerting on health checks, request errors, latency; automated rollback when thresholds breach.- Post-deploy: automated compliance scans, SBOM validation, and tag retention policy for audit.

**Trade-offs**- Baking images increases build time but ensures reproducible, hardened instances and faster boot time.- Instance refresh via ASG simplifies orchestration; blue-green gives safer cutover at cost of extra capacity.

This pipeline enforces immutability, automated promotion, safe replacement with LB draining/health checks, and minimizes disruption for OS/security patch rollouts while maintaining fast rollback and auditability.

HardSystem Design

45 practiced

Describe how you would integrate observability into automated provisioning workflows so that every infrastructure change emits structured telemetry. Specify which events, metrics, and logs you would capture (plan results, apply durations, drift incidents, resource lifecycle events), how to correlate them with CI runs and change requests, and how to create alerts and dashboards that surface risky changes or regressions.

Sample Answer

**Clarify requirements & goals**

Instrument automated provisioning (IaC runs) so every change emits structured telemetry usable for auditing, alerting, and dashboards; correlate with CI job IDs, MR/CR numbers, and user identity; detect risky changes, regressions, and drift.

**High-level design**

- Integrate telemetry emitter into the provisioning pipeline (pre-plan hook, post-plan hook, apply hook, drift scanner).- Send structured events to a telemetry backend (OpenTelemetry collector → vendor backend like Datadog/CloudWatch/Prometheus + Loki/ELK).- Enrich events with CI metadata and change-request identifiers.

**Events, metrics, logs to capture**

- Events (structured JSON) - plan_result: plan_id, outcome (changes: add/modify/destroy counts), diff_summary, run_id, repo, commit, MR/CR_id, user, timestamp - apply_result: apply_id, duration_ms, outcome, resources_changed[], run_id, approver_id - resource_lifecycle: resource_id, type, action (create/update/destroy), timestamp, before/after hashes - drift_incident: resource_id, observed_state_hash, desired_state_hash, severity, last_apply_id- Metrics (time-series) - provisioning.duration (labels: environment, resource_type, run_id) - plan.change_count (adds, mods, deletes) - drift.count by environment/resource_type - failed_applies.count, flaky_apply.rate- Logs - Full plan diff (access-controlled), step-by-step apply logs, error traces, API call latency

**Correlation strategy**

- Propagate identifiers: run_id (UUID), ci_job_id, commit_sha, cr_id, user; attach as tags/attributes on every event/metric/log.- Use trace_id for long-running applies and drift scans; link traces to CI runs via trace metadata.- Store canonical relationships in a change-index service (small DB mapping run_id → cr_id → resources touched).

**Alerts & dashboards**

- Alerts: - High-severity: apply failed for production with destroy_count > 0 OR apply.duration > threshold for critical resources - Drift alert: drift.count by critical resource_type > 0 for > X minutes - Risk regression: sudden increase in plan.change_count or destroy_rate vs baseline (anomaly detection) - Flaky applies: repeated failures for same resource within N runs - Alert channels: PagerDuty for P0, Slack with run link for ops, ticket auto-create with context- Dashboards: - Change overview: rate of plans/applies, success rate, average durations, change heatmap by service - Resource risk: top resources by drift frequency, top crashing resource types - CI correlation: list of recent runs, linked MR/CR, diff summaries, expand to apply logs and plan artifacts - Trend & baselining: rolling baseline of change counts per service; anomaly markers

**Implementation notes & trade-offs**

- Store heavy artifacts (full diffs) in object store with signed links; keep lightweight summaries in telemetry.- Use sampling for verbose apply logs but ensure retention when errors occur.- Ensure RBAC and encryption for telemetry (sensitive fields masked).- Start with key resources and prod envs, iterate to reduce noise (tune alerts, SLAs).

This approach gives end-to-end visibility: plan → apply → runtime drift, correlated to CI/CR context, enabling fast detection of risky changes and continuous improvement.

Unlock Full Question Bank

Get access to hundreds of Infrastructure Automation and Provisioning interview questions and detailed answers.

Join thousands of developers preparing for their dream job.