Infrastructure Automation and Provisioning Questions

Covers designing, implementing, and operating automated infrastructure provisioning and configuration using Infrastructure as Code practices and complementary automation patterns. Candidates should be able to select and author declarative infrastructure definitions with tools such as Terraform, CloudFormation, and Azure Resource Manager templates, and discuss configuration management tools such as Ansible, Puppet, or Chef. Core skills include modular and reusable code organization for multiple environments, variable and output management, remote state management and locking, idempotency and atomicity of operations, and version control integration for infrastructure artifacts. Candidates should understand testing and validation practices including linting, plan or dry run validation, unit and integration testing of infrastructure changes, and drift detection and remediation. The topic includes strategies for safe changes and rollbacks, change coordination, error handling and recovery, and deployment patterns such as canary and blue green where applicable. It also encompasses automation and orchestration patterns, immutable infrastructure and self healing practices, autoscaling and scaling policies, automated patching and updates, secrets handling patterns using secret managers, and integrating observability and monitoring into automated workflows. Finally, candidates should be able to reason about trade offs between imperative and declarative approaches, scaling Infrastructure as Code across large projects and teams, and security and compliance considerations for automated provisioning.

HardSystem Design

47 practiced

Compare monorepo and multi-repo strategies for managing Infrastructure as Code at enterprise scale. Discuss module sharing, CI and testing complexity, ownership boundaries, release cadence, discoverability, and how to publish and consume shared modules. Recommend a strategy for an organization with 200 engineers and justify trade-offs.

Sample Answer

**Clarify constraints**200 engineers, enterprise scale, multiple teams, cloud-native (K8s), need for reuse, security/compliance, moderate release velocity.

**Monorepo vs Multi-repo — key comparisons**- Module sharing - Monorepo: easy atomic changes across infra modules; single source of truth; simple refactors. - Multi-repo: requires published versioned packages (registry); stronger API boundaries; safer isolation.- CI & testing complexity - Monorepo: CI must support targeted builds/tests and scale (incremental CI, cache, path-based triggers). Complexity centralized but can be optimized. - Multi-repo: simpler CI per repo but harder to run cross-repo integration tests; drift detection harder.- Ownership boundaries - Monorepo: soft boundaries via CODEOWNERS and directory-level permissions; risk of accidental changes. - Multi-repo: clear ownership, easier access control and compliance auditing.- Release cadence - Monorepo: can coordinate large cross-cutting changes atomically; individual teams still need lightweight branching/release rules. - Multi-repo: teams release independently, faster for isolated components but cross-repo upgrades require coordination.- Discoverability - Monorepo: easy search/discovery of modules and examples. - Multi-repo: requires catalog/registry + docs; otherwise discoverability suffers.- Publish & consume shared modules - Monorepo: consume via local path / git subdirs; use tooling to stitch Terraform/Helm modules; use internal registries for packaged modules. - Multi-repo: publish semantic-versioned artifacts to internal registry (artifact repo, Terraform Registry, Helm repo), consumers pin versions.

**Recommendation for 200 engineers**Hybrid approach: primary monorepo for core platform and shared libraries/modules (networking, common IAM, base Terraform/Helm modules) + team-owned multi-repos for product-specific infra and app stacks. Rationale:- Centralize high-value shared modules for discoverability, easy refactor, and policy enforcement.- Keep team autonomy and clear ownership for application infra to scale controllers and release cadence.- Implement: internal module registry (Terraform/Helm), strict CI patterns (incremental builds, path filters), automated dependency upgrade bots, CODEOWNERS, policy-as-code (OPA/Gatekeeper), and cross-repo integration tests in a dedicated pipeline.

**Trade-offs**- Operational cost: monorepo tooling and CI scale effort vs. governance overhead in multi-repo.- Risk: monorepo increases blast radius — mitigate with granular permissions and automated checks.- Velocity: hybrid preserves autonomy while enabling safe reuse.

I would implement the hybrid in phases: migrate core modules to monorepo, deploy registry, enable CI patterns, then onboard teams.

MediumTechnical

63 practiced

You need to implement autoscaling for a web service on AWS using Terraform. Describe which metrics you would use (CPU, request latency, custom metrics), the differences between target-tracking and step-scaling policies, how to set cooldowns and stabilization windows, and how to safely test scaling behavior in non-prod environments.

Sample Answer

**Approach (role: DevOps Engineer)** I’d pick metrics that reflect user experience and capacity: primary = request latency (p99/p95), secondary = request rate (RPS) and CPU for process-bound apps, plus custom metrics (queue depth, JVM GC, active connections). Use CloudWatch Alarms and Application/Target tracking for autoscaling configured via Terraform.

**Target-tracking vs Step-scaling**- Target-tracking: keep a metric at a target value (e.g., maintain p95 latency <= 200 ms or CPU = 60%). Autoscaling adjusts continuously; simpler and preferred for stable targets.- Step-scaling: reacts to threshold breaches with discrete steps (add 2 instances if RPS > X). Good for predictable, bursty workloads where you want explicit control.

**Cooldowns & Stabilization**- Cooldown (scaling-in): e.g., 300s to avoid flapping after scale-out; scale-out cooldown shorter (60–120s).- Stabilization window (Application Auto Scaling): use 3–5 minutes for latency-based targets; ensures policy waits for metric to stabilize before another adjustment.- Configure evaluation periods and datapoints to alarm to reduce false positives.

**Terraform example (target-tracking ALB target group)**

hcl

resource "aws_appautoscaling_target" "tg" {
  max_capacity       = 10
  min_capacity       = 2
  resource_id        = "service/${aws_ecs_cluster.cluster.name}/${aws_ecs_service.svc.name}"
  scalable_dimension = "ecs:service:DesiredCount"
  service_namespace  = "ecs"
}

resource "aws_appautoscaling_policy" "cpu_target" {
  name               = "cpu-target"
  policy_type        = "TargetTrackingScaling"
  resource_id        = aws_appautoscaling_target.tg.resource_id
  scalable_dimension = aws_appautoscaling_target.tg.scalable_dimension
  service_namespace  = aws_appautoscaling_target.tg.service_namespace

  target_tracking_scaling_policy_configuration {
    predefined_metric_specification {
      predefined_metric_type = "ECSServiceAverageCPUUtilization"
    }
    target_value       = 60.0
    scale_in_cooldown  = 300
    scale_out_cooldown = 120
  }
}

**Safe testing in non-prod**- Use separate non-prod stacks with identical autoscaling configs but smaller max_capacity and synthetic traffic.- Run chaos tests: ramp traffic gradually and spike tests; observe scale decisions and metrics.- Use canary or blue/green to validate behavior without impacting users.- Inject metric anomalies (CloudWatch PutMetricData) to simulate custom-metric triggers and verify alarms, cooldowns, and rollback behavior.

This balances UX-focused metrics, predictable control, and safe validation using Terraform-managed infra.

EasyTechnical

80 practiced

Describe recommended Git workflows for Infrastructure as Code in a collaborative team: branching model (feature branches, protected main for prod), pull request checks, gating with CI plan/dry-run, and how to handle secrets/state outside VCS. Mention policies you would enforce to reduce risky merges to production.

HardSystem Design

52 practiced

Design an automated patching and update strategy for thousands of cloud instances using IaC. Compare image-baking (immutable images) versus in-place patching, define maintenance windows and canary rollout strategy, integration with vulnerability scanners, health checks, and rollback procedures. Explain how to minimize disruption for stateful workloads.

Sample Answer

**Approach summary**I’d implement IaC-driven automated patching with a default immutable image-baking pipeline plus a limited in-place patching option for special stateful cases. Use Terraform/CloudFormation for infra, Packer for images, and CI (Jenkins/GitHub Actions) to orchestrate.

**Image-baking (immutable) vs in-place**- Immutable images (preferred): Bake OS + app + patches into AMI/container image. Pros: reproducible, fast rollback (switch to previous image), safer for scale. Cons: longer build time, needs graceful replacement for stateful services.- In-place patching: Use configuration management (Ansible) or OS package manager. Pros: smaller updates, quicker for urgent fixes. Cons: drift, harder rollback, more risk.

**Maintenance windows & canary rollout**- Define maintenance windows in SLO-aware slots (e.g., 02:00–05:00 local) with Ops calendar and automated gating.- Canary strategy: 1) Deploy new image to 1–2% of fleet (or single AZ) 2) Run automated integration/SMoke tests and health checks for X minutes 3) If pass, incrementally roll out (5% -> 25% -> 50% -> 100%) with automated gating and rollback thresholds.

**Vulnerability scanning & gating**- Integrate SCA/OSCA: run Trivy/Clair/Anchore on images during CI; run scanner agents (Qualys/Tenable) on running instances nightly.- Block promotion if high/critical CVEs without approved compensating control.

**Health checks & rollback**- Health checks: service probe, latency/SLO, error-rate, dependency tests, readiness/liveness endpoints.- Rollback: automated traffic switch to previous image via autoscaling group/ASG blue/green or Kubernetes rollout undo; if in-place, run automated rollback playbook or restore from snapshot.

**Minimizing disruption for stateful workloads**- Use leader-election, sticky sessions, quiesce writes, DB replicas.- For stateful nodes: prefer rolling replace with draining (connection drain, rsync/replication), use workload-aware orchestration (K8s StatefulSets with podDisruptionBudgets), take consistent filesystem/db snapshots, or migrate to managed stateful services to avoid patching individual hosts.

I’d codify all steps in IaC and CI pipelines, add observability alerts for failed gates, and run periodic chaos tests to validate rollback.

EasyTechnical

60 practiced

Explain the purpose of remote state in Terraform. Describe common backend options (S3 with DynamoDB locking, GCS, Azure Storage, Terraform Cloud) and explain why locking, state encryption, and access control are important for team-based workflows and disaster recovery.

Unlock Full Question Bank

Get access to hundreds of Infrastructure Automation and Provisioning interview questions and detailed answers.

Join thousands of developers preparing for their dream job.