Configuration Management and Desired State Questions

Configuration Management and Desired State focuses on ensuring systems remain in a known, consistent state across fleets and environments. Candidates should understand desired state configuration, idempotent operations, differences between declarative and imperative approaches, templating and variable management, configuration versioning, change control, validation and rollback strategies, drift detection and automated remediation, handling environment specific configuration, and the interaction between configuration management and operational concerns such as autoscaling. This topic also covers managing test environment configurations and using platform specific configuration systems such as Group Policy to apply and troubleshoot system settings at scale.

MediumTechnical

66 practiced

Design an automated remediation framework that can repair common configuration drift (e.g., package versions, missing services) but avoids interfering with intentional manual changes. Explain how you would handle locking, record change provenance, implement safe backoffs, and enable human approvals for risky remediations.

Sample Answer

Approach: build a policy-driven remediation framework that separates detection, decision, and execution with safety gates for manual intent. Core components: detectors (monitoring/config scans), policy engine (rules + risk scoring), orchestrator (locks, execution, provenance), approval workflow, and audit store.

Locking:- Use distributed leases keyed by resource (e.g., hostname/service ID) via etcd/Consul/Redis with TTL.- Acquire optimistic lock before any remediation; fail-fast if lock held.- Provide a human “override” flag with higher-priority lock and explicit audit reason.

Record change provenance:- Every planned and executed action is recorded in an immutable audit store (append-only DB or object store) with fields: detector-id, rule-id, before/after config snapshot (diff), timestamp, actor (automation or user), run-id, and confidence score.- Sign or hash snapshots to detect tampering and support rollback.

Safe backoffs:- Implement exponential backoff with jitter for retries; cap attempts per resource per window.- Use canary-first execution: test remediation on one low-risk node, validate health checks and metrics, then rollout gradually (10%, 50%, 100%).- If post-remediation health-checks fail or error budget spike detected, auto-rollback and escalate.

Human approvals for risky remediations:- Classify rules by risk (low/medium/high) via policy metadata. High-risk actions (service restarts, major version changes) require synchronous approval in the workflow UI/Slack with TTLed approval tokens.- Provide contextual data: diff, risk score, recent changes, and automated test results. Allow “dry-run” which shows proposed state change without applying.- Maintain approval audit trail and require post-approval reconciliation to ensure manual changes weren’t made after approval.

Example workflow:1. Detector flags drift → policy engine computes remediation plan + risk.2. Orchestrator attempts lock; if acquired, records planned change.3. If risk high → pause and send approval request; else run canary.4. Execute remediation, run health checks, record outcome; if failure, rollback and backoff.

Safeguards:- Respect manual annotations: resources tagged “no-auto-remediate” or annotated with last-manual-change timestamp; if recent manual change detected, skip and notify.- Periodic reconciliation reports to owners listing skipped items and suggestions.

This design balances automated reliability with human intent, traceability, and safe rollouts.

HardTechnical

68 practiced

You need to serve templated configuration to millions of ephemeral instances per hour. Describe optimizations to minimize latency and CPU cost: precompilation, caching compiled templates, sharding, parallel rendering, low-latency data access, and safe invalidation strategies for updated templates.

HardTechnical

53 practiced

Provide HCL pseudocode and design notes for a Terraform module that manages an autoscaling group but allows an external autoscaler to control desired_capacity. The module must remain idempotent and avoid races with the external autoscaler. Explain use of lifecycle ignore_changes, remote state locking, and ownership signals.

Sample Answer

Approach- Keep Terraform responsible for ASG creation, lifecycle, launch template, scaling policies, tags, health checks, and bounds (min/max). Let an external autoscaler (K8s HPA, custom controller) be the single source of truth for desired_capacity.- Prevent Terraform from clobbering desired_capacity and avoid races by: (1) lifecycle ignore_changes for desired_capacity, (2) using an ownership/lease signal to coordinate who can set desired_capacity, (3) using remote state locking to prevent concurrent applies.

HCL pseudocode (concise, illustrative)

hcl

terraform {
  backend "s3" {
    bucket = "tf-state-prod"
    key    = "asg/module.tfstate"
    region = "us-east-1"
    dynamodb_table = "tf-state-lock" # remote state locking
    encrypt = true
  }
}

variable "asg_name" {}
variable "min_size" { default = 1 }
variable "max_size" { default = 10 }
# We DO NOT accept desired_capacity as a managed input

resource "aws_launch_template" "lt" {
  name_prefix = "${var.asg_name}-lt-"
  # ... launch configuration ...
}

resource "aws_autoscaling_group" "asg" {
  name                      = var.asg_name
  launch_template {
    id      = aws_launch_template.lt.id
    version = "$Latest"
  }
  min_size = var.min_size
  max_size = var.max_size

  # Important: ignore external autoscaler's desired_capacity changes
  lifecycle {
    ignore_changes = [
      desired_capacity
    ]
    prevent_destroy = false
  }

  tag {
    key                 = "managed_by"
    value               = "terraform"
    propagate_at_launch = true
  }

  tag {
    key                 = "desired_owner"
    value               = "external-autoscaler"
    propagate_at_launch = true
  }
}

# Optional: a lightweight ownership/lease resource (SSM parameter or DynamoDB item)
resource "aws_ssm_parameter" "asg_lease" {
  name  = "/leases/${var.asg_name}"
  type  = "String"
  value = jsonencode({
    owner      = "terraform"
    timestamp  = timestamp()
    expires_at = timeadd(timestamp(), "1h")
  })
  lifecycle { prevent_destroy = false }
}

Design notes / reasoning- lifecycle.ignore_changes(desired_capacity): ensures Terraform will not attempt to write desired_capacity when it detects a different value set by the external autoscaler. This keeps Terraform idempotent: it will reconcile all managed attributes while ignoring externally-controlled field.- Managing min_size/max_size: keep bounds in Terraform so you can enforce safe limits. External autoscaler modifies desired_capacity but cannot exceed configured min/max if you want that guarantee — or you can allow autoscaler to manage bounds too, in which case add them to ignore_changes as well.- Ownership signals: use tags (managed_by, desired_owner) and/or a small lease resource (SSM parameter or DynamoDB item) to advertise intent. The external autoscaler should respect these signals; when temporarily wanting to hand control back to Terraform, it can clear or update the lease.- Lease pattern: terraform creates/updates a lease item with owner="terraform". An external autoscaler can check lease and yield control by setting owner="external-autoscaler" or eviction flag. Alternatively, the external autoscaler can acquire a short-lived lease before making changes; Terraform should read that lease (data source) and error or pause apply if lease indicates external ownership.- Remote state locking: configure S3 backend with DynamoDB table as lock. This prevents concurrent terraform applies which might race with each other. It does not prevent the external autoscaler from changing desired_capacity, so combine with ignore_changes and ownership signals.- Avoiding races: do not make Terraform read-modify-write desired_capacity. Avoid data-dependent updates to desired_capacity. If Terraform must set desired_capacity on first create, set desired_capacity only during create (use lifecycle create_before_destroy or a local-exec to initialize), then add ignore_changes so subsequent runs don't override external changes.- Reconciliation and drift: run periodic scans to detect drift in ignored fields; if you need to reconcile (e.g., emergency restore), implement an explicit workflow: external autoscaler yields lease, operator runs terraform apply to set capacity, then returns lease to autoscaler.- Edge cases: - Initial create: ASG may need a sensible initial desired_capacity; set it in resource creation, then immediately mark desired_capacity in ignore_changes so external autoscaler can take over. - Deletion: if external autoscaler prevents safe teardown, ensure lifecycle.prevent_destroy and clear ownership signals during destroy flows. - Concurrent external writers: prefer external autoscaler to be single-writer; use its own locking (leader election) or a lease in DynamoDB/SSM to coordinate.

Operational practices- Document ownership in README and enforce via CI checks: e.g., terraform plan fails if lease indicates external ownership.- Monitoring: emit metrics when lease ownership changes; alert on unexpected desired_capacity divergence from autoscaler intent.- Testing: simulate external autoscaler writes in staging and verify terraform respects ignore_changes and lease semantics.

This pattern keeps Terraform authoritative for infrastructure configuration and safe bounds, while allowing an external autoscaler to control runtime scaling without Terraform fighting it.

HardSystem Design

70 practiced

A kernel parameter and node OS setting must be changed across a Kubernetes node pool. Nodes host both stateless and stateful workloads. Design a rolling update strategy that applies the node-level change with zero-to-minimal disruption, addresses scheduling capacity, and ensures safe rollback if performance regresses.

Sample Answer

Requirements & constraints:- Change a kernel parameter + OS setting that requires node reboot.- Mixed stateless and stateful workloads; minimal disruption.- Must control scheduling capacity and permit safe rollback if perf regresses.

High-level strategy: do a canary-by-node-pool rolling replacement (create new nodes with desired OS/kernel config), validate, then progressively replace old nodes in controlled batches while using autoscaling, taints, PDBs, monitoring, and automated rollback.

Steps:1. Preparation- Define acceptance metrics (latency, error-rate, IO, CPU, pod startup time) and baseline from current monitoring.- Create a new node template/image (or MachineImage) with the kernel/OS change. Bake and smoke-test offline.- Ensure Cluster Autoscaler & cloud autoscaling quotas available.

2. Canary pool- Create a small canary node pool (1–3 nodes) with the new image; label e.g. node-role.kubernetes.io/canary=true and taint it (canary=true:NoSchedule).- Deploy canary workloads: a subset of stateless pods and one instance from stateful sets if possible (use volumeReplica or a read-only replica). For stateful, prefer redirecting non-critical replica or use snapshot test environment.

3. Validate- Remove taint for scheduled canary apps or use specific tolerations.- Run automated e2e & load tests; monitor defined metrics for a rolling window (e.g., 30–60m).- If regression -> tear down canary pool (scale down) and abort.

4. Rolling replacement- Set batch size based on capacity (e.g., 10% nodes or N nodes). Pre-scale: ensure cluster has spare capacity — temporarily increase autoscaler max nodes or provision extra temporary nodes.- For each batch: a. Cordon target old nodes. b. Evict/drain pods respecting PodDisruptionBudgets (PDBs) and StatefulSet update strategy. For StatefulSets, use partitioned rolling updates: move replicas that can be rescheduled safely; avoid moving primary DB masters unless leader handoff performed. c. Wait for pods to be ready on new nodes and for health checks to pass. d. Monitor performance for a short quorum period. e. If healthy, delete old node (or scale down old node pool) to let autoscaler maintain capacity.

5. Safety & rollback- Automated rollback policy: if any key metric crosses threshold during a batch, stop the rollout immediately, scale up old node pool (or re-create nodes from previous image), cordon new nodes, and re-schedule traffic to old nodes. Use rollout controller (Argo Rollouts-like) or custom operator to automate detection + rollback orchestration.- Keep old node images around to speed rollback. Prefer replacing nodes rather than in-place upgrades so rollback is node-pool level (scale down new pool, scale up old pool).

6. Edge cases & specifics- Stateful workloads: prefer leader-stepdown (DB), volume migration checks (CSI), or update replicas one at a time with readiness probes. If volumes cannot be moved, run updates in-place with careful drain timing and maintenance windows.- Low-capacity clusters: create temporary capacity (spot/ondemand) to avoid violating PDBs.- Use taints/tolerations to prevent scheduling until node is fully ready.

Observability & automation:- Pre-built dashboards and alerts for regressions; automated Canary analyzer (SLO-based) to decide pass/fail.- CI/CD pipeline to create node images, trigger canary, and drive rolling replacement.- Runbook & playbooks for manual rollback, and run a simulation in staging beforehand.

Trade-offs:- Replacing nodes (blue/green node pools) increases resource usage temporarily but maximizes safety and fast rollback.- In-place edits are faster but risk longer recovery and complex rollback.

This approach ensures minimal disruption by validating on canaries, controlling scheduling via taints/PDBs/partitioned updates, using autoscaling to preserve capacity, and providing automated + manual rollback paths if performance regresses.

MediumTechnical

69 practiced

Design a process to deploy and validate Windows Group Policy Objects (GPOs) to 10,000 enterprise desktops. Include staging strategies, pilot rollouts, methods for troubleshooting conflicting policies (RSoP/GPResult), automated validation, and audit trails to prove compliance.

Sample Answer

Overview: treat GPO rollout like any large-scale release — automated, staged, observable, reversible, auditable.

1) Requirements & packaging- Define intent per GPO (security, config), impact window, rollback plan and acceptance criteria (key settings, performance SLOs).- Store GPOs in source control (ADMX templates + change docs) and generate a versioned deployment package.

2) Staging strategy- Dev OU: test templates on lab VMs.- Pre-prod OU: mirror subset of desktop images and AD structure (~0.5% of fleet).- Pilot OUs: phased cohorts (1%, 5%, 25%, 100%) across geos and business units.

3) Automated rollout & validation- Use PowerShell DSC or Group Policy cmdlets + automation pipeline (Azure DevOps/Ansible) to apply GPO links incrementally.- After apply, run parallel validation jobs: - gpresult /h report.html and Get-GPResultantSetOfPolicy for samples - Invoke-Command to run scripts collecting registry keys, services, and event log entries - Endpoint health checks (login time, app launch latency) vs baseline- Gate promotions on automated test pass and KPI thresholds.

4) Troubleshooting conflicts- Automate RSoP collection (gpresult /r and /v) and centralize in ELK/Splunk.- Use Group Policy Modeling (GPMC) to simulate policies and identify precedence, inheritance, and Deny flags.- Correlate with Event Viewer (GroupPolicy operational logs) and SYSVOL replication status (DFSR/NTFRS) checks.- Apply temporary GPO enforcement or WMI filtering to isolate issues and implement rollbacks by unlinking or disabling problematic GPOs.

5) Observability & audit trails- Centralize logs: gpupdate/gpresult outputs, GPO change events (4662), SYSVOL changes, and deployment pipeline logs into SIEM.- Maintain immutable audit: signed change records in source control, deployment runbooks, and timestamped attestation stored in compliance DB.- Generate reporting: per-OU compliance % (desired vs actual), failed nodes list, and remediation tickets.

6) Rollback & remediation- Fast rollback via pipeline to previous GPO version + unlink/disable; run validation tests.- For stuck clients: force gpupdate /force, check network/DNS, verify AD replication and client time sync, then remediate.

This approach ensures safe, observable, repeatable GPO deployment at 10k scale with automated validation and auditable evidence for compliance.

Unlock Full Question Bank

Get access to hundreds of Configuration Management and Desired State interview questions and detailed answers.

Join thousands of developers preparing for their dream job.