Infrastructure as Code and GitOps Questions
Defining and managing infrastructure and delivery state declaratively: provisioning as code (Terraform, CloudFormation, Pulumi, Ansible, Puppet), configuration management, idempotency, drift detection and remediation, and version control for infrastructure definitions, extended by GitOps where git is the source of truth for deployment and infrastructure state. Covers keeping environments consistent, treating config as a first-class versioned artifact, pull-based deployment and continuous reconciliation toward the committed state (ArgoCD, Flux, and similar controllers), Kubernetes manifest and configuration delivery via git, secrets handling for IaC and GitOps pipelines, policy-as-code guardrails (OPA, Sentinel), Terraform state management and locking, and auditable change through version control: branching strategy, pull request review, commit conventions, and code review policy for infrastructure code. Distinct from the CI/CD pipeline design topic, which owns generic pipeline structure and platform-scale release orchestration (build, test, artifact publishing, runner mechanics) and the architectural choice between push-based CI/CD and pull-based GitOps, even when the payload is infrastructure code. Distinct from the safe deployment and rollback strategies topic, which owns deployment-strategy mechanics: canary and blue-green traffic shifting, automated rollback triggered by metrics or SLOs, feature-flag progressive delivery, Kubernetes rollout mechanics (maxSurge, maxUnavailable, health-check gating), and database or schema migration safety as it gates a release, even when the delivery mechanism is GitOps. Distinct from the automation and scripting topic, which owns operational-scripting disciplines (retry and backoff logic, CLI tool design, generic file, checksum, or diff utilities) when the task is not specifically about declarative infrastructure or configuration state. This topic keeps the GitOps reconciliation loop itself, drift detection and remediation, and IaC state and module lifecycle management regardless of which adjacent discipline a question also touches.
Design a process and tooling to detect and reconcile configuration drift for critical infrastructure components (for example network ACLs or firewall rules). Explain how drift alerts are surfaced in Git workflows, when to auto-reconcile versus notify humans, and how to avoid fighting legitimate manual changes.
Sample Answer
Direct answer
For critical, security-relevant infrastructure (network access-control lists, firewall rules), drift detection and reconciliation needs a MORE conservative default than ordinary application config: detect continuously and alert always, but auto-reconcile only for changes that match a pre-classified LOW-RISK pattern, requiring human review for anything that could plausibly represent a legitimate emergency response (an on-call engineer opening a port during an active incident) rather than genuine unauthorized drift, since silently reverting a legitimate emergency network change mid-incident can make that incident materially worse.
Structured elaboration
Process and tooling. A scheduled (or event-driven, via cloud-provider change-notification APIs where available) comparison of the live ACL/firewall configuration against the Git-declared desired state, using the same GitOps controller or a dedicated drift-detection tool; network-security resources specifically benefit from EVENT-DRIVEN detection over pure polling where the cloud provider offers it (a cloud-native change-notification event fires the moment a rule changes, versus discovering it up to a full poll interval later), since the window between an unauthorized change and detection matters more here than for most other resource types.
How drift alerts surface in Git workflows. A detected drift on a security-relevant resource opens an automated ISSUE or PR (a PR proposing to revert to the Git-declared state is often the most actionable form, since it lets a reviewer see the exact diff and approve the revert with one click) rather than only a chat notification, so the drift has a durable, trackable record with an obvious next action attached, not just an alert that can be dismissed and forgotten.
When to auto-reconcile versus notify humans. Classify changes ahead of time into risk tiers based on WHAT changed, not just THAT something changed: a rule change that WIDENS access (opening a previously-closed port, broadening a source CIDR range) is high-risk and should always notify a human rather than auto-revert immediately, since auto-reverting a widening change during an active incident is exactly the scenario that makes things worse; a rule change that NARROWS access, or an exact match to a previously-approved-and-since-reverted state, is lower-risk and a reasonable auto-reconcile candidate, since narrowing access back toward the declared state carries little downside even if the classification turns out to be wrong.
Avoiding fighting legitimate manual changes. The core mechanism: a short, explicit SUPPRESSION WINDOW tied to a declared active incident (an on-call engineer marks "network changes to this resource are expected for the next N hours" via the incident-management tool, which the drift-detection system checks before auto-reverting or even before alerting at normal urgency), so a legitimate emergency change during a declared incident does not trigger the same alarm, or worse, an automatic revert, that an actually unauthorized change would.
Worked example
A network ACL protecting a payment-processing subnet, with the following classification applied:
| Change type | Example | Response |
|---|---|---|
| Access widened, no active incident declared | New inbound rule allowing 0.0.0.0/0 on port 22 | Immediate high-urgency alert, PR opened proposing revert, NOT auto-reverted (human confirms it wasn't an intended emergency change first) |
| Access widened, active incident declared on this resource | Same rule, but an on-call engineer marked an active incident with this resource in scope | Alert at LOWER urgency (informational), no automatic revert, explicit reminder attached to reconcile back into Git before incident closure |
| Access narrowed back toward declared state | A previously-drifted rule removed, matching Git's declared state again | Auto-reconciled silently (this IS the declared state; nothing to revert) |
| Unrelated metadata-only change (a rule's description field edited) | Description text changed, no access-control effect | Low-urgency alert only, not treated as a security-relevant drift at all |
Trade-offs and pitfalls
- Common mistake: treating ALL drift on security-relevant resources identically, regardless of whether it widens or narrows access. The risk profile of "someone opened a port" and "someone's manual change happened to match what Git already says should be there" are completely different, and a detection system that alerts (or auto-reverts) with equal urgency on both either desensitizes the team to real risk or creates unnecessary friction on harmless events.
- The incident-declared suppression window needs an EXPIRY and a hard requirement to reconcile the change into Git before the window closes, without both, "we're mid-incident" becomes a standing, indefinite excuse for undocumented network drift that never gets captured, exactly the outcome the whole reconciliation discipline exists to prevent.
- Event-driven detection via cloud-provider notifications is not universally available for every resource type or provider, and where it is not, the fallback (polling) needs a SHORTER interval specifically for security-relevant resources than for ordinary application config, reflecting that detection latency matters more here.
- Auto-reconciling "access narrowed back to declared state" without any human visibility at all can hide a genuine, worth-investigating pattern: if the SAME rule keeps drifting wide and getting silently auto-corrected repeatedly, that repeated pattern (not any single instance) is itself a signal something upstream is misconfigured or someone keeps making the same unauthorized change; even low-risk auto-reconciled events should feed a trend metric, not disappear entirely once handled.
Design release gates and approval workflows for a GitOps pipeline that must satisfy compliance requirements (auditable approvals for production). Describe where gates are enforced (CI, PR, Argo CD), what metadata should be stored in Git, and how to implement human approvals without breaking declarative principles.
Sample Answer
Direct answer
Enforce release gates at TWO complementary layers: the pull request itself (peer/approver review, required status checks, branch protection rules) as the primary compliance-auditable approval, and Argo CD's sync policy (manual sync for production, or SyncWindows restricting when auto-sync can fire) as a technical backstop that prevents a merged-but-not-yet-approved-for-deployment change from auto-applying. The key design constraint: human approval happens on the PR (a Git-native, naturally auditable event with reviewer identity and timestamp already captured), never as an out-of-band manual step performed against the live cluster, which is what would actually break the declarative, Git-is-source-of-truth principle.
Structured elaboration
Where gates are enforced. CI enforces AUTOMATED gates (linting, terraform plan/manifest-diff generation, policy-as-code checks, security scanning) as required status checks that must pass before merge is even possible. The PR itself enforces the HUMAN approval gate via required reviewers (branch protection rules requiring N approvals from a CODEOWNERS-defined group for production-targeting paths). Argo CD enforces a FINAL technical gate: production Application objects configured with syncPolicy.automated OMITTED (manual sync only) or restricted via SyncWindows, so even a merged commit doesn't immediately roll out until an authorized operator (or an automated promotion step gated on its own separate approval) triggers the sync.
What metadata should live in Git. The PR itself is the metadata record: title/description linking to the change ticket, the diff itself (what actually changed), the list of approvers and their approval timestamps (captured natively by the Git hosting platform, exportable via API for audit), and any required-context status check results (policy-as-code pass/fail, security scan results) attached to the commit. For compliance frameworks requiring a documented "why" beyond the diff, a structured commit message or PR template field (change ticket ID, risk assessment, rollback plan) keeps that context IN the same auditable artifact rather than in a separate system that can drift out of sync with what was actually deployed.
Implementing human approval without breaking declarative principles. The subtlety: "declarative" means the DESIRED STATE is fully specified in Git and reconciliation is automatic given that state, not that every state transition must be fully automatic with no human in the loop. A human approval gate is compatible with GitOps as long as the approval ITSELF is captured declaratively (the merged PR, an approved state) and the reconciler then acts deterministically on whatever is in Git; what would BREAK the principle is a human directly editing the live cluster to "approve" a change out of band, since that reintroduces exactly the drift-inducing manual intervention GitOps exists to eliminate. A common, clean pattern: a promotion PR (bumping an image tag or config value in the production overlay) requires its own separate, higher-bar approval from the PR that merged the change into staging, so "promote to production" is itself a reviewed, auditable Git event, not a live cluster action.
Trade-offs and pitfalls
- Common mistake: implementing the "human approval" step as a manual
argocd app syncperformed by an operator outside of any recorded process. This satisfies "a human looked at it" but produces NO durable audit record of WHO approved WHAT and WHY beyond whatever is (or isn't) in a chat log; the PR-based approval model is specifically what makes the approval auditable without extra tooling, since the Git platform already records it. - Common mistake: conflating CI's automated checks with the compliance-required human approval. A passing lint/policy-as-code check is necessary but not sufficient for most compliance frameworks, which specifically require a human decision-maker's sign-off; treating "all checks green" as equivalent to "approved" removes the human judgment step the compliance requirement exists to capture.
- SyncWindows and manual-sync settings are a technical backstop, not a substitute for PR-level review. They prevent an ALREADY-MERGED change from auto-deploying at the wrong time, but by the time a change is merged, the PR-level approval should already have happened; relying on the sync gate as the PRIMARY approval mechanism (skipping rigorous PR review because "someone will catch it before syncing") inverts the intended order and risks a rushed, under-reviewed sync-time approval replacing a properly reviewed PR.
- CODEOWNERS-based required-reviewer rules need to be scoped to the RIGHT paths. A blanket "any 2 approvers" rule does not satisfy most compliance requirements for segregation of duties if the two approvers can be the same person who wrote the change plus a rubber-stamping teammate; production-path changes typically need a reviewer who is NOT the author and, in stricter regimes, a reviewer from a DIFFERENT team or a designated compliance role.
Explain the key differences between application code and infrastructure-as-code with respect to side effects, idempotency, testing strategies, stateful behavior, and deployment workflows. Why do these differences change how you design Git workflows and code reviews for IaC?
Sample Answer
Direct answer
Application code and infrastructure-as-code (IaC) differ along four axes that all trace back to one root cause: application code typically runs against a FRESH, disposable process each time, while IaC runs against a persistent, STATEFUL external system it must reconcile with, not simply execute from scratch. That single difference is why IaC's side effects are inherently real-world and often hard to undo, why idempotency is a REQUIREMENT rather than a nice-to-have, why testing needs a genuinely different strategy, and why Git workflows and code review for IaC carry more weight per line changed than the equivalent application-code change typically does.
Structured elaboration
Side effects. Application code's side effects (writing to a database, calling an API) are usually scoped to WITHIN a request or process lifecycle and are, in aggregate, reversible by design (a failed deploy just doesn't serve the new code; the old process keeps running). IaC's side effects ARE the actual state of real infrastructure: a resource created, modified, or destroyed persists independent of any process, and some effects (data loss on a destroyed database, an IP address released and reassigned elsewhere) are not reversible by simply "running the old version again."
Idempotency. Well-written application code handles a request; running it twice generally means "handled two requests." IaC tools are explicitly designed to be idempotent by CONTRACT, running apply against an already-matching state should do nothing, this is a foundational property Terraform and similar tools are built around because IaC is normally run repeatedly against the SAME target (every CI run, every scheduled reconciliation), and non-idempotent behavior there would mean re-running a pipeline creates duplicate resources or repeatedly performs an action that should only happen once.
Testing strategies. Application code is tested primarily against IN-MEMORY or locally-mocked dependencies, fast, cheap, run thousands of times per day trivially. IaC testing (unit-level linting/validation, integration tests like terratest that actually provision real or near-real infrastructure) is inherently slower and often costs real money per test run, since meaningfully testing "does this correctly create a VPC" often requires actually creating one, at least in an ephemeral test account.
Stateful behavior. Application code is typically STATELESS between invocations (or explicitly manages its own state via a database, a clearly-owned concern). IaC tools maintain an external STATE FILE representing what they believe is currently deployed, a persistent, shared, lockable artifact with its own consistency and concurrency concerns that has no real analog in typical application-code deployment.
Deployment workflows. Deploying application code usually means: build an artifact, run it, and if something is wrong, roll back by deploying the PREVIOUS artifact, a symmetric, cheap operation. Deploying IaC changes means: compute a PLAN of real-world actions against a stateful system, and "rolling back" is not automatically symmetric, reverting the CONFIGURATION and reapplying can trigger an entirely different, sometimes equally disruptive, set of real actions, not simply "redeploy the old version."
Why these differences change Git workflows and code review design. Because IaC changes have real, sometimes-irreversible side effects on shared, persistent infrastructure, review needs to weigh the PLANNED EFFECT, not just the code diff, and branch protection and approval requirements typically need to be STRICTER for infrastructure changes than for a typical application-code PR of similar size, and the CI validation gate needs to include a real plan/dry-run step application-code CI generally does not have an equivalent of.
Trade-offs and pitfalls
- Common mistake: reviewing an infrastructure PR with the SAME mental model used for application code, reading the diff, checking style and logic, and approving, without separately reading the PLAN OUTPUT that shows the actual real-world effect; this is the single biggest practical consequence of these differences, and "review the plan, not just the diff" is the discipline that closes the gap.
- Idempotency is easy to accidentally break with a poorly-designed custom provisioner or a script-based resource (a
local-execprovisioner running an imperative script) that itself is NOT idempotent, even inside an otherwise well-designed Terraform configuration; the tool's OWN idempotency guarantee does not automatically extend to arbitrary code a configuration chooses to shell out to. - Testing costs real money for IaC in a way application-code testing generally does not, a testing strategy that ignores this (running full integration tests against real cloud resources on every single commit, for every PR, with no cost-aware tiering) is a common, avoidable expense; structuring fast, free checks (lint/validate) for every commit and reserving the expensive, real-infrastructure integration tests for a smaller, gated tier manages this cost directly.
- Treating "rollback" as symmetric for IaC (assume reverting the config is automatically as safe as the original apply was) is a recurring, costly mistake: a revert can trigger the SAME class of disruptive action (a forced destroy-and-recreate) in the opposite direction, not a clean undo.
Design a secure, versioned configuration backup and storage solution for sensitive system configs. Requirements: immutable historical versions, encrypted at rest with key rotation, access logging for audits, retention policies, and fast retrieval for rollback. Describe storage choices, access control, and how to integrate with CI/CD for automatic backups pre-deployment.
Sample Answer
Direct answer
A secure, versioned configuration backup system needs storage that is IMMUTABLE by construction (not merely "not usually overwritten"), because the entire point of a backup is trusting it even after something has gone wrong with the live system, including a compromise that might otherwise let an attacker tamper with backups too. Object storage with versioning enabled AND object-lock (write-once-read-many, WORM) configured for a defined retention period, encrypted at rest with periodic key rotation, access-logged for every read (not just every write), and integrated into the CI/CD pipeline as an automatic pre-deployment step, meets all five stated requirements together rather than any one addressing them individually.
Structured elaboration
Storage choice. Object storage (S3, Azure Blob, GCS) with BUCKET VERSIONING enabled captures every historical version automatically on each write, and OBJECT LOCK / WORM mode (available on all three major providers in some form) makes a version genuinely immutable for its retention period, not deletable even by an account with otherwise-broad permissions, which is the property a plain "don't delete old backups" convention cannot provide, since a convention is a policy, not an enforced guarantee.
Immutable historical versions. With object-lock in COMPLIANCE mode (the strictest setting, available on the major providers), not even the account root user can delete or overwrite a locked version before its retention period expires; this specifically defends against the scenario where an attacker who has compromised write credentials tries to destroy backup evidence of what the system looked like before the compromise.
Encrypted at rest with key rotation. Server-side encryption using a customer-managed key, with the key itself under a defined rotation schedule (automatic annual rotation is a common baseline for a KMS, key management service, managed key). Rotating the KEY does not require re-encrypting every historical backup object (the KMS handles decrypting old objects with prior key versions transparently), so rotation does not conflict with the immutability requirement above.
Access logging for audits. Every READ of a backup object (not just writes) should be logged with identity, timestamp, and object version, since "who has looked at our backups" is itself audit-relevant, a backup containing sensitive configuration (secrets references, internal topology) being READ by an unexpected identity is a signal worth catching even if nothing was modified.
Retention policies. A tiered retention schedule (for example: every version retained for 30 days, then thinned to one per day for 90 days, then one per month for a compliance-required multi-year window) balances storage cost against how far back a rollback or an audit realistically needs to reach; this needs to be an explicit POLICY on the bucket/lifecycle configuration, not a manual cleanup process someone is expected to remember to run.
Fast retrieval for rollback. Object storage's normal read latency is generally fast enough for a rollback scenario directly; for very large configuration sets where scanning many versions to find "the last known-good one" would be slow, maintaining a separate lightweight INDEX (a small database or manifest file mapping timestamps/commit-SHAs to object versions) makes "find and retrieve the right backup" a fast lookup rather than a storage-level scan.
Worked example
CI/CD integration for automatic pre-deployment backups: before any Terraform apply or GitOps sync that would change a sensitive system's configuration, a pipeline step snapshots the CURRENT live configuration (not just the about-to-be-applied Git state, since the point is capturing what was ACTUALLY running before the change) and writes it to the versioned, object-locked bucket, tagged with the commit SHA of the change about to be applied and a timestamp. The write uses a scoped, write-only credential (this pipeline step needs to WRITE new backups; it never needs to read or delete existing ones), separate from the credentials used for the actual apply step, so a compromise of the apply credential does not automatically also grant tampering access to the backup store.
Trade-offs and pitfalls
- Common mistake: relying on "we never delete backups" as an operational convention rather than an enforced object-lock policy. A convention is defeated by a single mistake, a misconfigured lifecycle rule, an overly broad IAM permission, or a genuine compromise; object-lock in compliance mode is what makes immutability a property of the STORAGE, not a property of everyone's good behavior.
- Encrypting backups with the SAME key used for the live system's own secrets creates an unnecessary shared blast radius. A dedicated key for the backup store means a compromise of the live system's operational key does not automatically also expose historical backups, and vice versa.
- Retention policy needs to account for the compliance-required MINIMUM retention window explicitly, separate from operational convenience. A tiered thinning schedule optimized purely for storage cost can accidentally violate a longer regulatory retention requirement if the two are not reconciled explicitly when the policy is designed, not discovered later during an audit.
- Write-only credentials for the backup step are a real, worthwhile scoping decision, not excess caution. A pipeline identity that can both write new backups and read/delete old ones is a single compromise away from an attacker being able to both plant a fake "known good" backup and destroy the real historical record, exactly the two things a good backup system needs to be resistant to simultaneously.
An operator reports the cluster drifted because someone updated a Deployment directly on the cluster. Using Argo CD, explain how you'd detect this drift, alert the team, and define a policy for auto-sync vs manual reconciliation. Mention concrete Argo CD configuration knobs (e.g., app settings, autoPrune, project restrictions) you'd use.
Sample Answer
Direct answer
Argo CD detects this drift automatically: its application-controller continuously compares the live cluster state against the Git-sourced desired state and marks the Application OutOfSync the moment they diverge, no separate detection mechanism needed, drift detection is the reconciliation loop's core function, not an add-on. What the operator needs to DECIDE is the response policy: whether that drift should be automatically corrected (syncPolicy.automated with selfHeal: true), left for a human to review and manually sync, or actively PREVENTED at the source via Project-level restrictions and RBAC that stop the direct edit from being possible in the first place.
Structured elaboration
Detecting the drift. No extra configuration is needed for detection itself: Argo CD's reconciliation loop diffs live vs. desired state on every poll/watch cycle (typically every few minutes, plus event-driven watches for faster detection on supported resource types) and surfaces the result as the Application's sync status (Synced / OutOfSync) directly in the UI and via the API/CLI, with a resource-level diff showing exactly which fields changed.
Alerting the team. Configure argocd-notifications (or the Notifications controller) to fire on the on-sync-status-unknown / on-app-out-of-sync triggers, routed to Slack/PagerDuty/email, so the team learns about drift proactively rather than discovering it next time someone happens to open the Argo CD UI. Alerting on drift specifically (not just on sync failures) is what catches exactly this "someone edited the cluster directly" scenario, since a manual edit does not fail anything, it just diverges silently until observed.
Auto-sync vs. manual reconciliation policy. The core knob is syncPolicy.automated.selfHeal: true means Argo CD will AUTOMATICALLY revert any detected drift back to the Git-declared state on its next reconciliation pass (the direct edit gets overwritten, enforcing Git as the actual source of truth in practice, not just in principle); false (or automated omitted entirely, requiring manual sync) means drift is detected and surfaced but NOT auto-corrected, leaving a human to decide whether to sync back to Git or, if the manual change was actually a legitimate emergency fix, capture that fix in Git properly and let reconciliation then match it. automated.prune: true additionally ensures resources REMOVED from Git also get removed from the cluster, not just resources CHANGED.
Concrete Argo CD knobs.
| Knob | Effect |
|---|---|
syncPolicy.automated.selfHeal | Auto-revert live drift back to Git state on next reconcile |
syncPolicy.automated.prune | Auto-delete cluster resources no longer present in Git |
syncPolicy.automated (unset) | Manual-sync-only; drift detected and surfaced but not auto-corrected |
project.destinations / clusterResourceWhitelist | Restrict WHAT and WHERE a project's applications can even touch, narrowing where drift can occur |
spec.syncWindows | Restrict WHEN automated sync may fire (e.g. block auto-sync during a change freeze, still allowing manual/emergency sync) |
| RBAC on the target cluster (outside Argo CD itself) | The actual technical control preventing the direct kubectl edit in the first place; Argo CD's own policy cannot stop someone with independent cluster access |
Trade-offs and pitfalls
- Common mistake: treating
selfHeal: trueas strictly "safer" than manual reconciliation for every application. Auto-reverting drift is exactly right for steady-state config, but if the direct edit was a legitimate emergency mitigation (someone scaled up replicas by hand during an incident because the Git-based promotion pipeline was too slow), an aggressive self-heal will silently UNDO that mitigation on the next reconcile, which can make an active incident worse; high-blast-radius or incident-prone applications sometimes deliberately run with manual sync so a human is in the loop before any revert happens. - Detecting drift is not the same as PREVENTING it, and a mature answer addresses both. Alerting-and-reverting is a reactive control; the stronger, complementary control is removing direct cluster-edit capability from anyone who isn't supposed to have it in the first place (RBAC scoped so that the ONLY path to change cluster state is through Git and the GitOps controller's own service account), which is a Kubernetes RBAC concern outside Argo CD's own configuration surface entirely.
- Common mistake: assuming faster polling intervals meaningfully "fix" drift-related risk. Detection latency matters less than the RESPONSE policy; an application drift-detected in 10 seconds but left unaddressed for a day because nobody reviews the alert has the same practical risk exposure as one detected an hour later and immediately reverted.
selfHealreverting a change does not, by itself, explain WHY someone made the direct edit; a mature incident-response process pairs auto-revert with a required postmortem or at least a recorded reason whenever selfHeal fires against a change that wasn't accidental, so the underlying reason for the manual edit (a broken pipeline, a missing self-service path) actually gets fixed rather than just repeatedly reverted.
Unlock Full Question Bank
Get access to all Infrastructure as Code and GitOps interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.