Infrastructure as Code and GitOps Questions

Direct answer

The workflow itself (branch, edit, commit, push, open a pull request, wait for CI, merge) is IDENTICAL in mechanics to an application-code change; what differs for infrastructure-as-code (IaC) is what the required CI validation actually checks (a terraform plan showing the real-world effect of the change, not just a test suite) and the weight given to review, since a merged infra change can directly reconfigure live, shared resources the moment it applies, with no separate build-and-deploy step standing between merge and real-world effect the way an application change usually has.

Structured elaboration

Step-by-step, using the question's own example commands as the backbone.

git checkout -b feat/update-vpc

Start from an up-to-date base branch and create a descriptively named feature branch; naming it after the CHANGE ("update-vpc") rather than a ticket number alone helps a reviewer understand intent from the branch list without opening the PR.

Edit the Terraform configuration (for example, widening a VPC's CIDR range in the relevant .tf file).

git add .
git commit -m "feat(vpc): increase cidr range"

A commit message following this project's convention: a concise subject describing WHAT and, ideally, enough context to infer why, referencing a ticket if one exists.

git push origin feat/update-vpc

Push the branch and open a pull request describing the intended change and its expected blast radius.

Running CI validations, the infra-specific step. The pipeline runs terraform fmt -check and terraform validate (syntax and internal consistency), then terraform plan, and posts the PLAN OUTPUT directly on the PR as a comment or check annotation, so the reviewer sees exactly what will change in the real infrastructure (which resources get created, modified in place, or destroyed and recreated) alongside the code diff, not just the code diff alone.

Review. The reviewer reads both the CODE diff and the PLAN output together; particular attention goes to any resource marked for DESTRUCTION or REPLACEMENT in the plan, since those carry real operational risk a pure code-review pass could miss if the reviewer only read the .tf diff without checking what Terraform actually intends to do with it.

Merging. Once approved and CI is green, merge; depending on the pipeline design, the actual apply may run automatically post-merge, or require a SEPARATE manual approval gate before applying, distinct from the PR's own code-review approval.

Worked example

The exact commands from the question, extended with the infra-specific CI step made explicit:

git checkout -b feat/update-vpc
# ... edit network/vpc.tf, widening the CIDR block ...
git add .
git commit -m "feat(vpc): increase cidr range"
git push origin feat/update-vpc
# open PR; CI runs:
#   terraform fmt -check
#   terraform validate
#   terraform plan -out=tfplan   (plan output posted to the PR)
# reviewer reads the code diff AND the posted plan output together
# on approval + green CI: merge
# post-merge: terraform apply (auto or gated by a separate approval, per pipeline design)

Trade-offs and pitfalls

• The single biggest practical difference from an application-code change: the CODE DIFF alone does not tell a reviewer what will actually happen. A one-line change to a resource argument can trigger an in-place update OR a destroy-and-recreate depending on that specific argument's ForceNew behavior in the provider; reviewing the diff without reviewing the PLAN is reviewing the WRONG artifact for infrastructure changes specifically.
• Common mistake: treating the plan-output-on-PR step as informational only, not something the reviewer is expected to actually read. If nobody reads the posted plan, the extra CI step provides no real safety benefit over reviewing application code the normal way; the workflow only earns its infra-specific rigor if the plan output is genuinely part of what gets reviewed, not merely generated.
• Merge and apply being separate steps (rather than merge automatically applying) is itself a real design decision with a trade-off, immediate auto-apply on merge keeps the mental model simple (merged means live) but removes a final safety check between "this was approved" and "this is now actually changing production"; a separate, explicit apply gate adds a step but catches the case where the plan changed between review time and merge time (another change merged in between) before it silently applies.
• Common mistake: naming the branch or commit purely after a ticket number with no description of the actual change ("fix/JIRA-4521"); this workflow's whole value depends on Git history being genuinely READABLE later, and a ticket-only name defeats that the moment the ticket system itself becomes hard to search or is retired.

Direct answer

GitOps is an operating model where Git holds the DECLARED DESIRED STATE of a system (Kubernetes manifests, in this case) and an in-cluster controller (Argo CD or Flux) continuously reconciles the live cluster toward whatever is committed, so every change to the running system happens BY committing to Git first, never by an operator applying something directly. An implementation plan for a company adopting this needs six concrete decisions made up front: repository layout, promotion strategy, secrets handling, drift detection policy, rollback mechanics, and which controller to run, each of which shapes how the team actually works day to day, not just which tool sits in the diagram.

Structured elaboration

What GitOps is, precisely. Three properties together define it: (1) the ENTIRE desired state is expressed declaratively and lives in Git, (2) an automated controller, not a human running kubectl or a CI job pushing changes, PULLS from Git and applies it, and (3) the controller continuously RECONCILES, correcting any divergence between live and declared state, not just applying once at deploy time. Missing any one of the three is a related but different pattern (a CI pipeline that runs kubectl apply on merge is Git-triggered but PUSH-based, not GitOps in the pull-based, continuously-reconciling sense).

Repo layout (app vs infra). A common, workable split: an APPLICATION repo per service, owned by the team that builds it, containing the service's own Dockerfile/build config and a BASE Kubernetes manifest (or Helm chart) with no environment-specific values; and a separate INFRASTRUCTURE (or "config") repo, owned by the platform team, containing the per-environment OVERLAYS (Kustomize overlays or per-environment values files) that the GitOps controller actually watches and syncs from. This separation means an application team can iterate on their own manifests without needing infra-repo write access for every change, while environment promotion (below) becomes a change to the infra repo alone, not a rebuild of the application.

Promotion strategy. Promotion means moving a specific, already-built artifact (an image digest) from one environment's declared state to the next, dev to staging to production, by committing that digest into the NEXT environment's overlay in the infra repo. A common pattern: CI automatically opens a PR bumping the dev overlay on every successful build; promotion to staging and production are each their own PR (often automated with a required approval gate for production specifically), so each promotion step is its own reviewed, auditable Git event rather than one build automatically cascading through every environment unchecked.

Handling secrets. Secret VALUES never live in the plain manifests the controller applies. Use one of the established patterns: Sealed-Secrets or SOPS (an encrypted blob committed to Git, decrypted only in-cluster or by the controller), or an External Secrets Operator pulling live from a dedicated secrets store (Vault or a cloud key management service) and syncing into native Kubernetes Secrets. For a company just adopting GitOps, External Secrets Operator is often the simpler operational starting point (no per-secret encryption workflow for developers to learn), while Sealed-Secrets/SOPS avoid taking on a new runtime dependency; the choice should follow the team's existing secrets infrastructure, not be picked in isolation.

Drift detection. The controller's reconciliation loop detects drift automatically (comparing live vs. Git-declared state on every cycle); the IMPLEMENTATION decision is the response policy, auto-heal (revert drift automatically) for steady-state config, or alert-and-require-manual-sync for higher-risk applications where an unreviewed automatic revert could make an active incident worse. A newly adopting team should default to alert-and-manual-sync broadly at first, and graduate specific, well-understood applications to auto-heal as confidence builds, rather than enabling aggressive auto-heal everywhere from day one.

Rollback mechanisms. Because the desired state is just a Git history, a rollback is conceptually reverting the relevant commit (the overlay change that introduced the bad state) and letting the controller reconcile the cluster back to the prior state automatically; no separate rollback tooling is required beyond normal Git operations, though the controller's OWN rollback/history UI (Argo CD's rollback-to-a-prior-sync view, for instance) is often faster in practice than manually identifying and reverting the right commit under incident pressure.

Trade-offs and pitfalls

• Common mistake: treating "we run Argo CD" as equivalent to "we do GitOps." If the team still routinely applies emergency changes directly to the cluster and later reconciles Git to match afterward, the actual source of truth in practice is the CLUSTER, not Git, even though a GitOps controller is technically running; the tooling alone does not create the discipline.
• Splitting app and infra repos too early, before there is more than one or two services, adds coordination overhead with no real benefit yet. For a small initial adoption, a single repo with clearly separated app and overlay directories is often the right starting point, splitting into separate repos as the number of teams and services genuinely grows past what one repo can coordinate cleanly.
• A company new to GitOps often underestimates the secrets-handling decision's operational weight. Whichever pattern is chosen, someone has to own key rotation, access review, and the promotion workflow for secrets specifically (a secret often needs different values per environment, unlike an image digest that promotes unchanged); treating secrets as an afterthought after the manifest-promotion pipeline is designed tends to produce a bolted-on, inconsistent secrets workflow.
• Rollback via Git revert only fully restores the prior state if image references were immutable (digests, not mutable tags) at the time of the original commit, a dependency worth calling out explicitly rather than assuming, since Git revert alone does not guarantee it.

Direct answer

Migrating from ad hoc Puppet manifests with frequent emergency fixes to a tested, Git-promoted infrastructure-as-code (IaC) approach needs the SAME evidence-first discipline as any drift-remediation migration (discover actual state before declaring desired state, prove the pipeline on a small representative slice before scaling it), with one addition specific to this case: because the CURRENT pain point is described as manual fixes and inconsistency, not merely "we want new tooling," the plan's early KPIs need to demonstrate a REDUCTION in exactly that pain (fewer emergency manual fixes, fewer inconsistency-caused incidents), not just "the new pipeline is running," or the migration will not have addressed the actual problem that motivated it.

Structured elaboration

Tooling choices. Given an EXISTING Puppet investment, the pragmatic choice is usually not a wholesale tool replacement but adding what Puppet's ad hoc usage was missing: a TESTING framework for manifests (rspec-puppet, or an equivalent for whichever tool is chosen), and a Git-based PROMOTION pipeline wrapped around the existing Puppet manifests, rather than a parallel rewrite in a different tool competing for the same migration budget. Where Puppet's own model is genuinely a poor fit for the target architecture (moving toward Kubernetes/GitOps rather than long-lived VM fleets, for instance), a tool change is justified, but that decision should follow from the target architecture, not be assumed as part of "modernizing."

Pilot selection. A small set of manifests (or a small server cohort) chosen for BOTH representativeness (touches a real cross-section of the fleet's actual complexity, not the simplest, least-risky manifests available) and for being an area where the CURRENT pain (frequent emergency fixes) is genuinely felt, so a successful pilot produces evidence directly relevant to the problem the migration is meant to solve, not just evidence that the new pipeline can run at all.

Testing. Manifest-level tests (does this manifest produce the expected resource state, run in an isolated test environment, not against production) run in CI before any promotion; this is the concrete mechanism addressing "inconsistent environments," since a manifest that passes its own tests before promotion is far less likely to be the SOURCE of a new inconsistency than the current ad hoc process.

Git-based promotion. The SAME dev-to-staging-to-production overlay-and-promotion pattern used for other configuration-management migrations, adapted to Puppet's own environment model (Puppet environments map naturally onto this pattern already, often requiring less structural change than migrating from a completely unstructured starting point would).

Policy changes. The organizational change that makes the technical migration STICK: a policy that emergency fixes now go through a bounded, audited emergency-change process (a fast-path change still captured back into the Puppet manifests afterward, not a permanent exception), and that DIRECT, un-declared changes to manifest-managed servers are explicitly discouraged and, once the migration matures, actively caught by drift detection.

KPIs measuring improvement in consistency and incident reduction. Directly tied to the stated pain points, not generic pipeline-health metrics: (1) count of emergency/manual out-of-band fixes per month (should trend down as the new process absorbs what used to require an ad hoc fix); (2) count of incidents attributable to environment inconsistency specifically (a bug reproducing on some hosts but not others due to drift), tracked as its own incident category if not already; (3) percentage of the fleet under tested, Git-promoted management (a straightforward progress metric); (4) MEAN TIME between a manifest change being merged and it being live in production (should trend toward a consistent, predictable value, itself evidence the promotion pipeline is working reliably rather than being bypassed).

Worked example

A phased plan for a 150-server Puppet fleet with a documented history of 8 to 10 emergency manual fixes per month:

1. Weeks 1 to 2, discovery and pilot selection. Audit current Puppet manifest coverage and identify the 3 to 4 manifest classes responsible for the most emergency fixes historically (using existing incident/change-ticket history, not guesswork), these become the pilot.
2. Weeks 3 to 6, pilot. Add rspec-puppet tests for the pilot manifests, wrap them in a Git-based CI/promotion pipeline (dev/staging/prod environments), and run the emergency-fix policy change for JUST the pilot cohort.
3. Weeks 7 to 8, evaluate. Compare pilot-cohort emergency-fix count and any inconsistency-related incidents against the pre-migration baseline for the SAME servers; a clean pilot shows a measurable drop in both.
4. Weeks 9 to 20, phased expansion, remaining manifest classes migrated in risk-ordered cohorts, each with the same test-then-promote treatment.
5. Ongoing, KPI tracking, the four metrics above tracked monthly and reported specifically against the pre-migration baseline, not just as an absolute number, so "improvement" is demonstrable, not assumed.

Trade-offs and pitfalls

• Common mistake: treating this as primarily a tooling migration and measuring success by "percentage migrated" alone, without also tracking the emergency-fix and inconsistency-incident counts the actual business case rests on; a fleet that is 100% migrated but still shows the same emergency-fix rate has not actually solved the problem that motivated the migration.
• Choosing pilot manifests for ease rather than for where the CURRENT pain is concentrated produces a smooth pilot that generates weak evidence for the specific problem being solved; the pilot's value depends on it being drawn from the actual pain, not merely the easiest starting point.
• A policy change (routing emergency fixes through a bounded, audited process) without the accompanying technical drift-detection backstop is easy to informally erode over time, once the initial migration excitement fades, an unenforced policy alone tends to regress toward the old ad hoc habit; pairing the policy with detection (even lightweight, at first) gives it staying power.
• Comparing post-migration metrics only against an absolute target, rather than against the SAME servers' own pre-migration baseline, makes it hard to attribute improvement (or its absence) to the migration specifically versus unrelated organizational changes happening concurrently; a same-server, before-and-after comparison is a meaningfully stronger form of evidence.

Direct answer

The attack surface of a configuration repository and its promotion pipeline splits into four zones an attacker could target: the REPOSITORY itself (who can commit, whose commits get merged), the CI RUNNERS that execute pipeline steps (what they can read and reach), the SUPPLY CHAIN of dependencies and base images the pipeline pulls in, and the PROMOTION MECHANISM that actually applies changes to production. A prioritized control set addresses the zone with the highest (likelihood times blast radius) first: branch protection and required review (repository), ephemeral, least-privilege runner credentials (runners), pinned and verified dependencies (supply chain), and a promotion gate that cannot be bypassed by a single compromised identity (promotion mechanism).

Structured elaboration

Repository zone risks and controls. Risk: a compromised or malicious contributor merges a change that exfiltrates secrets or grants themselves broader access, or force-pushes to rewrite history and hide evidence. Controls, roughly ordered by effectiveness per unit of operational cost: branch protection requiring review from someone OTHER than the author (cheap, high effect, stops the single most common path); required status checks (policy-as-code, secret-scanning) that must pass before merge (cheap, catches accidental leaks and known-bad patterns); disallowing force-push and requiring signed commits on protected branches (moderate cost, closes the history-tampering path); a CODEOWNERS-style requirement that changes to the pipeline definition ITSELF need a security or platform-team reviewer, not just any team member (moderate cost, closes the meta-attack of modifying the pipeline to weaken its own controls).

CI runner zone risks and controls. Risk: a compromised runner (via a malicious dependency executed during the build, or a vulnerability in the runner's own environment) reads secrets injected into the job, or pivots to reach other systems the runner's network position allows. Controls: short-lived, scoped credentials issued PER JOB (OIDC-based federation to the cloud provider rather than a long-lived static secret stored in the CI system) so a compromised runner's blast radius is bounded to that one job's narrow scope and a short time window; network-isolate runners so they cannot reach anything beyond what THAT specific job legitimately needs; and treat runner logs as potentially containing secrets, redacting known secret patterns automatically rather than trusting every script to avoid printing them.

Supply chain zone risks and controls. Risk: a malicious or compromised third-party action, base image, or dependency executes arbitrary code during the pipeline run. Controls: pin dependencies to exact versions or content hashes rather than floating tags (actions/checkout@<sha> not @v4), scan and sign container images used as pipeline steps, and restrict which third-party actions/images are allowed at all (an explicit allowlist for anything with write access to secrets or production credentials).

Unauthorized promotions zone risks and controls. Risk: a compromised identity, or a legitimate identity acting outside intended process, triggers a production promotion without the required review. Controls: require the promotion trigger (a merge to a protected production-overlay path, or an explicit approval gate in the deployment tool) to be tied to a REVIEWED PR event, not a webhook or manual trigger any single credential can fire; and log every promotion event with enough context (who/what triggered it, which commit, which approvals) to make an unauthorized promotion immediately detectable even if it cannot be prevented outright.

Worked example

Prioritized by (likelihood times blast radius), for a mid-size organization with an existing but not security-hardened pipeline:

The first four sit at the top because they each close a HIGH-BLAST-RADIUS path (a merged malicious change, a compromised runner with broad reach, an unpinned supply-chain dependency, or a promotion nobody reviewed) at comparatively low implementation cost; signed commits and some of the more process-heavy controls are real but address a narrower slice of risk relative to their rollout cost, so they land lower without being unimportant.

Trade-offs and pitfalls

• Common mistake: treating a required status check as equivalent to a required human review. An automated check catches known patterns; it does not catch a change that is subtly malicious but syntactically clean, which is exactly what human review is for. The two are complementary, not substitutes.
• Ephemeral, per-job credentials are the single highest-leverage control on this list and also the one most often skipped, because migrating away from a long-lived static secret already embedded in a working pipeline feels riskier to touch than leaving it; the actual risk asymmetry runs the other way, a long-lived static secret is a standing target for as long as it exists.
• An allowlist for third-party actions/images needs an owner and a review process for ADDING to it, otherwise it either calcifies (blocking legitimate new tooling and creating pressure to bypass it) or erodes (approvals granted too casually under time pressure, defeating its purpose).
• Common mistake: prioritizing controls by ease of implementation rather than by risk reduction per unit of cost. The cheapest controls to implement are not always the highest-leverage ones; a genuinely prioritized list, as above, has to weigh blast radius explicitly, not just roll out whatever is fastest to configure first.

Direct answer

Terraform state snapshots, PKI (public key infrastructure) binaries, and large AMI (Amazon Machine Image) artifacts share one property that makes storing them directly in Git a bad fit: Git's storage model is optimized for TEXT that diffs well and history that stays small, while these are large, frequently-regenerated, mostly-BINARY blobs that Git would store as a full new copy on every change, bloating every clone forever. The right home depends on what the artifact actually needs: Git LFS (Large File Storage) for files that genuinely need Git-native versioning and PR-based review despite being binary; a dedicated artifact repository (Artifactory, or plain object storage like S3) for anything that is really a BUILD OUTPUT with its own release lifecycle; and Terraform's own remote backend, never Git at all, for state specifically, since state's locking, current-value, and sensitive-data properties do not map onto Git's model at all.

Structured elaboration

Git LFS. Stores large files as pointers in Git (small text references) while the actual bytes live in a separate LFS-aware storage backend; Git operations (clone, diff, history) stay fast because the large content is fetched separately and only on demand. This fits files that genuinely benefit from being tied to a specific commit and reviewed via a normal PR flow (a PKI root certificate bundle that changes rarely and where "which commit introduced this exact cert" matters), but LFS storage and bandwidth typically cost more per gigabyte than plain object storage, and LFS is a worse fit for anything that changes FREQUENTLY (every LFS-tracked version is still a distinct stored blob, so frequent large-file churn is expensive regardless of the pointer trick).

Artifact repositories (Artifactory, S3 as a plain artifact store). Purpose-built for exactly this: large binary build outputs, versioned, with lifecycle policies (retention, promotion between repositories for dev/staging/prod), typically far cheaper per gigabyte than Git LFS, and with retrieval APIs designed for CI/CD consumption (pull the specific version a pipeline needs, not the whole history). This is the right home for large AMI artifacts specifically, an AMI has its OWN natural versioning and promotion lifecycle (built once, promoted through environments, eventually deprecated) that maps directly onto an artifact repository's model, and gains nothing from Git's commit-and-diff semantics since an AMI is not something anyone reviews as a text diff.

Remote backends, for Terraform state specifically. State is fundamentally different from the other two categories: it needs LOCKING (to prevent concurrent-write corruption), it represents CURRENT truth (not a historical build artifact with old versions kept around for their own sake), and it routinely contains SENSITIVE VALUES. None of Git, Git LFS, or a generic artifact repository provide locking as a first-class concept; Terraform's own remote backends (S3+DynamoDB, Terraform Cloud) are purpose-built for exactly this and are the only appropriate home, state should never be committed to Git in any form, LFS-tracked or otherwise.

Worked example

A concrete assignment for a mid-size infrastructure org:

Integration into PR reviews, CI, and access controls. For the LFS-tracked PKI bundles, a PR reviewer sees the LFS pointer diff and, critically, needs LFS-aware tooling to actually inspect the real content change (a plain git diff on an LFS pointer shows only the pointer hash changing, not the certificate content itself, so the review process needs an explicit step, a CI job that renders and posts the actual diff, to make review meaningful rather than rubber-stamping a hash change). For AMI artifacts in the artifact repository, CI publishes a new version on build and the PROMOTION step (dev to staging to prod) is a separate, reviewed action referencing that specific version by its immutable identifier. Access control differs meaningfully by artifact type: PKI material needs the tightest read scoping of anything on this list (arguably tighter than most application secrets), while AMI artifacts typically need broad READ access (anything provisioning from them) but narrow WRITE access (only the build pipeline that produces them).

Trade-offs and pitfalls

• Common mistake: committing Terraform state, even encrypted, to Git or Git LFS "for convenience," because it is technically just a file. This is a recurring, real mistake distinct from the general large-artifact question; even encrypted, Git provides no locking, so this reintroduces the exact concurrent-write-corruption risk remote backends specifically exist to solve, an LFS-stored state file is still Git-adjacent enough to tempt someone into a normal git commit workflow for it, defeating the whole point.
• Git LFS's per-gigabyte cost, multiplied by CHANGE FREQUENCY, is the single most common reason a team regrets choosing it for something that turned out to churn more than expected. A file that seemed like a natural LFS candidate at adoption time (infrequent changes) can quietly become expensive if its actual change cadence increases later; periodically reviewing what is tracked in LFS against its actual current churn rate catches this before the cost surprise does.
• A plain git diff on an LFS pointer is actively misleading if reviewers are not aware LFS is in use, it shows a hash changing, which LOOKS like a trivial change and can get rubber-stamped, while hiding a potentially significant underlying content change; any team adopting LFS needs an explicit review-tooling step (or at minimum, clear reviewer training) to close this gap.
• Artifact repository retention/lifecycle policies need to reconcile with any compliance-driven minimum-retention requirement: cost-optimized automatic cleanup can silently violate a longer regulatory retention window if the two policies are not designed together.