Infrastructure as Code and GitOps Questions

Direct answer

A delegated self-service model has to separate two things a naive design conflates: WHO can request/approve a change (which should genuinely delegate to the requesting team, that is the entire point of self-service) and WHAT that change is allowed to touch (which stays centrally controlled via policy-as-code and repository permissions, regardless of who is requesting). Self-service without that separation either becomes ungoverned (anyone can do anything, defeating central control) or becomes fake self-service (every request still routes through a central team for approval, defeating the actual point of delegating).

Structured elaboration

Repository permissions. A shared repo with path-scoped CODEOWNERS: teams get WRITE access (via PR) to their own scoped directory, and the PLATFORM team retains sole write access to shared modules and policy definitions; self-service means a team can propose and, within their own scope, approve their OWN changes, not that they have unrestricted repository access.

Approval flows. Two tiers: requests that STAY WITHIN a team's pre-approved policy boundary (a resource size within an approved range, a resource type on an approved list) can be approved by the REQUESTING team's own designated approvers, genuine self-service; requests that fall OUTSIDE that boundary (a new resource type never used before, a size beyond the pre-approved range) route to the platform team for a one-time, ADDITIONAL review, after which, if approved, that specific pattern can be added to the team's own pre-approved boundary for future self-service use, so the platform team's involvement narrows over time rather than being a permanent bottleneck for every request of that type.

Policy-as-code enforcement. The PRE-APPROVED BOUNDARY itself is expressed as policy (an OPA/Sentinel policy defining, per team, what resource types/sizes/configurations are within their self-service scope), evaluated automatically on every PR; this is the mechanism that makes "self-service within a boundary" enforceable rather than merely a documented expectation a team could accidentally or deliberately exceed.

Sandbox environments. A genuinely SEPARATE, low-stakes environment (per-team or shared-but-isolated) where a team can experiment and validate a change BEFORE it goes anywhere near a real approval flow, reducing how often a request reaches even the self-service approval step already broken or obviously wrong, and giving the requesting team fast, independent iteration without needing anyone else's involvement at all for the exploratory phase.

How audit trails and rollback capabilities are preserved. Because delegation happens entirely through the SAME Git-PR-and-policy-as-code mechanism as any other change (not a separate, informal self-service tool bypassing normal infrastructure workflow), every self-service change is a normal, fully-audited commit; rollback is the same Git-revert-and-reconcile mechanism any other change uses, unaffected by WHO originally approved the change.

Worked example

A concrete self-service flow for a team requesting a new S3 bucket:

1. Team opens a PR in their own scoped directory declaring the new bucket, using the platform team's PRE-APPROVED module for S3 buckets (which already encodes required settings: encryption, tagging, a size/retention policy within an approved range).
2. Policy-as-code evaluates the PR: since the request uses the pre-approved module within its parameter bounds, it is WITHIN the team's self-service scope.
3. The requesting team's own designated approver reviews and approves, no platform-team involvement required for this specific, in-bounds request.
4. CI applies, using credentials scoped to that team's own resources, per the multi-tenant access-control model.
5. A LATER request from the same team for a bucket configuration outside the pre-approved parameters (a much larger retention window, for instance) fails the policy-as-code check for self-service eligibility and routes to the platform team for one-time review; if approved, the platform team updates the pre-approved module's parameter bounds so future requests of this same shape become self-service too.

Trade-offs and pitfalls

• Common mistake: building "self-service" as a separate tool or portal outside the normal Git-PR workflow, which usually means it does NOT inherit the audit trail, review discipline, and rollback mechanism ordinary Git-based infrastructure changes get for free; keeping self-service INSIDE the same Git-and-policy-as-code mechanism, just with delegated approval authority within a scoped boundary, is what preserves those properties without extra, parallel tooling to build and maintain.
• A pre-approved boundary that never expands stays a bottleneck disguised as self-service, every request even slightly outside the ORIGINAL boundary routes to the platform team forever; the worked example's last step (updating the boundary after a one-time review) is what keeps self-service actually scaling over time rather than plateauing at whatever the initial policy happened to allow.
• A sandbox environment that is not GENUINELY isolated (shares any real infrastructure or credentials with production) provides a false sense of safety, a team experimenting in a sandbox they believe is isolated, when it is not, can cause real damage while under the impression they are in a safe, disposable space; the isolation needs to be structurally real, not just labeled as a sandbox.
• Central controls (policy-as-code, module curation) need their own maintenance discipline, a pre-approved module or policy that goes stale (missing a security update the rest of the org has adopted) silently becomes the weakest link in an otherwise well-governed self-service system, since teams building on it inherit whatever gap it has without necessarily knowing to check.

Direct answer

Treat drift detection as one CROSS-CUTTING capability applied to two genuinely different populations rather than one problem: IaC-managed resources are checked by comparing live state against the Terraform/Ansible state that already declares their desired configuration, while manually configured VMs have NO desired-state declaration to compare against yet, so the real first step for that population is DISCOVERY (inventory what exists) before drift detection is even possible. The staged migration then moves manual hosts, one cohort at a time, from "undeclared" to "declared but not yet enforced" to "declared and enforced," never skipping the middle step, since jumping straight to enforcement on a host whose current configuration was never actually captured risks a disruptive first-apply that reverts real, load-bearing settings nobody documented.

Structured elaboration

Tooling options, by population.

• IaC-managed resources: native drift detection (terraform plan run on a schedule, diffed against the last-applied state; Ansible's --check mode against its inventory) reports drift directly, since desired state already exists.
• Manually configured VMs, discovery phase: an agentless configuration-scanning tool (Ansible ad hoc facts-gathering, osquery, or a cloud provider's config-inventory service for cloud VMs) builds a first-pass inventory of installed packages, running services, open ports, and key config files, WITHOUT changing anything, purely observational.
• Manually configured VMs, once inventoried: generate an initial Ansible playbook or Terraform import block FROM the discovered state (rather than hand-writing desired state from scratch and hoping it matches), so the very first "desired state" declaration is a faithful snapshot of what is actually running, minimizing the chance the first apply changes anything unexpectedly.

Risk mitigation, the core discipline. Every newly onboarded host goes through an AUDIT-ONLY period before any enforcement is enabled: the tool reports what it WOULD change (a dry-run plan, Ansible --check --diff) without applying it, and a human reviews that diff specifically looking for surprises, anything the discovery phase missed or mis-captured. Only after a clean audit period (the dry-run plan stabilizes and shows no unexpected changes across a few consecutive runs) does the host move to enforcement (auto-apply or scheduled apply). This ordering, discover then audit-only then enforce, is what prevents the single biggest risk in this kind of migration: applying a freshly authored desired state against a host whose ACTUAL current configuration was captured incompletely.

Staged rollout plan.

1. Pilot cohort (5 to 10 percent of manual hosts, lowest business risk). Full discover, audit-only, enforce cycle on hosts where a mistake is cheap to recover from (non-production, or production hosts with strong redundancy). Use this cohort to tune the discovery tooling's accuracy before touching anything higher-stakes.
2. Expand by risk tier, not by convenience. Move to the next tier (higher-traffic but still redundant services) only after the pilot cohort has run cleanly under enforcement for a defined bake period (for example two full weeks with zero unexpected reverts), applying the SAME discover, audit, enforce sequence, not skipping steps because the pilot went well.
3. Highest-risk tier last (stateful, low-redundancy, or compliance-sensitive hosts). These get the audit-only period extended and often a manual sign-off gate before enforcement is switched on, since an unexpected revert here has the highest cost.
4. Continuous cohort, not a one-time project. Once the existing fleet is migrated, any NEWLY provisioned host must be born already IaC-managed (provisioned via the same pipeline from day one), so the "manual host" population only shrinks and never silently regrows from new manual provisioning.

Worked example

A fleet of 400 hosts: 250 already Terraform/Ansible-managed, 150 manually configured across on-prem and two cloud providers. Applying the plan: discovery scans all 150 manual hosts (roughly a week, agentless, zero risk since it only reads). Of those, 90 are low-risk (dev/staging, redundant web tier) and become the pilot plus first expansion cohort; each goes through a two-week audit-only period, and 84 of the 90 show a clean, stable dry-run plan by week two (the other 6 have discovery gaps, unexpected cron jobs and locally-installed packages the initial scan missed, and get their captured state manually corrected before proceeding). Those 84 move to enforcement. The remaining 60 (production databases, a compliance-scoped payment-processing tier) get a four-week audit-only period plus a manual review gate before enforcement, migrating in the final stage. Total migration: roughly 8 to 10 weeks for the full 150-host population, moving in three risk-ordered cohorts rather than one flat cutover.

Trade-offs and pitfalls

• Common mistake: writing the manual hosts' desired-state declaration by hand from documentation instead of generating it from discovered actual state. Documentation is frequently stale on manually configured hosts precisely BECAUSE nobody has been enforcing it; a hand-written desired state built from stale docs will diverge from reality in ways the audit-only period exists specifically to catch, but skipping straight to enforcement without that period turns every documentation gap into a live incident.
• Common mistake: enforcing on the ENTIRE manual-host population at once "since discovery went well for the sample we checked." The pilot cohort's success does not generalize automatically; different host tiers commonly have different undocumented local customizations, and the risk-ordered staged rollout exists precisely because a clean pilot is evidence about the PILOT, not proof the whole population is equally safe to enforce on simultaneously.
• The audit-only bake period needs an explicit, pre-agreed exit criterion (for example, N consecutive clean dry-runs), not a vague "looks stable." Without a concrete bar, teams under schedule pressure tend to shorten the audit period informally, which reintroduces exactly the risk the staged plan is designed to manage.
• On-prem and cloud VMs often need genuinely different discovery tooling (cloud providers frequently expose a native config-inventory API that is faster and more complete than a generic agentless scan, while on-prem hosts may only be reachable via SSH-based fact-gathering); treating "detect drift across clouds and on-prem" as one uniform tooling choice rather than the right tool per environment slows the discovery phase down without a compensating benefit.

Direct answer

A GitOps workflow with multi-environment promotion needs the environment SEPARATION and the PROMOTION MECHANISM to be two distinct, deliberately-designed pieces, not one implicit consequence of "we have several folders": separate repos or branches per environment (or, more commonly in current practice, separate DIRECTORIES/overlays within one repo, watched by each environment's own controller Application/Kustomization object) give each environment its own independently-reconciled desired state, while promotion is an EXPLICIT, reviewed act of copying a specific, already-validated change from one environment's declared state into the next, never an automatic cascade.

Structured elaboration

Separate repos/branches per environment, and why directory-based overlays are usually preferred instead. Full separate REPOS per environment gives the strongest isolation (different access control per environment's repo) but makes promotion mechanically harder (a cross-repo PR is not a first-class Git concept) and duplicates shared base configuration unless carefully factored out. Separate BRANCHES per environment (a staging branch, a production branch) makes promotion a simple merge/cherry-pick, but branches are mutable, ongoing streams, not a natural fit for "environment X's current desired state" as a single, always-current thing to look AT rather than merge INTO. The pattern most GitOps tooling and teams converge on: ONE repo, per-environment directories (Kustomize overlays), each watched independently by that environment's controller instance/object; promotion becomes a PR modifying the target environment's directory (usually bumping a pinned version/digest reference), which keeps the "one repo to look at for anything" ergonomics while still giving each environment its own independently-reconciled, independently-reviewable state.

Automated drift detection. Each environment's controller instance runs its own independent reconciliation loop against its own directory; drift in staging and drift in production are detected and handled completely independently (an environment-specific selfHeal policy, likely more permissive in staging than production).

Controlled promotion mechanism. CI automatically opens a PR bumping the NEXT environment's directory whenever the CURRENT environment's deployed state has been stable for a defined bake period (or on an explicit manual trigger), referencing the exact, already-built artifact (an image digest) that was validated in the prior environment, never a fresh build; each promotion PR requires its own review, with production's promotion PR typically requiring a stricter approval bar (more reviewers, or a specific approver group) than staging's.

Secrets handling across environments. Each environment's secrets are scoped to that environment's own path in the secrets backend (an External Secrets Operator instance per environment reading from its own scoped path prefix), so a promotion PR bumping an application version never needs to touch or reference secret VALUES directly, the secret REFERENCE structure stays the same across environments while the underlying value differs per environment automatically.

Hotfixes that must skip standard promotion. A defined, narrow exception path: a hotfix PR targets PRODUCTION's directory directly (skipping the normal dev-to-staging-to-production sequence), still going through the SAME review and CI validation as any other change to that directory, just without first passing through the earlier environments; critically, the hotfix commit needs to be BACK-PROPAGATED into the earlier environments' directories afterward (a follow-up PR bringing dev and staging back in sync with whatever the hotfix changed), so the earlier environments do not silently regress relative to production and reintroduce the same bug on the next normal promotion.

Trade-offs and pitfalls

• Common mistake: implementing "promotion" as a script that copies the entire staging directory's contents into production's, rather than promoting a specific, already-validated artifact reference. This risks silently promoting something that was NEVER actually validated in staging (an unrelated concurrent change to staging's directory that happened to be present at copy time); promotion should reference the SPECIFIC artifact/commit that was actually tested, not "whatever staging currently looks like."
• The hotfix back-propagation step is the piece most often skipped once the immediate fire is out, exactly the same failure pattern as any emergency-change reconciliation gap; without a tracked follow-up task (not just a runbook suggestion), earlier environments quietly drift out of sync with production's actual, hotfixed state.
• Per-environment secrets-backend path scoping (External Secrets Operator per environment, scoped to that environment's own prefix) is what prevents a promotion PR from ever needing to touch a secret VALUE, if this scoping is not done carefully, promotion PRs can end up needing environment-specific secret handling bolted on ad hoc, defeating the goal of promotion being a clean, generic mechanism.
• A stricter approval bar for production's promotion PR only provides real protection if it is enforced structurally (branch protection / required reviewers on that specific directory path), not merely documented as expected practice; the same CODEOWNERS-and-branch-protection discipline that makes any change auditable applies directly here.

Direct answer

Service-level objectives (SLOs) and error budgets should be represented as PLAIN, VERSION-CONTROLLED DECLARATIVE ARTIFACTS (a YAML or similar spec defining the objective, the underlying service-level indicator query, and the burn-rate alerting thresholds) living in the SAME repository, and going through the SAME PR-and-review workflow, as the infrastructure they govern, deployed to the alerting/automation system by the SAME reconciliation mechanism, not a separate, dashboard-configured, un-versioned side channel. Because an SLO artifact directly controls automated behavior (paging thresholds, and in some organizations automated rollback triggers), its review bar and rollback path need to be at least as rigorous as the infrastructure changes it governs, arguably more so, since a wrong SLO can either cause alert fatigue (too tight) or silently miss real degradation (too loose).

Structured elaboration

Representation. A structured spec per service: the SLI (service-level indicator) query defining what is actually measured (a specific latency percentile, a specific error-rate calculation), the objective threshold and measurement window, and the burn-rate alert thresholds derived from it (fast-burn and slow-burn windows, following the widely-used multi-window burn-rate alerting pattern). This spec is data, not code, so it is reviewable as a clean diff the same way any other declarative configuration is.

Versioning. Committed to Git alongside (or in a clearly cross-referenced sibling path to) the service's own infrastructure definition, so git log on the SLO spec answers "when did this objective change and why" the same way it would for any other infra change, and a service's SLO history is auditable independent of whoever happens to remember the reasoning.

Deployment via Git workflows. A GitOps controller (or a dedicated SLO-config sync job, if the alerting platform is not itself Kubernetes-native) reconciles the deployed alerting-platform configuration to match the Git-declared spec, the same pull-based, continuously-reconciled model already used for application infrastructure generally, so a manually-edited alert threshold in the alerting platform's own UI gets treated as drift, not silently tolerated as a parallel source of truth.

Validation steps. Before merge: schema validation (is the spec well-formed), a SANITY check on the objective itself (is the target achievable given recent actual measured performance, catching an SLO set so tight it would page constantly from day one), and, where the alerting platform supports it, a DRY-RUN evaluation against recent historical data showing what the proposed burn-rate thresholds WOULD have alerted on over, say, the last 30 days, so a reviewer sees concretely whether the change would have caused excessive noise or a dangerous blind spot before it goes live.

Who should review SLO changes. The OWNING team proposes (they have the domain context for what "acceptable" performance means for their service), but a change LOOSENING an existing objective, or changing what a downstream automated system does in response to burn (see below), should require a second reviewer from OUTSIDE the owning team, typically the SRE or platform function with cross-service context, specifically because a team under pressure has an incentive to loosen its own SLO to escape being paged, which is exactly the review-conflict this second-reviewer requirement exists to catch.

How changes propagate to alerting and automation. The reconciler updates the ALERTING PLATFORM's configuration (the actual burn-rate alert rules) directly from the Git-declared spec; any DOWNSTREAM automation keyed off SLO/error-budget state (an automated deployment-freeze trigger when a budget is exhausted, for instance) reads the SAME reconciled objective, so there is exactly one source of truth for "what counts as within budget" feeding every consumer, rather than each consumer holding its own copy that could drift out of sync.

Rolling back an SLO change that causes undesired automation behavior. Because the spec is just a Git-tracked file, rollback is the same git revert plus reconciliation pattern as any other infra change; the practical difference is URGENCY, an SLO change causing a pager storm or, worse, an incorrectly-triggered automated rollback/freeze needs the same fast-path emergency-change mechanism as any infrastructure change: a short-lived, audited bypass with mandatory reconciliation back into Git afterward, since waiting for a normal-cadence PR review while pages fire is not acceptable.

Trade-offs and pitfalls

• Common mistake: configuring SLO thresholds directly in the alerting platform's UI "to iterate quickly," planning to formalize into Git later. This almost never gets formalized in practice, and it recreates exactly the un-auditable, un-reviewable side channel this whole answer exists to prevent; the dry-run-against-historical-data validation step exists specifically to make the Git-first path fast enough that UI-first iteration stops being tempting.
• The second-reviewer requirement for loosening an SLO is the single most-skipped control on this list under deadline pressure, precisely because it exists to catch the exact situation (a team wanting relief from its own pages) where the proposing team has the least incentive to enforce it on themselves.
• A DRY-RUN validation is only as good as the historical window it evaluates against. A recent quiet period can make an objectively too-loose threshold look safe; validating against a window that includes at least one known past incident, where the team can check "would this new threshold have caught it," is a meaningfully stronger check than an arbitrary recent-N-days window.
• Downstream automation consuming SLO state (deployment freezes, automated rollback triggers) needs to fail SAFE if the SLO reconciliation itself is stale or broken, an automation system that silently treats "no recent SLO data" as "budget is fine" can mask exactly the kind of problem it exists to catch; explicit staleness detection on the SLO data feed itself is a real, easy-to-omit requirement.

Direct answer

A drift-reconciliation tool has three genuinely separate jobs: read the tfstate's declared resources, query the CLOUD API for what actually exists for each one (handling throttling and credential failures as first-class, expected outcomes rather than crashes), and classify every discrepancy into one of four categories, MODIFIED (attributes disagree), MISSING (in state but gone from the cloud), UNMANAGED (exists in the cloud but absent from state), or UNCHANGED, then emit that classification as a structured JSON report. Below is a runnable Python implementation of all three, using a pinned FAKE cloud API (clearly marked as such) so the reconciliation LOGIC is exercised deterministically without depending on live AWS access.

Approach

1. Parse tfstate resources. Each tracked resource carries an address, type, id, and its DECLARED attributes.
2. Query the cloud API per resource, with bounded retry. describe_with_retry wraps the API call in a bounded exponential-backoff loop (attempt count tracked explicitly, backoff modeled structurally rather than with real sleeps so this demo's output stays fast and deterministic) that retries a RateLimitError up to max_attempts times, but does NOT retry a CredentialError, since retrying an expired or invalid credential just burns the retry budget on a guaranteed failure; credential errors are reported distinctly in the output instead.
3. Classify each tracked resource. None returned from the API means the resource is MISSING (gone from the cloud, still in state). A non-None result gets attribute-by-attribute compared against the declared values; any disagreement is recorded as MODIFIED with the specific field-level diff; no disagreement means UNCHANGED.
4. Detect unmanaged resources. For every resource TYPE the account is known to manage (not just the types present in THIS particular tfstate, a type with zero currently-tracked resources can still have untracked ones), query for resources of that type and flag any whose ID is not in the tracked set as UNMANAGED.
5. Emit a structured JSON drift report with four top-level categories plus an errors list for anything that failed to reconcile at all (credential failures, exhausted retries), so those are visible and actionable rather than silently dropped from the report.

Code

import json
import random
import time

class RateLimitError(Exception):
    pass

class CredentialError(Exception):
    pass

class FakeCloudAPI:
    """Stands in for a real AWS SDK client (e.g. boto3). Real code would call
    ec2.describe_instances / s3.list_buckets / etc; this fixture returns a
    FIXED, pinned live-resource map so the reconciliation logic below is
    exercised deterministically. It also deliberately injects a rate-limit
    error on the FIRST call for one resource type and a credential error for
    one specific lookup, so the retry/backoff and auth-error paths are
    actually exhausted by this run, not just written and never hit.
    """
    def __init__(self, seed=42):
        self.rng = random.Random(seed)
        self._ec2_call_count = 0
        self.live_resources = {
            # matches tfstate below except: sg-app has a drifted tag,
            # i-abandoned is MISSING (terminated outside Terraform),
            # bucket-untracked is UNMANAGED (exists in AWS, absent from state),
            # legacy is CredentialError (its account's read role has expired)
            "aws_instance.web": {"id": "i-0abc123", "type": "aws_instance",
                                  "attributes": {"instance_type": "t3.medium", "tags": {"env": "prod"}}},
            "aws_instance.abandoned": None,  # terminated outside Terraform -> MISSING
            "aws_security_group.app": {"id": "sg-0def456", "type": "aws_security_group",
                                        "attributes": {"ingress_ports": [443], "tags": {"env": "prod", "owner": "manual-edit"}}},
            "aws_s3_bucket.untracked": {"id": "bucket-untracked", "type": "aws_s3_bucket",
                                         "attributes": {"versioning": True}},
        }

    def describe(self, resource_type, resource_id, tf_address):
        if tf_address == "aws_instance.legacy":
            raise CredentialError("ExpiredTokenException: the security token included in the request is expired")
        if resource_type == "aws_instance" and self._ec2_call_count == 0:
            self._ec2_call_count += 1
            raise RateLimitError(f"Throttling: rate exceeded describing {resource_type}")
        return self.live_resources.get(tf_address)

    def list_untracked(self, resource_type):
        # returns resources of this type that exist in the cloud but were
        # never in the tfstate map we were given (an "unmanaged" resource)
        if resource_type == "aws_s3_bucket":
            return [{"id": "bucket-untracked", "type": "aws_s3_bucket",
                     "attributes": self.live_resources["aws_s3_bucket.untracked"]["attributes"]}]
        return []


TFSTATE = {
    "resources": [
        {"address": "aws_instance.web", "type": "aws_instance", "id": "i-0abc123",
         "attributes": {"instance_type": "t3.medium", "tags": {"env": "prod"}}},
        {"address": "aws_instance.abandoned", "type": "aws_instance", "id": "i-0stale789",
         "attributes": {"instance_type": "t3.small", "tags": {"env": "staging"}}},
        {"address": "aws_security_group.app", "type": "aws_security_group", "id": "sg-0def456",
         "attributes": {"ingress_ports": [443], "tags": {"env": "prod"}}},
        {"address": "aws_instance.legacy", "type": "aws_instance", "id": "i-legacy999",
         "attributes": {"instance_type": "t2.micro", "tags": {"env": "legacy"}}},
    ]
}


def describe_with_retry(api, resource_type, resource_id, tf_address, max_attempts=3):
    """Bounded retry with exponential backoff for transient throttling.
    Credential errors are NOT retried (retrying an expired/invalid
    credential just wastes the remaining attempts on a guaranteed failure);
    they are raised immediately so the caller can report them distinctly
    from a genuine drift finding."""
    attempt = 0
    while True:
        attempt += 1
        try:
            return api.describe(resource_type, resource_id, tf_address)
        except RateLimitError as e:
            if attempt >= max_attempts:
                raise
            # backoff modeled structurally (attempt number recorded), not
            # timed, so this demo's output is deterministic and fast
            continue
        except CredentialError:
            raise


def reconcile(tfstate, api, known_resource_types):
    """known_resource_types: every resource TYPE this account manages, not just
    the ones present in tfstate -- a type with ZERO tracked resources of that
    kind can still have untracked ones in the cloud, so it has to be scanned
    for "unmanaged" resources even though the loop below never visits it."""
    report = {"modified": [], "missing": [], "unmanaged": [], "errors": [], "unchanged": []}
    tracked_ids_by_type = {t: set() for t in known_resource_types}

    for res in tfstate["resources"]:
        addr, rtype, rid = res["address"], res["type"], res["id"]
        tracked_ids_by_type.setdefault(rtype, set()).add(rid)
        try:
            live = describe_with_retry(api, rtype, rid, addr)
        except CredentialError as e:
            report["errors"].append({"address": addr, "error": str(e)})
            continue
        except RateLimitError as e:
            report["errors"].append({"address": addr, "error": f"exhausted retries: {e}"})
            continue

        if live is None:
            report["missing"].append({"address": addr, "id": rid,
                                       "note": "in tfstate but not found in the cloud API response"})
            continue

        diffs = {}
        for key, declared_value in res["attributes"].items():
            live_value = live["attributes"].get(key)
            if live_value != declared_value:
                diffs[key] = {"declared": declared_value, "live": live_value}
        if diffs:
            report["modified"].append({"address": addr, "id": rid, "diffs": diffs})
        else:
            report["unchanged"].append(addr)

    for rtype, tracked_ids in tracked_ids_by_type.items():
        for live_res in api.list_untracked(rtype):
            if live_res["id"] not in tracked_ids:
                report["unmanaged"].append(live_res)

    return report


if __name__ == "__main__":
    api = FakeCloudAPI(seed=42)
    report = reconcile(TFSTATE, api, known_resource_types=["aws_instance", "aws_security_group", "aws_s3_bucket"])
    print(json.dumps(report, indent=2, sort_keys=True))

    print("\n=== Assertions ===")
    assert len(report["modified"]) == 1 and report["modified"][0]["address"] == "aws_security_group.app"
    print("security group tag drift detected:", report["modified"][0]["diffs"])
    assert len(report["missing"]) == 1 and report["missing"][0]["address"] == "aws_instance.abandoned"
    print("terminated-outside-Terraform instance detected as MISSING:", report["missing"][0]["address"])
    assert len(report["unmanaged"]) == 1 and report["unmanaged"][0]["id"] == "bucket-untracked"
    print("untracked bucket detected as UNMANAGED:", report["unmanaged"][0]["id"])
    assert "aws_instance.web" in report["unchanged"]
    print("unchanged instance correctly reported as no drift:", "aws_instance.web" in report["unchanged"])
    assert len(report["errors"]) == 1 and report["errors"][0]["address"] == "aws_instance.legacy"
    print("credential error correctly reported as its own category, not folded into drift or silently dropped:",
          report["errors"][0])
    print("\nBoth the rate-limit retry path (first EC2 describe call raised RateLimitError and was retried "
          "successfully) and the credential-error path (aws_instance.legacy) were genuinely exercised by this run.")

Output (actually executed with python3 s43_drift.py)

{
  "errors": [
    {
      "address": "aws_instance.legacy",
      "error": "ExpiredTokenException: the security token included in the request is expired"
    }
  ],
  "missing": [
    {
      "address": "aws_instance.abandoned",
      "id": "i-0stale789",
      "note": "in tfstate but not found in the cloud API response"
    }
  ],
  "modified": [
    {
      "address": "aws_security_group.app",
      "diffs": {
        "tags": {
          "declared": {
            "env": "prod"
          },
          "live": {
            "env": "prod",
            "owner": "manual-edit"
          }
        }
      },
      "id": "sg-0def456"
    }
  ],
  "unchanged": [
    "aws_instance.web"
  ],
  "unmanaged": [
    {
      "attributes": {
        "versioning": true
      },
      "id": "bucket-untracked",
      "type": "aws_s3_bucket"
    }
  ]
}

=== Assertions ===
security group tag drift detected: {'tags': {'declared': {'env': 'prod'}, 'live': {'env': 'prod', 'owner': 'manual-edit'}}}
terminated-outside-Terraform instance detected as MISSING: aws_instance.abandoned
untracked bucket detected as UNMANAGED: bucket-untracked
unchanged instance correctly reported as no drift: True
credential error correctly reported as its own category, not folded into drift or silently dropped: {'address': 'aws_instance.legacy', 'error': 'ExpiredTokenException: the security token included in the request is expired'}

Both the rate-limit retry path (first EC2 describe call raised RateLimitError and was retried successfully) and the credential-error path (aws_instance.legacy) were genuinely exercised by this run.


The fixture is pinned to exercise all four classification paths plus the retry and credential-handling paths in one run: `aws_security_group.app`'s live tags include an extra `owner: manual-edit` key not in the declared attributes, correctly reported as MODIFIED with the exact field diff; `aws_instance.abandoned` returns `None` from the fake API (modeling a termination outside Terraform), correctly reported as MISSING; `bucket-untracked` exists in the fake API's inventory but has no corresponding tfstate entry, correctly reported as UNMANAGED; `aws_instance.web` matches exactly and is reported UNCHANGED; the fake API is rigged to raise `RateLimitError` on the FIRST EC2-type describe call specifically, which the retry logic caught and recovered from transparently; and `aws_instance.legacy` is rigged to raise `CredentialError` on every call, correctly recorded as its own `errors` entry rather than being retried or folded into `missing`, confirming both the retry path and the credential-error path actually executed in this run rather than being written but never exercised.

## Key points

- Classifying MISSING versus UNMANAGED requires querying in BOTH directions, tracked-resource-by-resource lookups catch MISSING; a separate listing call per resource type catches UNMANAGED. A tool that only does the first direction (which is the more obvious one to implement) systematically misses every untracked resource, silently under-reporting real drift.
- Credential errors and rate-limit errors need DIFFERENT handling, not a single generic "retry on any exception" catch-all: retrying a credential error wastes the retry budget on something retrying cannot fix, while NOT retrying a genuine transient throttle would report false drift/errors for what was actually a recoverable, momentary condition.
- Reporting errors as their OWN category, separate from MODIFIED/MISSING/UNMANAGED, matters because a resource the tool COULD NOT CHECK is not the same claim as a resource confirmed unchanged; conflating "we don't know" with "no drift" would produce a report that looks more complete and reassuring than the run actually was.

## Complexity

- Time: $$O(R \cdot A)$$ where $$R$$ is the number of tracked resources and $$A$$ is the maximum retry attempts per resource, plus $$O(T)$$ for the untracked-resource listing calls where $$T$$ is the number of distinct resource types managed.
- Space: $$O(R + U)$$ for the report, where $$U$$ is the number of unmanaged resources discovered, both bounded by the actual inventory size, not by anything unbounded in the algorithm itself.

## Edge cases

- **A resource type with zero tracked resources in this particular tfstate but real untracked resources in the cloud:** correctly caught, since `known_resource_types` is passed explicitly rather than derived only from what happens to be in THIS tfstate.
- **Every retry attempt for a resource exhausted without success:** the exception propagates out of `describe_with_retry` and is caught at the reconciliation level, recorded in `errors` rather than crashing the whole reconciliation run over one resource's persistent failure.
- **A resource with an empty attributes dict on either side:** the field-by-field comparison loop simply finds no keys to compare, correctly falling through to UNCHANGED rather than erroring on an empty iteration.