Infrastructure as Code and GitOps Questions

Direct answer

Treat drift detection as one CROSS-CUTTING capability applied to two genuinely different populations rather than one problem: IaC-managed resources are checked by comparing live state against the Terraform/Ansible state that already declares their desired configuration, while manually configured VMs have NO desired-state declaration to compare against yet, so the real first step for that population is DISCOVERY (inventory what exists) before drift detection is even possible. The staged migration then moves manual hosts, one cohort at a time, from "undeclared" to "declared but not yet enforced" to "declared and enforced," never skipping the middle step, since jumping straight to enforcement on a host whose current configuration was never actually captured risks a disruptive first-apply that reverts real, load-bearing settings nobody documented.

Structured elaboration

Tooling options, by population.

• IaC-managed resources: native drift detection (terraform plan run on a schedule, diffed against the last-applied state; Ansible's --check mode against its inventory) reports drift directly, since desired state already exists.
• Manually configured VMs, discovery phase: an agentless configuration-scanning tool (Ansible ad hoc facts-gathering, osquery, or a cloud provider's config-inventory service for cloud VMs) builds a first-pass inventory of installed packages, running services, open ports, and key config files, WITHOUT changing anything, purely observational.
• Manually configured VMs, once inventoried: generate an initial Ansible playbook or Terraform import block FROM the discovered state (rather than hand-writing desired state from scratch and hoping it matches), so the very first "desired state" declaration is a faithful snapshot of what is actually running, minimizing the chance the first apply changes anything unexpectedly.

Risk mitigation, the core discipline. Every newly onboarded host goes through an AUDIT-ONLY period before any enforcement is enabled: the tool reports what it WOULD change (a dry-run plan, Ansible --check --diff) without applying it, and a human reviews that diff specifically looking for surprises, anything the discovery phase missed or mis-captured. Only after a clean audit period (the dry-run plan stabilizes and shows no unexpected changes across a few consecutive runs) does the host move to enforcement (auto-apply or scheduled apply). This ordering, discover then audit-only then enforce, is what prevents the single biggest risk in this kind of migration: applying a freshly authored desired state against a host whose ACTUAL current configuration was captured incompletely.

Staged rollout plan.

1. Pilot cohort (5 to 10 percent of manual hosts, lowest business risk). Full discover, audit-only, enforce cycle on hosts where a mistake is cheap to recover from (non-production, or production hosts with strong redundancy). Use this cohort to tune the discovery tooling's accuracy before touching anything higher-stakes.
2. Expand by risk tier, not by convenience. Move to the next tier (higher-traffic but still redundant services) only after the pilot cohort has run cleanly under enforcement for a defined bake period (for example two full weeks with zero unexpected reverts), applying the SAME discover, audit, enforce sequence, not skipping steps because the pilot went well.
3. Highest-risk tier last (stateful, low-redundancy, or compliance-sensitive hosts). These get the audit-only period extended and often a manual sign-off gate before enforcement is switched on, since an unexpected revert here has the highest cost.
4. Continuous cohort, not a one-time project. Once the existing fleet is migrated, any NEWLY provisioned host must be born already IaC-managed (provisioned via the same pipeline from day one), so the "manual host" population only shrinks and never silently regrows from new manual provisioning.

Worked example

A fleet of 400 hosts: 250 already Terraform/Ansible-managed, 150 manually configured across on-prem and two cloud providers. Applying the plan: discovery scans all 150 manual hosts (roughly a week, agentless, zero risk since it only reads). Of those, 90 are low-risk (dev/staging, redundant web tier) and become the pilot plus first expansion cohort; each goes through a two-week audit-only period, and 84 of the 90 show a clean, stable dry-run plan by week two (the other 6 have discovery gaps, unexpected cron jobs and locally-installed packages the initial scan missed, and get their captured state manually corrected before proceeding). Those 84 move to enforcement. The remaining 60 (production databases, a compliance-scoped payment-processing tier) get a four-week audit-only period plus a manual review gate before enforcement, migrating in the final stage. Total migration: roughly 8 to 10 weeks for the full 150-host population, moving in three risk-ordered cohorts rather than one flat cutover.

Trade-offs and pitfalls

• Common mistake: writing the manual hosts' desired-state declaration by hand from documentation instead of generating it from discovered actual state. Documentation is frequently stale on manually configured hosts precisely BECAUSE nobody has been enforcing it; a hand-written desired state built from stale docs will diverge from reality in ways the audit-only period exists specifically to catch, but skipping straight to enforcement without that period turns every documentation gap into a live incident.
• Common mistake: enforcing on the ENTIRE manual-host population at once "since discovery went well for the sample we checked." The pilot cohort's success does not generalize automatically; different host tiers commonly have different undocumented local customizations, and the risk-ordered staged rollout exists precisely because a clean pilot is evidence about the PILOT, not proof the whole population is equally safe to enforce on simultaneously.
• The audit-only bake period needs an explicit, pre-agreed exit criterion (for example, N consecutive clean dry-runs), not a vague "looks stable." Without a concrete bar, teams under schedule pressure tend to shorten the audit period informally, which reintroduces exactly the risk the staged plan is designed to manage.
• On-prem and cloud VMs often need genuinely different discovery tooling (cloud providers frequently expose a native config-inventory API that is faster and more complete than a generic agentless scan, while on-prem hosts may only be reachable via SSH-based fact-gathering); treating "detect drift across clouds and on-prem" as one uniform tooling choice rather than the right tool per environment slows the discovery phase down without a compensating benefit.

Direct answer

Enforce release gates at TWO complementary layers: the pull request itself (peer/approver review, required status checks, branch protection rules) as the primary compliance-auditable approval, and Argo CD's sync policy (manual sync for production, or SyncWindows restricting when auto-sync can fire) as a technical backstop that prevents a merged-but-not-yet-approved-for-deployment change from auto-applying. The key design constraint: human approval happens on the PR (a Git-native, naturally auditable event with reviewer identity and timestamp already captured), never as an out-of-band manual step performed against the live cluster, which is what would actually break the declarative, Git-is-source-of-truth principle.

Structured elaboration

Where gates are enforced. CI enforces AUTOMATED gates (linting, terraform plan/manifest-diff generation, policy-as-code checks, security scanning) as required status checks that must pass before merge is even possible. The PR itself enforces the HUMAN approval gate via required reviewers (branch protection rules requiring N approvals from a CODEOWNERS-defined group for production-targeting paths). Argo CD enforces a FINAL technical gate: production Application objects configured with syncPolicy.automated OMITTED (manual sync only) or restricted via SyncWindows, so even a merged commit doesn't immediately roll out until an authorized operator (or an automated promotion step gated on its own separate approval) triggers the sync.

What metadata should live in Git. The PR itself is the metadata record: title/description linking to the change ticket, the diff itself (what actually changed), the list of approvers and their approval timestamps (captured natively by the Git hosting platform, exportable via API for audit), and any required-context status check results (policy-as-code pass/fail, security scan results) attached to the commit. For compliance frameworks requiring a documented "why" beyond the diff, a structured commit message or PR template field (change ticket ID, risk assessment, rollback plan) keeps that context IN the same auditable artifact rather than in a separate system that can drift out of sync with what was actually deployed.

Implementing human approval without breaking declarative principles. The subtlety: "declarative" means the DESIRED STATE is fully specified in Git and reconciliation is automatic given that state, not that every state transition must be fully automatic with no human in the loop. A human approval gate is compatible with GitOps as long as the approval ITSELF is captured declaratively (the merged PR, an approved state) and the reconciler then acts deterministically on whatever is in Git; what would BREAK the principle is a human directly editing the live cluster to "approve" a change out of band, since that reintroduces exactly the drift-inducing manual intervention GitOps exists to eliminate. A common, clean pattern: a promotion PR (bumping an image tag or config value in the production overlay) requires its own separate, higher-bar approval from the PR that merged the change into staging, so "promote to production" is itself a reviewed, auditable Git event, not a live cluster action.

Trade-offs and pitfalls

• Common mistake: implementing the "human approval" step as a manual argocd app sync performed by an operator outside of any recorded process. This satisfies "a human looked at it" but produces NO durable audit record of WHO approved WHAT and WHY beyond whatever is (or isn't) in a chat log; the PR-based approval model is specifically what makes the approval auditable without extra tooling, since the Git platform already records it.
• Common mistake: conflating CI's automated checks with the compliance-required human approval. A passing lint/policy-as-code check is necessary but not sufficient for most compliance frameworks, which specifically require a human decision-maker's sign-off; treating "all checks green" as equivalent to "approved" removes the human judgment step the compliance requirement exists to capture.
• SyncWindows and manual-sync settings are a technical backstop, not a substitute for PR-level review. They prevent an ALREADY-MERGED change from auto-deploying at the wrong time, but by the time a change is merged, the PR-level approval should already have happened; relying on the sync gate as the PRIMARY approval mechanism (skipping rigorous PR review because "someone will catch it before syncing") inverts the intended order and risks a rushed, under-reviewed sync-time approval replacing a properly reviewed PR.
• CODEOWNERS-based required-reviewer rules need to be scoped to the RIGHT paths. A blanket "any 2 approvers" rule does not satisfy most compliance requirements for segregation of duties if the two approvers can be the same person who wrote the change plus a rubber-stamping teammate; production-path changes typically need a reviewer who is NOT the author and, in stricter regimes, a reviewer from a DIFFERENT team or a designated compliance role.

Direct answer

Convergence is the property that a system, given a fixed desired state and enough reconciliation cycles, will reach a state that MATCHES that desired state and then stop changing; eventual consistency is the weaker, related guarantee that if updates stop arriving, all observers will EVENTUALLY see the same value, without a bound on how long "eventually" takes. A converge-loop system (any GitOps controller, or a Kubernetes controller generally) combines both: it is eventually consistent in how the cluster's observed state settles after a change, and it is CONVERGENT in the stronger sense that its target (Git-declared desired state) is fixed and known, not merely "whatever the last writer said."

Structured elaboration

Concrete converge-loop example. Argo CD's reconciliation loop: on each cycle, it computes a diff between the live cluster and the Git-declared manifest, applies the difference, and re-checks. If nothing else is changing the cluster concurrently, this loop converges: the diff shrinks to nothing and stays there. If the loop runs again on an already-converged system, it does no work, this idempotent "no-op when already matching" behavior is itself a requirement for convergence to be a meaningful, stable end state rather than an accidental one-time alignment.

Failure modes where convergence may never be achieved.

• A conflicting external actor continuously undoes the reconciler's work (a separate automation, or a human, repeatedly reverting the exact field the reconciler is trying to set); the loop keeps correcting, the external actor keeps un-correcting, and the system oscillates indefinitely rather than settling.
• The desired state itself is invalid or unsatisfiable (a manifest requesting a resource quantity the cluster cannot schedule, or two declared resources with a genuine, unresolvable dependency cycle); the reconciler cannot converge to something that cannot exist, and without an explicit failure signal, it may simply retry forever.
• A transient dependency never becomes available (waiting on a resource that some OTHER, un-reconciled system is supposed to create, a manually-provisioned prerequisite that was never actually done); the loop is technically doing everything correctly but is blocked on an external precondition outside its own control.
• Reconciliation itself has a bug that makes each pass compute a slightly different "desired" outcome (non-deterministic ordering in how a diff is computed, or a template that renders differently depending on unrelated cluster state); each cycle "succeeds" at applying SOMETHING, but that something keeps changing, so the system never settles on one final state.

Mitigations for each.

• For a conflicting external actor: identify and eliminate the other write path (this is exactly the discipline behind Git-as-sole-source-of-truth); if the other actor is legitimate, its intent needs to be captured in the SAME desired-state definition the reconciler reads, not left to fight over the live object out-of-band.
• For an unsatisfiable desired state: validate BEFORE applying (a policy-as-code check, a schema validator, a dry-run) so an impossible desired state is rejected at PR time rather than discovered as an infinitely-retrying reconciler in production; and bound retries with backoff plus an alert after N consecutive failed attempts, so a genuinely stuck reconciliation surfaces to a human instead of looping silently forever.
• For a blocked external dependency: make the dependency EXPLICIT and reconciled by the SAME system where possible (bring the manual prerequisite under the same declarative management), or add an explicit readiness/precondition check the reconciler can report on clearly rather than retrying blindly.
• For non-deterministic reconciliation: pin every source of variability (sort collections before diffing, use content-addressed references like image digests rather than resolved-at-apply-time values) so the SAME desired state always produces the SAME computed diff.

Trade-offs and pitfalls

• Common mistake: treating "the reconciler is still running" as evidence it is making progress. A reconciler stuck in the conflicting-external-actor failure mode is BUSY (constantly reapplying) but never converging; the useful signal is not "is it running" but "is the size of the diff between live and desired state trending toward zero and staying there," which needs to be an EXPLICIT metric, not inferred from activity level.
• Bounding retries with backoff-and-alert is a mitigation for VISIBILITY, not a fix for the underlying cause. An alert after repeated failed reconciliation tells a human where to look; it does not itself resolve an unsatisfiable desired state or a fighting external actor, both of which need a human or a process change, not a tighter retry loop.
• Eventual consistency's lack of a time bound is a real, not just theoretical, operational risk. A system that is technically guaranteed to converge "eventually" but takes hours to do so under a specific failure pattern can still cause a real incident if nothing alerts on convergence taking longer than expected; "will eventually converge" and "converges fast enough to matter" are different claims, and only the metric-based bound (not the theoretical guarantee) tells you which one you actually have.

Direct answer

A GitOps workflow with multi-environment promotion needs the environment SEPARATION and the PROMOTION MECHANISM to be two distinct, deliberately-designed pieces, not one implicit consequence of "we have several folders": separate repos or branches per environment (or, more commonly in current practice, separate DIRECTORIES/overlays within one repo, watched by each environment's own controller Application/Kustomization object) give each environment its own independently-reconciled desired state, while promotion is an EXPLICIT, reviewed act of copying a specific, already-validated change from one environment's declared state into the next, never an automatic cascade.

Structured elaboration

Separate repos/branches per environment, and why directory-based overlays are usually preferred instead. Full separate REPOS per environment gives the strongest isolation (different access control per environment's repo) but makes promotion mechanically harder (a cross-repo PR is not a first-class Git concept) and duplicates shared base configuration unless carefully factored out. Separate BRANCHES per environment (a staging branch, a production branch) makes promotion a simple merge/cherry-pick, but branches are mutable, ongoing streams, not a natural fit for "environment X's current desired state" as a single, always-current thing to look AT rather than merge INTO. The pattern most GitOps tooling and teams converge on: ONE repo, per-environment directories (Kustomize overlays), each watched independently by that environment's controller instance/object; promotion becomes a PR modifying the target environment's directory (usually bumping a pinned version/digest reference), which keeps the "one repo to look at for anything" ergonomics while still giving each environment its own independently-reconciled, independently-reviewable state.

Automated drift detection. Each environment's controller instance runs its own independent reconciliation loop against its own directory; drift in staging and drift in production are detected and handled completely independently (an environment-specific selfHeal policy, likely more permissive in staging than production).

Controlled promotion mechanism. CI automatically opens a PR bumping the NEXT environment's directory whenever the CURRENT environment's deployed state has been stable for a defined bake period (or on an explicit manual trigger), referencing the exact, already-built artifact (an image digest) that was validated in the prior environment, never a fresh build; each promotion PR requires its own review, with production's promotion PR typically requiring a stricter approval bar (more reviewers, or a specific approver group) than staging's.

Secrets handling across environments. Each environment's secrets are scoped to that environment's own path in the secrets backend (an External Secrets Operator instance per environment reading from its own scoped path prefix), so a promotion PR bumping an application version never needs to touch or reference secret VALUES directly, the secret REFERENCE structure stays the same across environments while the underlying value differs per environment automatically.

Hotfixes that must skip standard promotion. A defined, narrow exception path: a hotfix PR targets PRODUCTION's directory directly (skipping the normal dev-to-staging-to-production sequence), still going through the SAME review and CI validation as any other change to that directory, just without first passing through the earlier environments; critically, the hotfix commit needs to be BACK-PROPAGATED into the earlier environments' directories afterward (a follow-up PR bringing dev and staging back in sync with whatever the hotfix changed), so the earlier environments do not silently regress relative to production and reintroduce the same bug on the next normal promotion.

Trade-offs and pitfalls

• Common mistake: implementing "promotion" as a script that copies the entire staging directory's contents into production's, rather than promoting a specific, already-validated artifact reference. This risks silently promoting something that was NEVER actually validated in staging (an unrelated concurrent change to staging's directory that happened to be present at copy time); promotion should reference the SPECIFIC artifact/commit that was actually tested, not "whatever staging currently looks like."
• The hotfix back-propagation step is the piece most often skipped once the immediate fire is out, exactly the same failure pattern as any emergency-change reconciliation gap; without a tracked follow-up task (not just a runbook suggestion), earlier environments quietly drift out of sync with production's actual, hotfixed state.
• Per-environment secrets-backend path scoping (External Secrets Operator per environment, scoped to that environment's own prefix) is what prevents a promotion PR from ever needing to touch a secret VALUE, if this scoping is not done carefully, promotion PRs can end up needing environment-specific secret handling bolted on ad hoc, defeating the goal of promotion being a clean, generic mechanism.
• A stricter approval bar for production's promotion PR only provides real protection if it is enforced structurally (branch protection / required reviewers on that specific directory path), not merely documented as expected practice; the same CODEOWNERS-and-branch-protection discipline that makes any change auditable applies directly here.

Direct answer

A drift-reconciliation tool has three genuinely separate jobs: read the tfstate's declared resources, query the CLOUD API for what actually exists for each one (handling throttling and credential failures as first-class, expected outcomes rather than crashes), and classify every discrepancy into one of four categories, MODIFIED (attributes disagree), MISSING (in state but gone from the cloud), UNMANAGED (exists in the cloud but absent from state), or UNCHANGED, then emit that classification as a structured JSON report. Below is a runnable Python implementation of all three, using a pinned FAKE cloud API (clearly marked as such) so the reconciliation LOGIC is exercised deterministically without depending on live AWS access.

Approach

1. Parse tfstate resources. Each tracked resource carries an address, type, id, and its DECLARED attributes.
2. Query the cloud API per resource, with bounded retry. describe_with_retry wraps the API call in a bounded exponential-backoff loop (attempt count tracked explicitly, backoff modeled structurally rather than with real sleeps so this demo's output stays fast and deterministic) that retries a RateLimitError up to max_attempts times, but does NOT retry a CredentialError, since retrying an expired or invalid credential just burns the retry budget on a guaranteed failure; credential errors are reported distinctly in the output instead.
3. Classify each tracked resource. None returned from the API means the resource is MISSING (gone from the cloud, still in state). A non-None result gets attribute-by-attribute compared against the declared values; any disagreement is recorded as MODIFIED with the specific field-level diff; no disagreement means UNCHANGED.
4. Detect unmanaged resources. For every resource TYPE the account is known to manage (not just the types present in THIS particular tfstate, a type with zero currently-tracked resources can still have untracked ones), query for resources of that type and flag any whose ID is not in the tracked set as UNMANAGED.
5. Emit a structured JSON drift report with four top-level categories plus an errors list for anything that failed to reconcile at all (credential failures, exhausted retries), so those are visible and actionable rather than silently dropped from the report.

Code

import json
import random
import time

class RateLimitError(Exception):
    pass

class CredentialError(Exception):
    pass

class FakeCloudAPI:
    """Stands in for a real AWS SDK client (e.g. boto3). Real code would call
    ec2.describe_instances / s3.list_buckets / etc; this fixture returns a
    FIXED, pinned live-resource map so the reconciliation logic below is
    exercised deterministically. It also deliberately injects a rate-limit
    error on the FIRST call for one resource type and a credential error for
    one specific lookup, so the retry/backoff and auth-error paths are
    actually exhausted by this run, not just written and never hit.
    """
    def __init__(self, seed=42):
        self.rng = random.Random(seed)
        self._ec2_call_count = 0
        self.live_resources = {
            # matches tfstate below except: sg-app has a drifted tag,
            # i-abandoned is MISSING (terminated outside Terraform),
            # bucket-untracked is UNMANAGED (exists in AWS, absent from state),
            # legacy is CredentialError (its account's read role has expired)
            "aws_instance.web": {"id": "i-0abc123", "type": "aws_instance",
                                  "attributes": {"instance_type": "t3.medium", "tags": {"env": "prod"}}},
            "aws_instance.abandoned": None,  # terminated outside Terraform -> MISSING
            "aws_security_group.app": {"id": "sg-0def456", "type": "aws_security_group",
                                        "attributes": {"ingress_ports": [443], "tags": {"env": "prod", "owner": "manual-edit"}}},
            "aws_s3_bucket.untracked": {"id": "bucket-untracked", "type": "aws_s3_bucket",
                                         "attributes": {"versioning": True}},
        }

    def describe(self, resource_type, resource_id, tf_address):
        if tf_address == "aws_instance.legacy":
            raise CredentialError("ExpiredTokenException: the security token included in the request is expired")
        if resource_type == "aws_instance" and self._ec2_call_count == 0:
            self._ec2_call_count += 1
            raise RateLimitError(f"Throttling: rate exceeded describing {resource_type}")
        return self.live_resources.get(tf_address)

    def list_untracked(self, resource_type):
        # returns resources of this type that exist in the cloud but were
        # never in the tfstate map we were given (an "unmanaged" resource)
        if resource_type == "aws_s3_bucket":
            return [{"id": "bucket-untracked", "type": "aws_s3_bucket",
                     "attributes": self.live_resources["aws_s3_bucket.untracked"]["attributes"]}]
        return []


TFSTATE = {
    "resources": [
        {"address": "aws_instance.web", "type": "aws_instance", "id": "i-0abc123",
         "attributes": {"instance_type": "t3.medium", "tags": {"env": "prod"}}},
        {"address": "aws_instance.abandoned", "type": "aws_instance", "id": "i-0stale789",
         "attributes": {"instance_type": "t3.small", "tags": {"env": "staging"}}},
        {"address": "aws_security_group.app", "type": "aws_security_group", "id": "sg-0def456",
         "attributes": {"ingress_ports": [443], "tags": {"env": "prod"}}},
        {"address": "aws_instance.legacy", "type": "aws_instance", "id": "i-legacy999",
         "attributes": {"instance_type": "t2.micro", "tags": {"env": "legacy"}}},
    ]
}


def describe_with_retry(api, resource_type, resource_id, tf_address, max_attempts=3):
    """Bounded retry with exponential backoff for transient throttling.
    Credential errors are NOT retried (retrying an expired/invalid
    credential just wastes the remaining attempts on a guaranteed failure);
    they are raised immediately so the caller can report them distinctly
    from a genuine drift finding."""
    attempt = 0
    while True:
        attempt += 1
        try:
            return api.describe(resource_type, resource_id, tf_address)
        except RateLimitError as e:
            if attempt >= max_attempts:
                raise
            # backoff modeled structurally (attempt number recorded), not
            # timed, so this demo's output is deterministic and fast
            continue
        except CredentialError:
            raise


def reconcile(tfstate, api, known_resource_types):
    """known_resource_types: every resource TYPE this account manages, not just
    the ones present in tfstate -- a type with ZERO tracked resources of that
    kind can still have untracked ones in the cloud, so it has to be scanned
    for "unmanaged" resources even though the loop below never visits it."""
    report = {"modified": [], "missing": [], "unmanaged": [], "errors": [], "unchanged": []}
    tracked_ids_by_type = {t: set() for t in known_resource_types}

    for res in tfstate["resources"]:
        addr, rtype, rid = res["address"], res["type"], res["id"]
        tracked_ids_by_type.setdefault(rtype, set()).add(rid)
        try:
            live = describe_with_retry(api, rtype, rid, addr)
        except CredentialError as e:
            report["errors"].append({"address": addr, "error": str(e)})
            continue
        except RateLimitError as e:
            report["errors"].append({"address": addr, "error": f"exhausted retries: {e}"})
            continue

        if live is None:
            report["missing"].append({"address": addr, "id": rid,
                                       "note": "in tfstate but not found in the cloud API response"})
            continue

        diffs = {}
        for key, declared_value in res["attributes"].items():
            live_value = live["attributes"].get(key)
            if live_value != declared_value:
                diffs[key] = {"declared": declared_value, "live": live_value}
        if diffs:
            report["modified"].append({"address": addr, "id": rid, "diffs": diffs})
        else:
            report["unchanged"].append(addr)

    for rtype, tracked_ids in tracked_ids_by_type.items():
        for live_res in api.list_untracked(rtype):
            if live_res["id"] not in tracked_ids:
                report["unmanaged"].append(live_res)

    return report


if __name__ == "__main__":
    api = FakeCloudAPI(seed=42)
    report = reconcile(TFSTATE, api, known_resource_types=["aws_instance", "aws_security_group", "aws_s3_bucket"])
    print(json.dumps(report, indent=2, sort_keys=True))

    print("\n=== Assertions ===")
    assert len(report["modified"]) == 1 and report["modified"][0]["address"] == "aws_security_group.app"
    print("security group tag drift detected:", report["modified"][0]["diffs"])
    assert len(report["missing"]) == 1 and report["missing"][0]["address"] == "aws_instance.abandoned"
    print("terminated-outside-Terraform instance detected as MISSING:", report["missing"][0]["address"])
    assert len(report["unmanaged"]) == 1 and report["unmanaged"][0]["id"] == "bucket-untracked"
    print("untracked bucket detected as UNMANAGED:", report["unmanaged"][0]["id"])
    assert "aws_instance.web" in report["unchanged"]
    print("unchanged instance correctly reported as no drift:", "aws_instance.web" in report["unchanged"])
    assert len(report["errors"]) == 1 and report["errors"][0]["address"] == "aws_instance.legacy"
    print("credential error correctly reported as its own category, not folded into drift or silently dropped:",
          report["errors"][0])
    print("\nBoth the rate-limit retry path (first EC2 describe call raised RateLimitError and was retried "
          "successfully) and the credential-error path (aws_instance.legacy) were genuinely exercised by this run.")

Output (actually executed with python3 s43_drift.py)

{
  "errors": [
    {
      "address": "aws_instance.legacy",
      "error": "ExpiredTokenException: the security token included in the request is expired"
    }
  ],
  "missing": [
    {
      "address": "aws_instance.abandoned",
      "id": "i-0stale789",
      "note": "in tfstate but not found in the cloud API response"
    }
  ],
  "modified": [
    {
      "address": "aws_security_group.app",
      "diffs": {
        "tags": {
          "declared": {
            "env": "prod"
          },
          "live": {
            "env": "prod",
            "owner": "manual-edit"
          }
        }
      },
      "id": "sg-0def456"
    }
  ],
  "unchanged": [
    "aws_instance.web"
  ],
  "unmanaged": [
    {
      "attributes": {
        "versioning": true
      },
      "id": "bucket-untracked",
      "type": "aws_s3_bucket"
    }
  ]
}

=== Assertions ===
security group tag drift detected: {'tags': {'declared': {'env': 'prod'}, 'live': {'env': 'prod', 'owner': 'manual-edit'}}}
terminated-outside-Terraform instance detected as MISSING: aws_instance.abandoned
untracked bucket detected as UNMANAGED: bucket-untracked
unchanged instance correctly reported as no drift: True
credential error correctly reported as its own category, not folded into drift or silently dropped: {'address': 'aws_instance.legacy', 'error': 'ExpiredTokenException: the security token included in the request is expired'}

Both the rate-limit retry path (first EC2 describe call raised RateLimitError and was retried successfully) and the credential-error path (aws_instance.legacy) were genuinely exercised by this run.


The fixture is pinned to exercise all four classification paths plus the retry and credential-handling paths in one run: `aws_security_group.app`'s live tags include an extra `owner: manual-edit` key not in the declared attributes, correctly reported as MODIFIED with the exact field diff; `aws_instance.abandoned` returns `None` from the fake API (modeling a termination outside Terraform), correctly reported as MISSING; `bucket-untracked` exists in the fake API's inventory but has no corresponding tfstate entry, correctly reported as UNMANAGED; `aws_instance.web` matches exactly and is reported UNCHANGED; the fake API is rigged to raise `RateLimitError` on the FIRST EC2-type describe call specifically, which the retry logic caught and recovered from transparently; and `aws_instance.legacy` is rigged to raise `CredentialError` on every call, correctly recorded as its own `errors` entry rather than being retried or folded into `missing`, confirming both the retry path and the credential-error path actually executed in this run rather than being written but never exercised.

## Key points

- Classifying MISSING versus UNMANAGED requires querying in BOTH directions, tracked-resource-by-resource lookups catch MISSING; a separate listing call per resource type catches UNMANAGED. A tool that only does the first direction (which is the more obvious one to implement) systematically misses every untracked resource, silently under-reporting real drift.
- Credential errors and rate-limit errors need DIFFERENT handling, not a single generic "retry on any exception" catch-all: retrying a credential error wastes the retry budget on something retrying cannot fix, while NOT retrying a genuine transient throttle would report false drift/errors for what was actually a recoverable, momentary condition.
- Reporting errors as their OWN category, separate from MODIFIED/MISSING/UNMANAGED, matters because a resource the tool COULD NOT CHECK is not the same claim as a resource confirmed unchanged; conflating "we don't know" with "no drift" would produce a report that looks more complete and reassuring than the run actually was.

## Complexity

- Time: $$O(R \cdot A)$$ where $$R$$ is the number of tracked resources and $$A$$ is the maximum retry attempts per resource, plus $$O(T)$$ for the untracked-resource listing calls where $$T$$ is the number of distinct resource types managed.
- Space: $$O(R + U)$$ for the report, where $$U$$ is the number of unmanaged resources discovered, both bounded by the actual inventory size, not by anything unbounded in the algorithm itself.

## Edge cases

- **A resource type with zero tracked resources in this particular tfstate but real untracked resources in the cloud:** correctly caught, since `known_resource_types` is passed explicitly rather than derived only from what happens to be in THIS tfstate.
- **Every retry attempt for a resource exhausted without success:** the exception propagates out of `describe_with_retry` and is caught at the reconciliation level, recorded in `errors` rather than crashing the whole reconciliation run over one resource's persistent failure.
- **A resource with an empty attributes dict on either side:** the field-by-field comparison loop simply finds no keys to compare, correctly falling through to UNCHANGED rather than erroring on an empty iteration.