Infrastructure as Code and GitOps Questions

Direct answer

A shared-repository, multi-team infrastructure Git workflow needs its structural controls (CODEOWNERS, branch protection, environment-promotion paths) to do the coordination work that a monorepo's convenience would otherwise let slide: without them, "everyone can technically edit anything" quietly becomes "nobody is accountable for reviewing changes to what they don't own," exactly the failure mode explicit path-based ownership and branch protection exist to prevent.

Structured elaboration

Branch strategy. TRUNK-BASED, a single long-lived main branch reflecting the current desired state, with short-lived feature branches per change; for a multi-team SRE org specifically, trunk-based keeps merge conflicts small and frequent rather than large and rare, which matters more as the number of independently-committing teams grows.

Pull request flow. Every change to main goes through a PR; CI runs the standard validation (lint, plan, policy-as-code) automatically; required reviewers are determined by CODEOWNERS (below) rather than "any team member," specifically because a shared repo without path-scoped review requirements tends toward whichever reviewer is fastest to click approve, not whichever reviewer actually owns the affected service.

Code ownership rules and CODEOWNERS. A CODEOWNERS file mapping each service's directory (or each shared platform-module directory) to the team that owns it, so a PR touching services/payments/** automatically requires approval from the payments team, and a PR touching platform-modules/network/** automatically requires approval from the platform team, REGARDLESS of who authored it; this is what makes ownership a structurally enforced fact rather than a wiki page nobody re-reads.

CI gating. Required status checks (lint, plan validation, and policy-as-code checks) block merge until green; for a shared repo specifically, CI should be PATH-FILTERED so a change to services/payments/** does not need to wait on or trigger validation for every OTHER team's unrelated services, keeping CI fast and relevant as the repo's scope grows across teams.

Environment promotion (dev to staging to prod). Each team's directory carries its own per-environment overlay structure, reconciled independently by environment-scoped GitOps controllers; a shared repo does not mean a shared PROMOTION CADENCE, team A promoting to production should not be gated on or coupled to team B's own promotion timeline just because they happen to share a repository.

Branch protection rules, with branch name patterns. Protected: main (requires CODEOWNERS approval, passing CI, no force-push). A naming CONVENTION for feature branches keyed to team and change type helps both CI path-filtering and human scanning of open branches: <team>/<type>/<short-description>, for example payments/feat/add-read-replica, platform/fix/network-acl-typo, sre/chore/update-provider-version. This is a convention, not a Git-enforced rule, but combined with CODEOWNERS path scoping it gives a reviewer or CI job an immediate, low-effort signal of WHICH team a branch belongs to before even opening the diff.

Worked example

Repo layout and the resulting review requirement for two concrete changes:

infra-repo/
  platform-modules/
    network/          # CODEOWNERS: @org/platform-team
    observability/    # CODEOWNERS: @org/platform-team
  services/
    payments/
      overlays/dev/ staging/ prod/    # CODEOWNERS: @org/payments-team
    checkout/
      overlays/dev/ staging/ prod/    # CODEOWNERS: @org/checkout-team

A PR on branch payments/feat/bump-replica-count touching only services/payments/overlays/prod/ automatically requires an approval from @org/payments-team (via CODEOWNERS) and triggers ONLY payments-scoped CI checks (via path filtering); it does not require, and does not wait on, any approval or CI run scoped to checkout or platform-modules. A PR on branch platform/feat/upgrade-network-module touching platform-modules/network/** requires @org/platform-team approval and, because platform modules are consumed broadly, ALSO triggers a wider validation pass checking that the change does not break any consuming service's plan, a genuinely different (and appropriately heavier) CI scope than a single-service change.

Trade-offs and pitfalls

• Common mistake: a CODEOWNERS file that is set up once at repo creation and never maintained as teams and ownership boundaries evolve. A stale CODEOWNERS entry either blocks progress (requiring approval from a team that no longer owns that path) or, worse, silently grants review authority to a team that no longer should have it; this needs a periodic ownership-accuracy review, not a one-time setup task.
• Path-filtered CI is real ongoing engineering work, not a one-time setup task, an inaccurate or unmaintained path filter either runs too much (slow, wasteful CI) or too little (a change slips through without the validation it actually needed).
• Branch-name-pattern conventions are NOT enforced by Git itself and need either a lightweight CI check (reject a PR whose branch name does not match the pattern) or acceptance that the convention will erode without one. Documenting a convention and expecting voluntary compliance across many independent teams is a common way this specific piece of the workflow quietly stops being followed within a few months.
• Trunk-based development with many independently-committing teams sharing one repo needs genuinely fast, reliable CI to stay pleasant to work in; if CI is slow or flaky, teams under pressure start finding ways around required checks (requesting an admin override, for instance), which erodes exactly the structural guarantees this workflow is designed to provide.

Direct answer

Helm packages Kubernetes manifests as a TEMPLATED chart (Go template syntax over YAML, plus a values.yaml providing parameters) with a release/versioning model and a package registry ecosystem; Kustomize takes plain, valid YAML manifests as a BASE and applies structural, patch-based OVERLAYS per environment with no templating language at all. For a multi-team organization, the practical recommendation is Kustomize for INTERNAL application manifests owned and reviewed by the teams that run them (overlays are easy to read as plain YAML diffs in a PR), and Helm specifically for THIRD-PARTY or widely-reused software (anything you install FROM someone else's chart, or genuinely reusable internal building blocks you want to parameterize and publish once for many independent consumers).

Structured elaboration

Parameterization and environment overlays, Helm. A chart defines {{ .Values.X }} template placeholders throughout its manifests; each environment supplies its own values-<env>.yaml overriding just the fields that differ. This is powerful (conditionals, loops, helper functions, subchart composition) but the templates themselves are not valid YAML until rendered, so reviewing a raw chart change means mentally executing the template logic, not just reading YAML.

Parameterization and environment overlays, Kustomize. A base/ directory holds plain, directly-appliable YAML; each environment's overlays/<env>/kustomization.yaml lists STRATEGIC MERGE PATCHES or JSON patches against that base (bump replica count, change an image tag, add an env-specific label). Every file at every layer is always valid, renderable YAML on its own, so a reviewer can read a base manifest and an overlay patch and understand the FULL resulting object without running anything.

Pros and cons.

Recommendation for a multi-team organization. Use Kustomize as the default for TEAM-OWNED application manifests: base manifests live in the service's own repo, environment overlays are small, auditable patch files that show up as clean diffs in pull requests, and a reviewer with zero Helm-template-rendering context can read exactly what changes between staging and production. Reserve Helm for two specific cases: consuming THIRD-PARTY software (installing Prometheus, cert-manager, ingress controllers from their published charts, since fighting the ecosystem's Helm-only distribution model is not worth it), and building genuinely reusable INTERNAL platform building blocks meant to be parameterized and consumed by many independent, less-GitOps-sophisticated teams, where Helm's stronger parameterization primitives (conditionals, defaults, schema validation via values.schema.json) earn their complexity. A common, workable hybrid: use helm template to render third-party charts into plain YAML, then apply Kustomize overlays ON TOP of the rendered output for environment-specific tweaks, getting Kustomize's review-friendliness for the parts a team actually edits while still consuming the Helm ecosystem for upstream software.

Trade-offs and pitfalls

• Common mistake: adopting Helm organization-wide "because it's the ecosystem standard" without weighing the code-review cost. A templated chart's diff in a pull request shows changes to TEMPLATE LOGIC or VALUES, not the resulting Kubernetes objects; a reviewer who does not mentally render the template can approve a change without fully understanding what will actually be applied, a real, recurring review-quality gap Kustomize's plain-YAML model avoids by construction.
• Common mistake: assuming Kustomize overlays scale cleanly to many environments without discipline. A patch-based model with no templating can accumulate subtle, hard-to-track divergence across a dozen environment overlays if there is no convention for what belongs in the base versus what belongs in each overlay; Kustomize's simplicity is a review-time advantage, not a substitute for a deliberate base/overlay design convention.
• Security: template injection is a REAL risk specific to Helm when chart values are ever sourced from something less trusted than the chart author (a CI variable an external contributor could influence, for instance), since Go template execution can, in pathological cases, be abused; Kustomize's patch-only model has no equivalent template-execution attack surface, a genuine, structural security argument in Kustomize's favor for team-owned manifests.
• The two tools are not mutually exclusive, and the strongest real-world setups typically use both, Helm for what the ecosystem forces (third-party software) and Kustomize for what teams actually author and review day to day; treating this as an either/or organizational mandate usually produces friction in whichever direction was NOT chosen.

Direct answer

A scalable, multi-account, multi-cluster GitOps pipeline needs its ROLLBACK PROCEDURE for a FAILED reconciliation to be a first-class design element, not an afterthought, because at this scale a partial reconciliation failure (some manifests applied, others rejected mid-sync) is a routine operational event, not an edge case. The design combines a repo layout mirroring the account/cluster/environment hierarchy, a controller instance per cluster (never one controller reaching across account boundaries, which would undermine the RBAC isolation the requirements explicitly call for), and an explicit, tested procedure for what happens when a sync fails PARTWAY, distinct from ordinary drift-correction rollback.

Structured elaboration

Repository layout. A structure mirroring the real topology: clusters/<cloud-account>/<cluster-name>/<environment>/, with shared platform manifests factored into a common base that each cluster-environment combination overlays. This makes "which cluster does this manifest actually apply to" answerable by directory path alone, essential once the fleet spans enough accounts and clusters that tribal knowledge stops scaling.

Automated reconciliation with drift detection. One controller instance PER CLUSTER (not one central instance spanning every account), each watching only its own directory in the shared repo, so a compromise or misconfiguration in one cluster's controller cannot reach across the account boundary to another.

Role-based access controls per environment. Each environment's controller service account holds Kubernetes RBAC scoped to exactly that cluster; human access to trigger manual syncs or view status is scoped via the GitOps tool's own project/tenancy construct, matching the SAME account/cluster/environment boundaries the repo layout encodes, so the repo structure, the RBAC structure, and the controller topology all reflect the SAME hierarchy rather than three independently-maintained mappings that can drift out of sync with each other.

Approval gates for production. A required-review gate on any PR touching a prod environment directory, structurally enforced via CODEOWNERS and branch protection, distinct from and in addition to non-production environments' lighter review bar.

Secure handling of secrets. Per-cluster External Secrets Operator instances, each scoped to that cluster's own path prefix in the secrets backend, so a promotion between environments never requires touching a secret VALUE directly, only the reference structure.

Rollback procedures for failed reconciliations, the part this question specifically emphasizes. Distinguish two failure shapes: (1) a sync REJECTED entirely before applying anything (a validation or policy-as-code failure catches it up front), which needs no rollback at all, nothing was ever applied; and (2) a sync that applied SOME resources before failing partway (a dependency ordering issue, a transient API error on one specific resource), which leaves the cluster in a genuinely INCONSISTENT intermediate state, some resources reflecting the new desired state, others still on the old one. For case 2, the controller's own automatic retry (most GitOps controllers retry a failed sync automatically on the next reconciliation cycle) is usually sufficient IF the underlying cause was transient; if it is not transient (a genuine configuration error), the safe procedure is reverting the OFFENDING commit specifically (not the whole recent history) and letting reconciliation catch the cluster back up to the last known-good state, the same Git-revert-and-reconcile pattern used for ordinary rollback, but explicitly TESTED against a partial-failure scenario before relying on it in production, since a partial failure's exact intermediate state is harder to reason about than a clean, fully-applied-then-reverted change.

Worked example

A concrete PR/merge workflow tying the pieces together: a change merges into clusters/account-b/cluster-3/prod/, requiring @org/platform-team approval (CODEOWNERS) before merge is even possible; on merge, cluster-3's dedicated controller instance (holding RBAC scoped only to that cluster) picks up the change and begins syncing; if the sync fails after applying 6 of 9 manifests (a dependency ordering problem causes the 7th to fail), the controller's automatic retry attempts the remaining 3 on the next cycle; if the underlying issue is NOT transient (a genuine error in the 7th manifest), an operator reverts specifically that commit, and reconciliation, on its next cycle, reconciles the cluster back to matching the reverted (last known-good) declared state, resolving the partial-application inconsistency without requiring a manual, resource-by-resource cleanup.

Trade-offs and pitfalls

• Common mistake: designing rollback procedures only around "a bad change reached prod and needs reverting" and never explicitly testing the PARTIAL-failure case, a sync that applies 6 of 9 manifests before failing leaves a genuinely different, harder-to-reason-about intermediate state than a clean, fully-applied bad change; if this specific case has never been deliberately exercised (a game-day exercise, or a staging-environment fault injection), the team's confidence in "we can just revert and reconcile" may not survive contact with a real partial failure.
• One controller instance per cluster, rather than one centralized instance across accounts, is the right default here specifically because the requirements name per-tenant RBAC and account isolation explicitly; a centralized controller would need its OWN cross-account credentials to reach every cluster, which is exactly the concentration-of-risk a single shared credential creates for isolation-sensitive requirements.
• The repo-layout, RBAC-structure, and controller-topology hierarchies all mirroring the SAME account/cluster/environment structure is a deliberate design choice, not a coincidence, letting any one of the three drift out of alignment with the others (a repo reorganization that doesn't get reflected in RBAC scoping, for instance) reintroduces exactly the kind of tribal-knowledge dependency this design exists to eliminate at scale.
• Automatic retry on transient failures is genuinely helpful but can mask a persistent problem if not paired with alerting on REPEATED failures for the same resource, a sync that keeps failing and retrying silently, cycle after cycle, without ever escalating to a human, is a worse outcome than a sync that fails once, loudly, and gets a deliberate revert.

Direct answer

A secure, versioned configuration backup system needs storage that is IMMUTABLE by construction (not merely "not usually overwritten"), because the entire point of a backup is trusting it even after something has gone wrong with the live system, including a compromise that might otherwise let an attacker tamper with backups too. Object storage with versioning enabled AND object-lock (write-once-read-many, WORM) configured for a defined retention period, encrypted at rest with periodic key rotation, access-logged for every read (not just every write), and integrated into the CI/CD pipeline as an automatic pre-deployment step, meets all five stated requirements together rather than any one addressing them individually.

Structured elaboration

Storage choice. Object storage (S3, Azure Blob, GCS) with BUCKET VERSIONING enabled captures every historical version automatically on each write, and OBJECT LOCK / WORM mode (available on all three major providers in some form) makes a version genuinely immutable for its retention period, not deletable even by an account with otherwise-broad permissions, which is the property a plain "don't delete old backups" convention cannot provide, since a convention is a policy, not an enforced guarantee.

Immutable historical versions. With object-lock in COMPLIANCE mode (the strictest setting, available on the major providers), not even the account root user can delete or overwrite a locked version before its retention period expires; this specifically defends against the scenario where an attacker who has compromised write credentials tries to destroy backup evidence of what the system looked like before the compromise.

Encrypted at rest with key rotation. Server-side encryption using a customer-managed key, with the key itself under a defined rotation schedule (automatic annual rotation is a common baseline for a KMS, key management service, managed key). Rotating the KEY does not require re-encrypting every historical backup object (the KMS handles decrypting old objects with prior key versions transparently), so rotation does not conflict with the immutability requirement above.

Access logging for audits. Every READ of a backup object (not just writes) should be logged with identity, timestamp, and object version, since "who has looked at our backups" is itself audit-relevant, a backup containing sensitive configuration (secrets references, internal topology) being READ by an unexpected identity is a signal worth catching even if nothing was modified.

Retention policies. A tiered retention schedule (for example: every version retained for 30 days, then thinned to one per day for 90 days, then one per month for a compliance-required multi-year window) balances storage cost against how far back a rollback or an audit realistically needs to reach; this needs to be an explicit POLICY on the bucket/lifecycle configuration, not a manual cleanup process someone is expected to remember to run.

Fast retrieval for rollback. Object storage's normal read latency is generally fast enough for a rollback scenario directly; for very large configuration sets where scanning many versions to find "the last known-good one" would be slow, maintaining a separate lightweight INDEX (a small database or manifest file mapping timestamps/commit-SHAs to object versions) makes "find and retrieve the right backup" a fast lookup rather than a storage-level scan.

Worked example

CI/CD integration for automatic pre-deployment backups: before any Terraform apply or GitOps sync that would change a sensitive system's configuration, a pipeline step snapshots the CURRENT live configuration (not just the about-to-be-applied Git state, since the point is capturing what was ACTUALLY running before the change) and writes it to the versioned, object-locked bucket, tagged with the commit SHA of the change about to be applied and a timestamp. The write uses a scoped, write-only credential (this pipeline step needs to WRITE new backups; it never needs to read or delete existing ones), separate from the credentials used for the actual apply step, so a compromise of the apply credential does not automatically also grant tampering access to the backup store.

Trade-offs and pitfalls

• Common mistake: relying on "we never delete backups" as an operational convention rather than an enforced object-lock policy. A convention is defeated by a single mistake, a misconfigured lifecycle rule, an overly broad IAM permission, or a genuine compromise; object-lock in compliance mode is what makes immutability a property of the STORAGE, not a property of everyone's good behavior.
• Encrypting backups with the SAME key used for the live system's own secrets creates an unnecessary shared blast radius. A dedicated key for the backup store means a compromise of the live system's operational key does not automatically also expose historical backups, and vice versa.
• Retention policy needs to account for the compliance-required MINIMUM retention window explicitly, separate from operational convenience. A tiered thinning schedule optimized purely for storage cost can accidentally violate a longer regulatory retention requirement if the two are not reconciled explicitly when the policy is designed, not discovered later during an audit.
• Write-only credentials for the backup step are a real, worthwhile scoping decision, not excess caution. A pipeline identity that can both write new backups and read/delete old ones is a single compromise away from an attacker being able to both plant a fake "known good" backup and destroy the real historical record, exactly the two things a good backup system needs to be resistant to simultaneously.

Direct answer

A practical three-way YAML merge does two DIFFERENT kinds of comparison depending on the node type it is looking at: for maps and scalars, it applies the classic three-way rule (if only one side changed a value relative to base, take the changed side; if both changed it to the SAME value, take that value; if both changed it to DIFFERENT values, that is a genuine conflict); for lists, it merges by a STABLE KEY when every element carries one (matching Kubernetes' own strategic-merge-patch approach), and escalates the WHOLE list to manual resolution when it cannot establish a reliable per-element identity, because merging unkeyed, order-sensitive lists automatically is fundamentally ambiguous, not merely unimplemented.

Approach

1. Map/scalar three-way rule. For each key present in base, local, or remote: if local == remote, take either (no real change or both sides agreed); if local == base, remote changed and local did not, take remote; if remote == base, local changed and remote did not, take local; otherwise both sides changed it to different values, RECORD A CONFLICT and recurse if both sides are still structured the same way (both dicts, or both lists), otherwise it is a genuine leaf-level conflict.
2. List merge, keyed case. If EVERY element across base, local, and remote is a dict carrying a designated merge key (name, matching how Kubernetes strategic-merge-patch identifies "the same" list element across edits), build key-indexed maps for each of the three lists and apply the map/scalar three-way rule to EACH element independently, by key. This is what lets local and remote each edit a DIFFERENT element of the same list without a false conflict, exactly the case this question is really testing.
3. List merge, unkeyed case. If elements are plain scalars or dicts without a shared key field, and both local and remote diverged from base in ways that are not identical, ESCALATE the entire list as a conflict rather than guessing at an ordering or membership resolution; a diff3-style positional merge for unkeyed lists exists in principle but is genuinely ambiguous in the presence of insertions or reordering, and silently guessing wrong here is worse than asking a human.
4. Escalation criterion, stated explicitly. Escalate to manual resolution whenever both branches changed the SAME scalar leaf to different values, or changed the SAME keyed list element's SAME field to different values, or diverged on an unkeyed list; do not attempt to auto-resolve any of these by picking a side, since a silent pick is a correctness bug wearing the appearance of a successful merge.

Code

import yaml

class Conflict:
    def __init__(self, path, base, local, remote):
        self.path = path
        self.base = base
        self.local = local
        self.remote = remote
    def __repr__(self):
        return f"CONFLICT at {self.path}: base={self.base!r} local={self.local!r} remote={self.remote!r}"

MERGE_KEY = "name"  # the field that identifies "the same element" across list edits

def merge_lists(base, local, remote, path):
    if local == remote:
        return local, []
    if local == base:
        return remote, []
    if remote == base:
        return local, []
    # both sides diverged from base differently: try key-based merge if every
    # element in every list is a dict carrying MERGE_KEY
    def all_keyed(lst):
        return all(isinstance(x, dict) and MERGE_KEY in x for x in lst)
    if all_keyed(base) and all_keyed(local) and all_keyed(remote):
        b_by_key = {x[MERGE_KEY]: x for x in base}
        l_by_key = {x[MERGE_KEY]: x for x in local}
        r_by_key = {x[MERGE_KEY]: x for x in remote}
        keys_in_order = []
        seen = set()
        for src in (base, local, remote):
            for x in src:
                k = x[MERGE_KEY]
                if k not in seen:
                    seen.add(k)
                    keys_in_order.append(k)
        merged = []
        conflicts = []
        for k in keys_in_order:
            b_el, l_el, r_el = b_by_key.get(k), l_by_key.get(k), r_by_key.get(k)
            if l_el is None and r_el is None:
                continue  # deleted on both sides
            if l_el is None or r_el is None:
                # deleted on exactly one side while the other kept or edited it:
                # ambiguous (was it a legitimate delete, or did the other side's edit
                # matter?) -- escalate rather than guess
                conflicts.append(Conflict(f"{path}[{MERGE_KEY}={k}]", b_el, l_el, r_el))
                merged.append(l_el if l_el is not None else r_el)
                continue
            el_merged, el_conflicts = three_way_merge(b_el or {}, l_el, r_el, f"{path}[{MERGE_KEY}={k}]")
            merged.append(el_merged)
            conflicts.extend(el_conflicts)
        return merged, conflicts
    # not uniformly keyed (e.g. a plain list of strings/scalars): order and
    # membership are both ambiguous to auto-merge, so this is a genuine
    # escalate-to-manual-resolution case, not something to guess at
    return local, [Conflict(path, base, local, remote)]

def three_way_merge(base, local, remote, path="$"):
    if isinstance(local, list) or isinstance(remote, list) or isinstance(base, list):
        b = base if isinstance(base, list) else []
        l = local if isinstance(local, list) else []
        r = remote if isinstance(remote, list) else []
        return merge_lists(b, l, r, path)

    if isinstance(local, dict) or isinstance(remote, dict) or isinstance(base, dict):
        b = base if isinstance(base, dict) else {}
        l = local if isinstance(local, dict) else {}
        r = remote if isinstance(remote, dict) else {}
        keys = sorted(set(b) | set(l) | set(r))
        merged = {}
        conflicts = []
        for k in keys:
            bv, lv, rv = b.get(k), l.get(k), r.get(k)
            if lv == rv:
                if k in l or k in r:
                    merged[k] = lv if k in l else rv
                continue
            if lv == bv:
                if k in r:
                    merged[k] = rv
                continue
            if rv == bv:
                if k in l:
                    merged[k] = lv
                continue
            # both sides changed this key to DIFFERENT values: recurse if both
            # sides are still structured the same way, otherwise it is a leaf
            # conflict
            if (isinstance(lv, dict) and isinstance(rv, dict)) or (isinstance(lv, list) and isinstance(rv, list)):
                sub_merged, sub_conflicts = three_way_merge(bv, lv, rv, f"{path}.{k}")
                merged[k] = sub_merged
                conflicts.extend(sub_conflicts)
            else:
                conflicts.append(Conflict(f"{path}.{k}", bv, lv, rv))
                merged[k] = lv  # placeholder value; caller must not trust this without resolving the conflict
        return merged, conflicts

    # scalar leaf reached directly (only happens for a root-level scalar document)
    if local == remote:
        return local, []
    if local == base:
        return remote, []
    if remote == base:
        return local, []
    return local, [Conflict(path, base, local, remote)]


BASE_YAML = """
metadata:
  name: api
  labels:
    team: payments
spec:
  replicas: 3
  containers:
    - name: api
      image: myrepo/api:v1
      port: 8080
    - name: sidecar
      image: myrepo/sidecar:v1
  tags:
    - production
    - payments
"""

LOCAL_YAML = """
metadata:
  name: api
  labels:
    team: checkout
    owner: platform-team
spec:
  replicas: 5
  containers:
    - name: api
      image: myrepo/api:v2
      port: 8080
    - name: sidecar
      image: myrepo/sidecar:v1
  tags:
    - production
    - payments
    - critical
"""

REMOTE_YAML = """
metadata:
  name: api
  labels:
    team: billing
spec:
  replicas: 3
  containers:
    - name: api
      image: myrepo/api:v1
      port: 8080
    - name: sidecar
      image: myrepo/sidecar:v2
  tags:
    - production
    - payments
    - compliance
"""

if __name__ == "__main__":
    base = yaml.safe_load(BASE_YAML)
    local = yaml.safe_load(LOCAL_YAML)
    remote = yaml.safe_load(REMOTE_YAML)

    merged, conflicts = three_way_merge(base, local, remote)

    print("=== Merged result ===")
    print(yaml.safe_dump(merged, sort_keys=False))

    print("=== Conflicts (require manual resolution) ===")
    for c in conflicts:
        print(" -", c)

    print("=== Assertions ===")
    assert merged["spec"]["replicas"] == 5, "replicas: local changed, remote unchanged -> local should win"
    print("replicas resolved to local's 5 (remote left it untouched):", merged["spec"]["replicas"] == 5)

    containers_by_name = {c["name"]: c for c in merged["spec"]["containers"]}
    assert containers_by_name["api"]["image"] == "myrepo/api:v2"
    assert containers_by_name["sidecar"]["image"] == "myrepo/sidecar:v2"
    print("containers: api took local's v2, sidecar took remote's v2, no conflict despite both changing the SAME list:",
          containers_by_name["api"]["image"] == "myrepo/api:v2" and containers_by_name["sidecar"]["image"] == "myrepo/sidecar:v2")

    assert merged["metadata"]["labels"]["owner"] == "platform-team"
    print("labels.owner: local-only addition merged cleanly:", merged["metadata"]["labels"]["owner"] == "platform-team")

    conflict_paths = {c.path for c in conflicts}
    assert "$.metadata.labels.team" in conflict_paths
    print("labels.team correctly flagged as a genuine conflict (checkout vs billing):", "$.metadata.labels.team" in conflict_paths)

    assert "$.spec.tags" in conflict_paths
    print("tags correctly escalated (unkeyed list, diverged both sides):", "$.spec.tags" in conflict_paths)

    print("\nTotal conflicts requiring manual resolution:", len(conflicts))

Output (actually executed with python3 s27_merge.py)

=== Merged result ===
metadata:
  labels:
    owner: platform-team
    team: checkout
  name: api
spec:
  containers:
  - image: myrepo/api:v2
    name: api
    port: 8080
  - image: myrepo/sidecar:v2
    name: sidecar
  replicas: 5
  tags:
  - production
  - payments
  - critical

=== Conflicts (require manual resolution) ===
 - CONFLICT at $.metadata.labels.team: base='payments' local='checkout' remote='billing'
 - CONFLICT at $.spec.tags: base=['production', 'payments'] local=['production', 'payments', 'critical'] remote=['production', 'payments', 'compliance']
=== Assertions ===
replicas resolved to local's 5 (remote left it untouched): True
containers: api took local's v2, sidecar took remote's v2, no conflict despite both changing the SAME list: True
labels.owner: local-only addition merged cleanly: True
labels.team correctly flagged as a genuine conflict (checkout vs billing): True
tags correctly escalated (unkeyed list, diverged both sides): True

Total conflicts requiring manual resolution: 2

The run demonstrates all four cases at once against one realistic base/local/remote Deployment-shaped document: spec.replicas merges cleanly because only local changed it (5, remote left it at base's 3); the KEYED containers list merges cleanly even though BOTH branches edited it, because they touched DIFFERENT elements (api and sidecar) identified by the name key; metadata.labels.owner (a brand-new key added only by local) merges cleanly; metadata.labels.team is correctly flagged as a genuine conflict (checkout vs billing, both differing from base's payments); and the UNKEYED spec.tags list is correctly escalated as a whole, since both branches added different tags to the same plain list and there is no reliable way to auto-resolve that.

Key points

• Keying list merges on a stable field (mirroring Kubernetes' own mergeKey concept in strategic merge patches) is what makes "both sides edited the same list" NOT automatically mean "conflict"; without it, any two-sided edit to a shared list would have to be treated as one big, unresolvable diff.
• A key present in exactly one of local/remote but absent from the other (an element deleted on one side while the other side kept or edited it) is deliberately treated as a conflict here rather than guessed at, since silently honoring the delete OR silently keeping the edit are both plausible-looking but potentially wrong resolutions of a genuinely ambiguous situation.
• Recording conflicts as a separate structured list (rather than, say, embedding conflict markers directly in the merged output the way git merge does for text files) makes them straightforward to surface as PR-blocking review comments in an automation context, exactly where "escalate to manual resolution" should actually land.

Complexity

• Time: $O(N)$ where $N$ is the total number of scalar/leaf values across base, local, and remote combined, each key or list element is visited a constant number of times regardless of how deeply nested the structure is.
• Space: $O(N)$ for the merged output plus the conflict list, both bounded by the size of the input documents.

Edge cases

• A key added independently, identically, by both branches (same new key, same value): the local == remote branch takes it cleanly, no conflict, even though technically NEITHER side "agreed with base" (base never had the key at all).
• A key deleted by one branch while genuinely unchanged by the other: treated the same as any other one-side-changed case (deletion IS a kind of change), the deleting side wins and the key is dropped from the merged output, since the OTHER branch is, by definition, == base on that key.
• Nested conflicts inside a keyed list element: handled by RECURSING the same three-way merge logic into each matched element (see the three_way_merge call inside merge_lists), so a conflict three levels deep inside one container's field is reported with a precise path, not just "the containers list has a conflict somewhere."
• An empty base (a brand-new file with no shared ancestor): every key present in either local or remote, with no ancestor value, correctly falls through to the rv == bv or lv == bv branches (both None at that key relative to an effectively empty base treated as {{}}), which functions correctly since Python's dict.get on a missing key returns None uniformly.

Trade-offs and pitfalls

• Common mistake: attempting a generic, order-aware list merge (a diff3-style algorithm) for EVERY list, including ones with no natural key. This is possible in principle (Git itself does something like this for line-based text), but for structured YAML with dict elements and no stable identity field, the "correct" resolution is often genuinely ambiguous even to a human without more context; escalating is the honest answer, not a limitation to apologize for.
• The merge key ("name" here) needs to be chosen per schema, not assumed universal. A Kubernetes container list keys naturally on name; a different structure (a list of firewall rules, for instance) might key on id or a composite of fields; a real implementation needs this configurable per document type, not hardcoded once.
• This implementation treats "both sides changed a key to the same new value" as automatically non-conflicting, which is usually right but can occasionally hide a coincidence worth flagging (two engineers independently arrived at the same fix for unrelated reasons); most real three-way merge tools accept this trade-off since surfacing every coincidental agreement as a review item would be far noisier than useful.