Network Automation and Software-Defined Networking Questions

I would design the rollout to keep the blast radius small and make every step observable.

• Concurrency control: roll by region and by small batches, with a cap on active sessions per device family.
• Pre-change checks: confirm device reachability, management-plane health, current config version, and maintenance windows.
• Health validation: after each batch, verify routing adjacency, reachability to key prefixes, and service health before continuing.
• Rollback criteria: fail fast on parser errors, control-plane instability, or any sign that the canary batch degraded service.
• Operator visibility: show batch progress, pass or fail status, recent diffs, and the exact rollback action available.

I would also require idempotent jobs, so rerunning the pipeline does not double-apply a change. A good pattern is canary, then one region at a time, then broad rollout only after the earlier batches stay healthy.

If the first batch shows a problem, the system should stop automatically and hand the operator a clear reason, not keep pushing forward.

I would treat drift as a comparison problem between three things: intended state, running state, and approved exceptions.

• Intended state is the baseline in Git or the CMDB.
• Running state is what the device actually has.
• Approved exception is a temporary or permanent deviation tied to a ticket, owner, scope, and expiry.

The reconciler should only auto-correct unapproved drift. If a difference matches an active exception, it should be marked as tolerated and revisited when the exception expires. That prevents the system from undoing a valid incident fix every audit cycle.

Example: during an outage, an ACL is opened for 10.2.0.0/16 for two hours. The audit sees the change, matches it to the incident record, and suppresses remediation until the TTL expires. After that, the next reconciliation run can restore the original policy if no one renews the exception.

I would normalize first, then diff. Normalization means collapsing harmless differences like extra whitespace and sorting object keys so the same config in different formats compares equally. Then I would walk the structure once per device and return a short CI-friendly report.

import json

def load(v):
    return json.loads(v) if isinstance(v, str) else v

# Collapse harmless formatting differences.
def norm(x):
    if isinstance(x, str):
        return ' '.join(x.split())
    if isinstance(x, dict):
        return {k: norm(v) for k, v in sorted(x.items()) if v is not None}
    if isinstance(x, list):
        return [norm(v) for v in x]
    return x

# Collect only real drift.
def diff(a, b, path='', out=None):
    out = [] if out is None else out
    a, b = norm(load(a)), norm(load(b))
    if type(a) != type(b):
        out.append(f'{path}: {type(a).__name__} != {type(b).__name__}')
    elif isinstance(a, dict):
        for k in sorted(set(a) | set(b)):
            p = f'{path}.{k}' if path else k
            if k not in a: out.append(f'{p}: missing in intended')
            elif k not in b: out.append(f'{p}: missing in running')
            else: diff(a[k], b[k], p, out)
    elif isinstance(a, list):
        for i in range(min(len(a), len(b))):
            diff(a[i], b[i], f'{path}[{i}]', out)
        if len(a) != len(b):
            out.append(f'{path}: list length {len(a)} != {len(b)}')
    elif a != b:
        out.append(f'{path}: {a!r} != {b!r}')
    return out

def compare_fleet(intended, running):
    report = {'compliant': [], 'drifted': {}}
    for d in sorted(set(intended) | set(running)):
        if d not in intended or d not in running:
            report['drifted'][d] = ['missing config']
            continue
        d1 = diff(intended[d], running[d])
        if d1:
            report['drifted'][d] = d1
        else:
            report['compliant'].append(d)
    return report

Example: description = 'uplink to core' and description = 'uplink to core' compare equal after normalization, but admin_distance: 110 versus 200 is still flagged. For CI, I would fail the gate if report['drifted'] is not empty and print the first few paths.

I would design it as a state machine with guardrails. The alert is first enriched with context: is there maintenance, a recent change, or an existing incident? Then I would correlate events in a time window, since three flaps in ten minutes is more meaningful than one isolated flap.

Decision rules:

• Auto-remediate only if the failure is known, the action is safe, and no recent remediation already failed.
• Ask for human approval if the cause is ambiguous, the blast radius is large, or the action could worsen routing.
• Suppress loops by tagging automation-generated events, adding a cooldown, and enforcing idempotency so the same incident is not remediated twice.

Example: if a link flaps three times, the workflow might first check whether the last action came from the automation itself. If yes, it stops and escalates instead of retrying. That keeps the system from creating its own flapping.

I would validate in layers, because different failures need different checks.

• Syntax mistake: catch it before traffic changes. Use vendor parse or commit validation in a dry-run so the device rejects bad syntax early.
• Reachability issue: verify control-plane and data-plane paths in a lab or canary. Reachability means packets can still get to critical prefixes and back.
• Behavior change at scale: use staged rollout plus traffic replay or a small canary, because some issues only appear under real load or vendor quirks.

Example: if a route policy changes a next hop, a syntax check will catch a malformed line, a reachability test will catch a blackhole to a key subnet, and a scale test may reveal slower convergence only after thousands of flows.

I would not treat the lab as production, but I would make it good enough to prove the obvious failures before the change reaches a broad blast radius.