Network Automation and Software-Defined Networking Questions

Situation: I was on call when an automated config push changed a routing policy on a subset of edge devices, and traffic started behaving oddly.

Task: My job was to stop the impact, identify the bad change, and make sure we did not repeat the failure.

Action: I paused the rollout immediately, rolled back only the affected devices, and coordinated with operations to watch service health while the change was contained. Afterward, I led the postmortem and changed the automation flow so risky routing updates had a canary stage, a lint check, and a human approval gate before broad deployment.

Result: The issue was contained quickly, and later rollouts were much safer because the same class of mistake could no longer go straight to production.

The main lesson was that trust in automation comes from guardrails, not just faster scripts.

I would normalize first, then diff. Normalization means collapsing harmless differences like extra whitespace and sorting object keys so the same config in different formats compares equally. Then I would walk the structure once per device and return a short CI-friendly report.

import json

def load(v):
    return json.loads(v) if isinstance(v, str) else v

# Collapse harmless formatting differences.
def norm(x):
    if isinstance(x, str):
        return ' '.join(x.split())
    if isinstance(x, dict):
        return {k: norm(v) for k, v in sorted(x.items()) if v is not None}
    if isinstance(x, list):
        return [norm(v) for v in x]
    return x

# Collect only real drift.
def diff(a, b, path='', out=None):
    out = [] if out is None else out
    a, b = norm(load(a)), norm(load(b))
    if type(a) != type(b):
        out.append(f'{path}: {type(a).__name__} != {type(b).__name__}')
    elif isinstance(a, dict):
        for k in sorted(set(a) | set(b)):
            p = f'{path}.{k}' if path else k
            if k not in a: out.append(f'{p}: missing in intended')
            elif k not in b: out.append(f'{p}: missing in running')
            else: diff(a[k], b[k], p, out)
    elif isinstance(a, list):
        for i in range(min(len(a), len(b))):
            diff(a[i], b[i], f'{path}[{i}]', out)
        if len(a) != len(b):
            out.append(f'{path}: list length {len(a)} != {len(b)}')
    elif a != b:
        out.append(f'{path}: {a!r} != {b!r}')
    return out

def compare_fleet(intended, running):
    report = {'compliant': [], 'drifted': {}}
    for d in sorted(set(intended) | set(running)):
        if d not in intended or d not in running:
            report['drifted'][d] = ['missing config']
            continue
        d1 = diff(intended[d], running[d])
        if d1:
            report['drifted'][d] = d1
        else:
            report['compliant'].append(d)
    return report

Example: description = 'uplink to core' and description = 'uplink to core' compare equal after normalization, but admin_distance: 110 versus 200 is still flagged. For CI, I would fail the gate if report['drifted'] is not empty and print the first few paths.

I would split it into three layers: a common orchestration layer, vendor adapters, and truly platform-specific logic.

• Common across the fleet: inventory, auth, retries, logging, diffing, approvals, rollback, and change-window handling.
• Vendor-specific: CLI vs API transport, syntax rendering, commit semantics, parsing, and capability detection.
• Keep special cases only when the device family really behaves differently, such as transactional commits versus CLI save commands.

I know the abstraction is becoming a maintenance problem when the core code fills with if vendor == ..., adapters copy each other, tests multiply for every platform, or a small change on one device type forces edits across the shared layer.

Example: a legacy box that only supports a CLI session can still use the same workflow, but its adapter would translate commit into the correct save command while API devices use a structured transaction. The shared engine should not care how that happens.

I would define the source of truth as the site intent model: one record per site that says what hardware, VLANs, IP ranges, ACLs, and monitoring settings should exist.

To keep it repeatable, I would use one provisioning pipeline from empty rack to ready site, with the same ordered steps every time: validate inputs, render configs, pre-check dependencies, push changes, and verify the result.

Site-specific values should live in parameters, not in the workflow itself. For example, the template stays the same, while site_id, uplink IPs, VLAN IDs, and hostnames change per branch. That keeps the logic stable and the data flexible.

Example: for a new office called DEN01, the pipeline would render the router hostname, management IP, baseline ACLs, and monitoring targets from the DEN01 parameter file, then validate the output before deployment. That gives local IT a usable site without hand-built steps.

I would design the rollout to keep the blast radius small and make every step observable.

• Concurrency control: roll by region and by small batches, with a cap on active sessions per device family.
• Pre-change checks: confirm device reachability, management-plane health, current config version, and maintenance windows.
• Health validation: after each batch, verify routing adjacency, reachability to key prefixes, and service health before continuing.
• Rollback criteria: fail fast on parser errors, control-plane instability, or any sign that the canary batch degraded service.
• Operator visibility: show batch progress, pass or fail status, recent diffs, and the exact rollback action available.

I would also require idempotent jobs, so rerunning the pipeline does not double-apply a change. A good pattern is canary, then one region at a time, then broad rollout only after the earlier batches stay healthy.

If the first batch shows a problem, the system should stop automatically and hand the operator a clear reason, not keep pushing forward.