Network Automation and Software-Defined Networking Questions

I would split it into three layers: a common orchestration layer, vendor adapters, and truly platform-specific logic.

• Common across the fleet: inventory, auth, retries, logging, diffing, approvals, rollback, and change-window handling.
• Vendor-specific: CLI vs API transport, syntax rendering, commit semantics, parsing, and capability detection.
• Keep special cases only when the device family really behaves differently, such as transactional commits versus CLI save commands.

I know the abstraction is becoming a maintenance problem when the core code fills with if vendor == ..., adapters copy each other, tests multiply for every platform, or a small change on one device type forces edits across the shared layer.

Example: a legacy box that only supports a CLI session can still use the same workflow, but its adapter would translate commit into the correct save command while API devices use a structured transaction. The shared engine should not care how that happens.

Situation: I was on call when an automated config push changed a routing policy on a subset of edge devices, and traffic started behaving oddly.

Task: My job was to stop the impact, identify the bad change, and make sure we did not repeat the failure.

Action: I paused the rollout immediately, rolled back only the affected devices, and coordinated with operations to watch service health while the change was contained. Afterward, I led the postmortem and changed the automation flow so risky routing updates had a canary stage, a lint check, and a human approval gate before broad deployment.

Result: The issue was contained quickly, and later rollouts were much safer because the same class of mistake could no longer go straight to production.

The main lesson was that trust in automation comes from guardrails, not just faster scripts.

I would rebuild trust before I try to scale automation. First, I would run in shadow mode, which means the automation produces the proposed change but does not touch devices yet. That lets the team compare output to manual work.

For the first workflows, I would pick things that are high-volume, low-risk, and reversible, like inventory checks, config backups, or drift reports. I would avoid disruptive changes until the team sees consistent results.

To avoid a pile of ad hoc scripts, I would create one supported automation path, shared libraries, code review, tests, and named owners. I would also publish runbooks and a small set of approved workflows, then deprecate one-off scripts that duplicate the platform.

The main goal is confidence through repeatability. When people see the same inputs produce the same output, and rollback is clear, adoption usually follows.

I would design the rollout to keep the blast radius small and make every step observable.

• Concurrency control: roll by region and by small batches, with a cap on active sessions per device family.
• Pre-change checks: confirm device reachability, management-plane health, current config version, and maintenance windows.
• Health validation: after each batch, verify routing adjacency, reachability to key prefixes, and service health before continuing.
• Rollback criteria: fail fast on parser errors, control-plane instability, or any sign that the canary batch degraded service.
• Operator visibility: show batch progress, pass or fail status, recent diffs, and the exact rollback action available.

I would also require idempotent jobs, so rerunning the pipeline does not double-apply a change. A good pattern is canary, then one region at a time, then broad rollout only after the earlier batches stay healthy.

If the first batch shows a problem, the system should stop automatically and hand the operator a clear reason, not keep pushing forward.

I would treat drift as a comparison problem between three things: intended state, running state, and approved exceptions.

• Intended state is the baseline in Git or the CMDB.
• Running state is what the device actually has.
• Approved exception is a temporary or permanent deviation tied to a ticket, owner, scope, and expiry.

The reconciler should only auto-correct unapproved drift. If a difference matches an active exception, it should be marked as tolerated and revisited when the exception expires. That prevents the system from undoing a valid incident fix every audit cycle.

Example: during an outage, an ACL is opened for 10.2.0.0/16 for two hours. The audit sees the change, matches it to the incident record, and suppresses remediation until the TTL expires. After that, the next reconciliation run can restore the original policy if no one renews the exception.