Network Automation and Software-Defined Networking Questions

I would validate in layers, because different failures need different checks.

• Syntax mistake: catch it before traffic changes. Use vendor parse or commit validation in a dry-run so the device rejects bad syntax early.
• Reachability issue: verify control-plane and data-plane paths in a lab or canary. Reachability means packets can still get to critical prefixes and back.
• Behavior change at scale: use staged rollout plus traffic replay or a small canary, because some issues only appear under real load or vendor quirks.

Example: if a route policy changes a next hop, a syntax check will catch a malformed line, a reachability test will catch a blackhole to a key subnet, and a scale test may reveal slower convergence only after thousands of flows.

I would not treat the lab as production, but I would make it good enough to prove the obvious failures before the change reaches a broad blast radius.

I would design the rollout to keep the blast radius small and make every step observable.

• Concurrency control: roll by region and by small batches, with a cap on active sessions per device family.
• Pre-change checks: confirm device reachability, management-plane health, current config version, and maintenance windows.
• Health validation: after each batch, verify routing adjacency, reachability to key prefixes, and service health before continuing.
• Rollback criteria: fail fast on parser errors, control-plane instability, or any sign that the canary batch degraded service.
• Operator visibility: show batch progress, pass or fail status, recent diffs, and the exact rollback action available.

I would also require idempotent jobs, so rerunning the pipeline does not double-apply a change. A good pattern is canary, then one region at a time, then broad rollout only after the earlier batches stay healthy.

If the first batch shows a problem, the system should stop automatically and hand the operator a clear reason, not keep pushing forward.

I would define the source of truth as the site intent model: one record per site that says what hardware, VLANs, IP ranges, ACLs, and monitoring settings should exist.

To keep it repeatable, I would use one provisioning pipeline from empty rack to ready site, with the same ordered steps every time: validate inputs, render configs, pre-check dependencies, push changes, and verify the result.

Site-specific values should live in parameters, not in the workflow itself. For example, the template stays the same, while site_id, uplink IPs, VLAN IDs, and hostnames change per branch. That keeps the logic stable and the data flexible.

Example: for a new office called DEN01, the pipeline would render the router hostname, management IP, baseline ACLs, and monitoring targets from the DEN01 parameter file, then validate the output before deployment. That gives local IT a usable site without hand-built steps.

I would keep intent and logic in Git, but leave secrets out of it.

• In version control: templates, device roles, site inventory, validation rules, and deployment code.
• Render or inject at deploy time: per-device values, generated interface stanzas, and credentials.
• Secrets should come from a secret manager, not plaintext files.

To make the result reproducible, the same commit should always render the same config. That means pinned template versions, deterministic rendering, and storing the rendered artifact or at least the exact commit SHA used for deployment. Reviewability comes from showing both the template diff and a sample rendered diff in CI.

Example: a PR changes the BGP template and a site variable file. CI renders the configs for one router from each site, and reviewers can see exactly what the device will receive before anything is pushed.

I would split it into three layers: a common orchestration layer, vendor adapters, and truly platform-specific logic.

• Common across the fleet: inventory, auth, retries, logging, diffing, approvals, rollback, and change-window handling.
• Vendor-specific: CLI vs API transport, syntax rendering, commit semantics, parsing, and capability detection.
• Keep special cases only when the device family really behaves differently, such as transactional commits versus CLI save commands.

I know the abstraction is becoming a maintenance problem when the core code fills with if vendor == ..., adapters copy each other, tests multiply for every platform, or a small change on one device type forces edits across the shared layer.

Example: a legacy box that only supports a CLI session can still use the same workflow, but its adapter would translate commit into the correct save command while API devices use a structured transaction. The shared engine should not care how that happens.