Network Automation and Software-Defined Networking Questions

Direct answer

Skeptics are right about the old scripts, so I do not argue with them. I earn trust in stages: first automation that only reads (nothing it does can break anything), then small writes that show a diff before they apply and can be undone, and only later the risky changes. Every piece of automation lives in Git with an owner, a test and a retirement rule, which is what stops the pile of unmaintained scripts from forming again.

Why the earlier scripts were not trusted

Before choosing anything, I ask the team what actually went wrong. The usual causes are specific and fixable:

• The script screen-scraped command-line (CLI) text with string matching, so a software upgrade that changed one output line broke it silently.
• It was not idempotent (safe to run twice with the same result), so a re-run after a failure doubled up configuration.
• Hostnames and passwords were hard-coded and it lived on one engineer's laptop.
• It pushed first and checked afterwards, with no way back.

Each trust-building step below answers one of these.

Building confidence in four stages

A concrete read-only check at stage 1 is a NAPALM (a Python library that gives one API over several vendors) call such as get_facts() or get_ntp_servers(), which returns structured data, not text to scrape. For example, get_facts() returns a dictionary with documented keys such as hostname, vendor, model, os_version, serial_number and interface_list, so the check reads facts['os_version'] instead of searching command output for a line that an upgrade might reword. Structured data is the direct answer to the screen-scraping failure.

Choosing the first workflows

I score candidates on three questions: can it break anything (read-only wins), how often does a human do it today (frequent chores pay back fastest), and how easy is it to check the result (a config backup is checkable by diffing). That puts these first: config backup, inventory and software-version reporting, compliance checks against the standard, and one standard-setting push such as NTP. It puts these last: BGP policy, spanning tree changes, anything that can partition the network.

Not ending up with a pile of unmaintained scripts

• One repository, one entry point. No script on a laptop. A change to automation goes through pull request review like a change to the network.
• Every workflow has a named owner and a test. The test renders the template against sample data and checks the output; it runs in CI (continuous integration, an automatic check on every proposed change). A workflow with no owner is not merged.
• Shared library, not copies. Connection handling, credentials, logging and diffing live in one module that every workflow imports, so a fix lands once.
• Credentials from a secret store, never in the repository.
• A retirement rule. I keep a short list of every automated workflow with its last-run date; one that nobody has run in a quarter is either given an owner or archived, deliberately.
• Measure adoption and trust, not script count. The numbers worth watching are the share of devices matching the standard, the share of changes made through the pipeline rather than by hand, and how many pipeline changes needed rolling back.

What the story sounds like in the room

Situation: a team that had been burned by scripts. Actions I took: sat with the on-call engineers to list how the old scripts failed, started with the nightly backup and compliance report that I ran next to their manual process for a few weeks, and let them compare results. When a report disagreed with an engineer's reading, I treated it as a bug in the check and fixed the check in the open. The first write was an NTP change to a single lab-adjacent branch device with the diff on screen and the rollback armed. Result to claim: they ask for the next workflow themselves, and nobody runs a one-off script on a production device any more. What I would do differently: involve the most skeptical engineer as the reviewer for the first template, because their objections were the most useful test cases.

Pitfalls

• Starting with a showcase change (a VLAN rollout) makes the first failure public and ends the program.
• Promising "full automation". Promise "fewer surprises" and show the evidence.
• Letting the pipeline become the only way to change a device before there is a console or out-of-band path (a way to reach the device that does not depend on the production network, such as a serial console or a separate management link) for emergencies.

Direct answer

I approve a change when three questions have each been answered by the kind of evidence that can answer them: will the device accept it (syntax), will the network behave as intended (design), and will this hardware and software do what the documentation says (platform). A lab and static analysis can answer the first two well; the third can only be tested on production-like hardware, which in practice means a limited production canary with a rollback ready. Say clearly what remains unproven after each check, and size the rollout to that.

Three failure classes and the evidence for each

Syntax mistakes: cheap and mechanical

• Render the configuration from templates and fail the build on a missing variable.
• Batfish (an open-source analyzer that reads configuration files without device access) answers named questions about a set of configs. fileParseStatus reports, for each file, whether it parsed fully, partially or not at all, and parseWarning lists the warnings, such as lines it did not recognise. undefinedReferences lists every place a config uses a named object that is never defined; each result row gives the file, the kind of object (for example a route-map), the missing name, and the line numbers where it is used. An empty result means every reference resolves.
• Ask the device itself without committing. With NAPALM (a Python library giving one API across vendors), load_merge_candidate() stages the candidate and compare_config() returns the diff; discard_config() throws it away. Junos has commit check, which validates a candidate (the proposed configuration, staged but not yet active) without activating it. NETCONF devices that advertise the :validate capability (urn:ietf:params:netconf:capability:validate:1.1) can validate a candidate datastore.
• Residual risk is low and the fix is cheap.

Design flaws: model, don't just lint

A syntactically perfect change can still be wrong. I check intent against behaviour:

• Run network-wide analysis: Batfish can compare two snapshots (the current configs and the proposed configs) with differentialReachability, which returns example flows that succeed in one snapshot but not the other, so gained or lost connectivity shows up before anything is pushed (it returns examples rather than every flow, so rerun it after each fix). testRoutePolicies feeds sample routes through a route policy (the rules a router uses to accept, reject or modify routes) and shows what comes out.
• Write down expected outcomes as testable statements ("branch A can still reach the data center on both uplinks; prefix X is not advertised to the internet") and check each before and after.
• Walk the failure scenarios: what if the link or peer the change relies on is down?
• Residual risk: the model uses its own idea of how a device behaves, and a lab topology that is smaller than production hides scale effects (route table size, number of ACL entries, convergence time).

Platform bugs: you cannot prove their absence in a lab

Lab and snapshot cannot catch problems that depend on production traffic, the vendor's real implementation or scale, which is exactly the limit stated in the question. So:

1. Search the vendor's release notes and bug database for the exact software train and feature before approval.
2. Run the change on the lowest-impact production device of the same model and version (a canary: a first, deliberately small deployment whose job is to reveal problems cheaply), not only in a lab with different hardware.
3. Compare device state before and after (NAPALM getters such as get_bgp_neighbors() and get_interfaces() return structured data) so unexpected changes show up even if no alarm fires.
4. Arm a rollback with a timer (commit-confirm: the device applies the change but reverts it by itself unless you confirm in time): Junos commit confirmed rolls back automatically unless confirmed (default 10 minutes); IOS XE's configure replace ... time with configure confirm does the same job.
5. Widen in stages, and treat one clean canary as weak evidence: it excludes frequent faults, not rare ones.

Worked example

Change: raise BGP local preference for routes learned from a new transit provider so it becomes the preferred exit. BGP is the routing protocol that exchanges routes between networks. Local preference is a number a router attaches to a route inside your own network; when the same destination is reachable several ways, the route with the higher local preference wins (RFC 4271). A prefix-list is a named list of address ranges, and a route-map is a named set of match-and-set rules that here says "for routes matching this prefix-list, set local preference to a higher value".

• Syntax: the prefix-list name in the new route-map is misspelled. undefinedReferences flags it in CI. Nothing reaches a device.
• Design: the route-map sets the higher local preference on every route from that provider, including prefixes of your own customers that should keep using their direct path. The differential analysis shows traffic to those prefixes moving. The fix is a narrower match, found before approval.
• Platform: with both fixes, the canary router shows its BGP process CPU staying elevated after the policy is applied, which the lab (with a far smaller route table) never showed. The vendor's release notes list a known issue for that version. The rollout stops at one router, the timer rolls the router back, and the approval waits for a software fix or a workaround.

Approval rule

I approve when the syntax checks are green, every stated expectation holds in the model, the vendor's known issues for the target version are read, and for any change touching routing or forwarding, a canary and a rollback are planned. A change with a higher blast radius (the amount of the network a mistake can hurt, as with an edge or core device) needs the canary even if every earlier check passed.

Direct answer

Version-control the intent and the code that turns intent into configuration: templates, per-device and per-group data, playbooks, tests and review rules. Generate the actual interface stanzas at review time and at deploy time from that source, and inject credentials only at run time from a secret store. Reviewers then approve two things on one pull request: the change to the source, and the exact configuration lines it produces, shown as a diff.

What goes where

Rendered output is kept out of the hand-edited tree so nobody "fixes" a generated file and loses the change at the next render.

Repository layout

network-config/
  CODEOWNERS
  inventory/
    hosts.yml
    group_vars/        # per role or region values
    host_vars/         # nyc-edge1.yml, one file per device
  templates/           # interfaces.j2, aaa.j2 ...
  playbooks/           # render.yml, deploy.yml
  tests/               # sample data and expected output
  ci/                  # pipeline definition

Reading the tree: CODEOWNERS is the review-rule file; inventory/ holds the data (hosts.yml lists devices, group_vars/ values shared by a role or region, host_vars/ one file per device); templates/ holds the Jinja2 files that turn data into configuration; playbooks/ holds the render and deploy entry points; tests/ holds sample data with the expected output; ci/ defines the pipeline that runs the checks. Nothing in it is a rendered config or a secret.

Credentials

• Preferred: device credentials live in a secret manager and the runner reads them when the job starts, so no secret exists in Git history at all and rotation needs no commit.
• Smaller teams: Ansible Vault. ansible-vault encrypt_string turns one value into an inline block that is safe to commit, for example a line ios_enable_secret: !vault | followed on the next, indented line by $ANSIBLE_VAULT;1.1;AES256 and then the encrypted text (I ran the command; the output began that way, over several lines). The vault password is supplied to the job by CI through a password file or script, and the playbook should use no_log: true (an Ansible task setting that hides the task's arguments and results from logs) on tasks that handle the secret so it never prints.
• Encrypted-in-Git still means anyone with the vault password can read every secret and history keeps old ciphertext, so rotate the vault password and the secrets together if either leaks.

Structuring the review

1. Branch and pull request for every change, one logical change per request.
2. Automatic checks first, before any person reads it: YAML lint, Ansible syntax check, data checks (valid IPs, no overlaps, required fields), and rendering for every affected device.
3. Rendered diff on the pull request. CI renders the target branch and the proposed branch and shows the unified diff. I executed this in a container: changing one host's address from 10.20.1.1 to 10.20.1.254 produced

- ip address 10.20.1.1 255.255.255.0
+ ip address 10.20.1.254 255.255.255.0

with every other line unchanged. A reviewer who sees a three-line template change cause 400 lines of diff across 60 devices knows to look closer.
4. Review by ownership. A CODEOWNERS file (a file in the repository that maps paths to the people or teams who must review changes to them) maps paths to people or teams (for instance templates/ to the design team and inventory/host_vars/nyc-* to the New York site owners). With branch protection (a repository setting that blocks merging until rules such as required reviews and passing checks are met) set to require code-owner review, GitHub needs approval from at least one code owner of each changed path.
5. Merge triggers deployment of the reviewed commit, pinned by its commit identifier, to a canary set (a small first group of devices that gets the change before the rest) first, then wider. The reviewed state and the deployed state must be the same commit.
6. Compare after deploy. A scheduled job compares each device's running configuration with the render of main and reports differences.

Trade-offs and pitfalls

• Template review without rendered output is the commonest mistake: people approve a loop they cannot mentally expand.
• Check mode against live devices is a useful pre-deploy gate but needs device access, so run it from a runner with network reach, not from the pull-request CI. For cisco.ios.ios_config, --check -v prints the commands list it would send; --diff alone prints nothing for that module unless diff_against is set (from the collection's source).
• Emergency changes will happen. Provide a fast path (pair review, deploy, then back-port to Git) rather than banning it, or engineers will bypass the whole process.
• Never print credentials in CI logs; test that the pipeline output contains no secret material.

Direct answer

I review a device-changing script as a change that can take down production at machine speed: the first questions are blast radius (how many devices a mistake can hurt), credentials and what happens on failure, and only then style. My checklist, in the order I read the code, has twelve items (below), each with the failure it prevents. Anything that is missing a dry run, a verification step or a stop condition does not merge.

The checklist and why each matters

1. Target selection. Which devices will it touch? Is the device list an explicit, reviewed input with a maximum size, or computed from a wildcard? A typo in a filter that matches all 4,000 devices instead of 4 is the classic outage. Require an allowlist and a confirmation or hard cap.
2. Dry run by default. The script should show a diff and exit unless told to apply. NAPALM's compare_config() returns the difference between running and candidate configuration, and a review without a diff is a review of intent only.
3. Credentials. No passwords or keys in code, logs, or environment dumps; secrets come from a vault, the account is dedicated to automation with least privilege (for example command authorization on the AAA (authentication, authorization and accounting) server limiting it to what the job needs), and logs must never print the session transcript with secrets.
4. Host key verification. A host key is the SSH server's identity, which the client should check so it is not talking to an impostor. Netmiko's ssh_strict defaults to False, which means unknown SSH host keys are accepted automatically. Set ssh_strict=True so unknown keys are rejected, and load the known keys with system_host_keys=True (the user's known-hosts file) or a pinned known-hosts file; leaving ssh_strict False and only adding system_host_keys still accepts every host that is not yet listed, and an attacker in the path collects your credentials.
5. Idempotency and preconditions. Idempotent means running it twice gives the same end state as running it once. Running it twice must not change anything the second time, and it must verify the device is in the expected state first (version, role, current config hash). Appending a line twice to an ACL (access-control list) or re-adding a route is the failure.
6. Error detection. send_config_set is Netmiko's call that sends a list of config commands, and it only raises on a rejected command if you pass error_pattern; it then raises ConfigInvalidException when the output matches (error_pattern is a regular expression for the device's error text). Without it a half-applied config looks like success. Check the exact pattern for each platform's error text.
7. Rollback path. Is there a tested way back: a candidate configuration with discard_config(), a confirmed commit that reverts if not confirmed (commit_config(revert_in=...) in NAPALM; NETCONF's confirmed commit reverts if no confirming commit arrives and defaults to 600 seconds), or a saved pre-change config? "We can revert manually" is not a rollback. Also check whether the script saves to startup configuration (save_config in Netmiko is implemented per platform, not in the base class) only after verification.
8. Post-change verification. Does it check what the change was meant to do (BGP neighbor Established, expected prefix count, interface up), not merely that the command was accepted?
9. Staged rollout and stop conditions. One device (the canary, a first trial whose result tells you whether to continue), then a small batch, then wider, with automatic halt on the first failure or when failures exceed a threshold. Parallelism without a circuit breaker (an automatic stop once failures cross a threshold) spreads a bad change at machine speed.
10. Concurrency and locking. Can two runs (or a human) touch the same device at once? Look for a lock or lease per device and a hold or maintenance-window check.
11. Timeouts and retries. Every connection and command has a timeout, and retries apply only to safe steps. Retrying a commit after a timeout, when the first may have succeeded, is how changes get applied twice.
12. Logging, audit and tests. Log who ran what against which device and the diff, with secrets masked, so an incident can be reconstructed; and unit tests for the template rendering and parsing that run without a device.

A short critique example

Reading the checklist by blast radius, items 1, 2 and 9 limit how many devices a mistake can reach, and items 7 and 8 limit how long a mistake lasts. Reviewing this fragment, I would reject it for items 1, 2, 3, 4, 6, 7, 8 and 9 at once:

from netmiko import ConnectHandler
for ip in open("all_devices.txt"):
    c = ConnectHandler(device_type="cisco_ios", host=ip.strip(), username="admin", password="Summer2024!")
    c.send_config_set(["router bgp 65000", "neighbor 192.0.2.1 shutdown"])
    c.save_config()

Mapping to the lines: reading all_devices.txt with no cap or review is item 1; going straight to send_config_set with no diff is item 2; password="Summer2024!" is item 3; ConnectHandler called without host key checking is item 4 (Netmiko's default is not to check); send_config_set without an error_pattern is item 6; save_config() on the next line writes the change to startup configuration before anything has been verified, which is item 7; nothing reads back the BGP neighbor state, which is item 8; and the plain for loop over every device with no stop condition is item 9.

Fixes: take the targets from a reviewed list with a cap, take credentials from a vault, enable host key checking, render the change from a template and show a diff first, pass an error_pattern, run on one canary device and verify the neighbor state, stop on the first failure, then save. (This fragment is written for the critique and was not run against a device.)

Trade-offs and pitfalls

• Reviewers fixate on code style and miss operational risk: order the review by blast radius.
• A mandatory dry run that nobody reads is theater: the diff must be attached to the change record.
• Checklists age. Add an item whenever an incident reveals a gap.

Direct answer

Build a smaller fabric that keeps everything that decides behavior identical to production, and fake the rest. A fabric here means the leaf-and-spine network inside a data center: leaf switches connect to servers, spine switches connect the leaves. Run the same network operating system (NOS, the switch's software) image and config templates as virtual devices in a containerlab topology launched from CI, keep the production number of spines so equal-cost multipath (ECMP, spreading traffic over several equally good paths, one path per flow) width is the same, represent traffic-sending hosts as Linux namespaces, represent the thousands of other hosts as routes injected by a BGP speaker, drive mixed long and short flows, inject failures, and measure convergence (how long traffic is lost before the network routes around a failure) with sequence-numbered probe streams. BFD (bidirectional forwarding detection) is a lightweight protocol that detects a dead link or neighbor in under a second, much faster than BGP's own timers; it appears in the tests below. A run passes only if every measured outage is under the gate written down beforehand.

What stays real and what is scaled down

Example sizes (a stated assumption, replace with your design): production is 64 leaf switches (top of rack) x 48 hosts = 3,072 hosts, 4 spines, 4 uplinks per leaf, 48 host links at 25G against 4 uplinks at 100G.

The 3,024 routes come from a BGP route injector such as GoBGP (gobgp global rib add <prefix> adds one route, gobgp global rib del <prefix> removes it, and gobgp mrt inject global <file> loads many from a file). Run it as extra neighbors on a few leaves so the routes arrive through the same code paths real hosts use.

Building it in containerlab

containerlab describes the lab in a YAML topology file: nodes with a kind and an image, and links as pairs of endpoints. Kinds include linux (any container, used for hosts and traffic generators) and vendor kinds such as arista_ceos. Excerpt (the full file repeats the pattern for 4 spines and 8 leaves; leaf ports eth1 to eth4 are the four uplinks that the generator below wires, so host links start at eth5):

name: fabric
topology:
  nodes:
    spine1: { kind: linux, image: <your FRR or NOS image> }
    leaf1:  { kind: linux, image: <your FRR or NOS image> }
    hosts1: { kind: linux, image: <image with iperf3 and iproute2> }
  links:
    - endpoints: ["spine1:eth1", "leaf1:eth1"]
    - endpoints: ["leaf1:eth5", "hosts1:eth1"]

The kind follows the image: linux for a plain container such as an FRR routing image, or a vendor kind such as arista_ceos for that vendor's image. To write the whole topology instead of repeating lines by hand, generate the links. This script prints the 32 spine-to-leaf links for 4 spines and 8 leaves (leaf l connects to spine s using spine port l and leaf port s) and shows the first and last line:

SPINES, LEAVES = 4, 8
links = [
    f'    - endpoints: ["spine{s}:eth{l}", "leaf{l}:eth{s}"]'
    for l in range(1, LEAVES + 1)
    for s in range(1, SPINES + 1)
]
print(len(links), "links")
print(links[0])
print(links[-1])

Output:

32 links
    - endpoints: ["spine1:eth1", "leaf1:eth1"]
    - endpoints: ["spine4:eth8", "leaf8:eth4"]

The generated lines go under links: in the topology file. containerlab deploy -t fabric.clab.yml builds it, and containerlab destroy -t fabric.clab.yml --cleanup removes the lab and its directory. Use the vendor image your production devices run when you can; a different NOS validates the design logic but not the vendor's convergence behavior.

Traffic: what to generate and why

• Long-lived heavy flows (elephants). ECMP picks an uplink per flow from a hash of the flow's 5-tuple (source address, destination address, protocol, source port, destination port), so a handful of elephants can land on the same uplink while the average utilisation looks healthy. Generate them with several parallel TCP streams per host pair, for example iperf3 -c <server> -P 8 -t 600 -J (-P runs parallel streams, each on its own ephemeral source port, which gives each its own hash input, and -J writes JSON that CI can parse). Check per-uplink interface counters: a gate such as "no uplink carries more than twice the average of its leaf's four uplinks" turns hash imbalance into a failing test.
• Many short flows (mice, the opposite of elephants). Run a loop that opens thousands of short TCP connections between random namespace pairs. This stresses hash distribution across many 5-tuples, ARP and neighbor churn, and any stateful device, and it is the traffic that exposes a fabric that only works for a few big flows.
• Probe streams for measurement. One UDP packet per millisecond (1,000 packets per second), each carrying a sequence number, across 64 distinct source ports per monitored host pair. Probes are cheap enough to run during every other test.
• Why 64 flows. A link failure only affects flows hashed onto it. Under uniform hashing each flow lands on any given uplink with probability 1/4, so it avoids the failed uplink with probability 3/4, and the flows are independent, so n flows all avoid it with probability (3/4)^n. For 4 flows that is 0.75 x 0.75 x 0.75 x 0.75 = 0.3164, so about a third of the time a 4-flow probe never sees the failure. The program below prints it: 3.16e-01 (31.6 percent) for 4 flows, 1.00e-02 (1.0 percent) for 16, and 1.01e-08 for 64. The lab's software hash is not the production ASIC's hash, so also read the uplink counters before each injection and confirm the failed link was actually carrying probes.

Control-plane behavior to exercise

The silent failure row is the one that justifies the whole lab. BGP keepalive messages are the periodic hellos a session sends; if none arrive within the hold time, the session is declared dead. FRR's default BGP keepalive is 60 seconds and hold time 180 seconds, so a session that depends only on those timers can blackhole traffic for up to 180 seconds. RFC 7938 notes that relying on keepalive timeouts can mean convergence delays of multiple seconds, and recommends fast fallover on link-down events. BFD (a lightweight protocol that detects forwarding failure between neighbors) defaults in FRR to a 300 ms interval and a multiplier of 3, so detection takes 300 x 3 = 900 ms. Enabling it on a BGP neighbor is neighbor <peer> bfd.

Proving convergence and gating it in CI

To read the results of the table above: a plain link-down shows as a short gap in the sequence numbers on the affected flows (typically the time to detect plus the time to reprogram the forwarding table, or FIB, the table the switch uses to pick output ports); a silent failure without BFD shows as a gap that can run up to the BGP hold time; a healthy return shows no gap at all.

Outage is the longest run of missing sequence numbers divided by the probe rate. The program below computes it from a pinned synthetic capture (5,000 probes, packets 2,000 to 2,929 lost, as BFD detection of 900 ms plus 30 ms of forwarding-table update would produce), so you can run it unchanged. The program prints several numbers; the one that proves convergence is the line outage_ms (the longest gap in the probe stream), and the other lines are the sizing and sampling arithmetic used above:

# Convergence from a probe stream: one UDP packet every 1 ms (1,000 pps), sequence-numbered.
# Outage = longest run of missing sequence numbers / probe rate.
RATE_PPS = 1000

def outage_ms(received_seqs, sent_total):
    got = set(received_seqs)
    longest = run = 0
    for seq in range(sent_total):
        run = 0 if seq in got else run + 1
        longest = max(longest, run)
    return longest * 1000 / RATE_PPS

# Pinned synthetic capture: 5,000 probes sent, packets 2,000..2,929 lost (link failure,
# BFD at 300 ms x 3 = 900 ms detection plus 30 ms of FIB reprogramming).
received = [s for s in range(5000) if not 2000 <= s < 2930]
print("probes sent:", 5000, "received:", len(received))
print("outage_ms:", outage_ms(received, 5000))

# Probability that none of n distinct-5-tuple probe flows hashes onto one failed uplink of 4
for n in (4, 16, 64):
    print(f"flows={n}: P(no flow on the failed uplink) = {(3/4)**n:.2e}")

print("BFD detect ms, default 300 x 3:", 300 * 3, "| tuned 100 x 3:", 100 * 3)
print("prod hosts:", 64 * 48, "| host links 48 x 25G =", 48 * 25, "G vs uplinks 4 x 100G =", 4 * 100, "G ->", 48 * 25 / (4 * 100), ": 1")
print("lab hosts as namespaces:", 8 * 6, "| injected host routes:", 64 * 48 - 8 * 6)
print("BGP sessions prod:", 64 * 4, "lab:", 8 * 4)

Output:

probes sent: 5000 received: 4070
outage_ms: 930.0
flows=4: P(no flow on the failed uplink) = 3.16e-01
flows=16: P(no flow on the failed uplink) = 1.00e-02
flows=64: P(no flow on the failed uplink) = 1.01e-08
BFD detect ms, default 300 x 3: 900 | tuned 100 x 3: 300
prod hosts: 3072 | host links 48 x 25G = 1200 G vs uplinks 4 x 100G = 400 G -> 3.0 : 1
lab hosts as namespaces: 48 | injected host routes: 3024
BGP sessions prod: 256 lab: 32

Gates, written before the first run and agreed with the application owners (the numbers below are an example, derived from the 900 ms BFD detection time plus a 600 ms budget for route recomputation and forwarding updates):

Pipeline shape: lint the configs, deploy the lab (containerlab deploy -t fabric.clab.yml), poll until every session is established, run each test from the table writing JSON results, evaluate the gates in a script that exits non-zero on any miss, and always destroy the lab in a final step even when a test fails. Pull-request runs use the smoke subset (bring-up, one link failure, one silent failure); a nightly run does the full table.

What this lab cannot prove

A virtual NOS does not have the production ASIC's (the switch's custom forwarding chip) forwarding-table capacity, hardware programming time or hash function, so the absolute outage in the lab is a regression signal and an upper-level sanity check, not a promise. Compare each run to the baseline of the previous design in the same lab, because relative change transfers to hardware better than the absolute number does. The hardware limits, such as maximum routes in the forwarding table, still need a small physical pod or the vendor's published scale numbers, and the first production rack should be rolled out as a canary under the same gates.