Switching, VLANs, and Layer 2 Segmentation Questions

Direct answer

Spanning tree is the protocol that stops loops in a switched network by electing one switch as the root bridge (the reference point, the one with the lowest bridge ID: the priority number first, with the MAC address as the tie-break) and blocking every redundant port so each switch keeps a single forwarding path toward the root. A pod is a group of racks built and managed as one unit, and a ToR (top-of-rack) switch is the switch at the top of each rack that the servers plug into. Spanning tree hurts at pod scale because it makes redundancy cost bandwidth: it blocks every redundant link so each switch has one forwarding path to the root bridge, which means half of dual-homed uplink capacity sits idle, and a topology change can disturb forwarding while it reconverges. For the access layer I would put a multi-chassis link aggregation (MLAG, called vPC, virtual PortChannel, on Cisco Nexus) pair at the top of each rack, with servers dual-homed using LACP (Link Aggregation Control Protocol, which bundles several cables into one logical link) and ToR uplinks routed (each uplink is a layer 3 point-to-point link with its own IP address, and the routing protocol spreads traffic over all equal-cost uplinks, so there is no layer 2 loop to block) or aggregated into a port-channel, so every link forwards and failure is handled by link aggregation, not by a spanning-tree recalculation. I would still keep spanning tree running as a safety net on edge ports to stop a mistaken cable from causing a loop, and keep it as the main mechanism where a legacy switch or a device without LACP support must attach with redundant links.

Where spanning tree hurts (computed example)

Spanning tree (Cisco defaults to Rapid PVST+ on Catalyst 9000, with bridge priority 32768 and port priority 128) forces redundant paths into a blocked state. Example: a pod where each of 20 access switches has two 40G uplinks to two aggregation switches. Under a single spanning tree shared by all VLANs (or per-VLAN trees that all have the same root), one uplink per switch forwards: usable = 20 x 40G = 800G of 1,600G installed, 50%. Per-VLAN trees (PVST+, one tree per VLAN) or multiple spanning tree (MST, one tree per group of VLANs) can alternate the blocked link per VLAN. Concretely, with 20 VLANs, make aggregation switch 1 the root for VLANs 1 to 10 and aggregation switch 2 the root for VLANs 11 to 20: on each access switch the uplink to switch 1 forwards VLANs 1 to 10 and blocks 11 to 20, and the uplink to switch 2 does the opposite. Each 40G uplink carries 10 VLANs, so both are used and, if the VLANs carry equal load, the pod uses its full 1,600G on average. It recovers capacity on average, but any one VLAN (and so any single large flow in it) still has only one 40G path, and the load balance is a manual VLAN-to-root assignment that drifts as workloads move.
Other costs at scale:

• Blocked links are not tested by traffic until failure, so a bad standby link surfaces during an outage.
• Root election is fragile: a switch added with a lower bridge priority value than the intended root wins the election and reshapes the whole tree.
• Large single broadcast domains and many VLANs on shared trees amplify any mistake into a pod-wide event.

What to use instead, at layer 2

When I would still keep spanning tree

1. Always on edge ports as a protection: PortFast (moves the port straight to forwarding) plus BPDU guard (error-disables the port if it receives spanning-tree messages), configured with spanning-tree portfast and spanning-tree bpduguard enable on Cisco, or globally with spanning-tree portfast default (PortFast on all non-trunking access ports) plus spanning-tree portfast bpduguard default (BPDU guard on PortFast ports; the second command alone does not turn PortFast on). A server NIC bridge or a stray switch then cannot create a loop.
2. As the main protocol where it is the only safe option: attaching a legacy switch with redundant links to the pod, a small branch or lab pod where two cables are the entire redundancy need, or a device that cannot run LACP.
3. Root guard (spanning-tree guard root) and a deliberately set root where access switches connect to a shared layer.

Pitfalls

• Disabling spanning tree entirely "because MLAG is loop-free": a miscabled port-channel or a bridged server then floods the pod.
• Treating MLAG as free: it couples two switches' software versions and makes the peer link critical.
• Mixing MST and PVST+ domains without checking the root placement.
• Believing the hashing balances evenly: the hash (a fixed recipe that turns address bits into a link number) sends every frame of one flow down the same link so frames stay in order, which is why a few large flows can still fill one member link.

Direct answer

A layer 2 switch forwards frames by destination MAC address using its MAC address table, and only inside a VLAN. A layer 3 (multilayer) switch does that too, and additionally routes packets between subnets by looking up the destination IP address in a routing table. Inter-VLAN routing can be done by a router on a trunk (router-on-a-stick), by an SVI (switch virtual interface, a routed interface for a VLAN) on a multilayer switch, or on a routed port (a physical port switched to layer 3). For a campus I would use SVIs for the VLAN gateways and routed ports for uplinks; router-on-a-stick only for a small branch.

Layer 2 versus layer 3 forwarding

A host sending to a different subnet addresses the frame to its default gateway's MAC address and the destination IP to the far host. The gateway removes the old frame header, looks up the IP, and builds a new frame for the next hop.

The three methods

What the lines mean:

• interface GigabitEthernet0/0.10 creates a subinterface, a virtual interface carved out of the router's one physical port, here one per VLAN (the .10 is just a label that is conventionally the VLAN number).
• encapsulation dot1Q 10 tells that subinterface to send and accept 802.1Q-tagged frames (frames carrying a VLAN tag) for VLAN 10 only.
• ip address 10.10.10.1 255.255.255.0 makes the subinterface the default gateway of VLAN 10's subnet.
• The parent physical interface carries no IP address itself and must be enabled with no shutdown, otherwise none of its subinterfaces come up.
• On the multilayer switch, ip routing turns on the routing function, interface vlan 10 is the SVI, the virtual gateway interface for VLAN 10, and no switchport turns a physical port into a routed port.

Router-on-a-stick: the switch port toward the router must be a trunk allowing the routed VLANs. Cisco's Catalyst 9300 VLAN guide has a Layer 3 subinterfaces chapter, so a 9300 can itself run subinterfaces; the router in this layout can therefore be a router or a multilayer switch, but with SVIs available a multilayer switch has no reason to hairpin traffic through one link. In this design all inter-VLAN traffic crosses one physical link. A flow from VLAN 10 to VLAN 20 enters the router tagged 10 and leaves tagged 20 on the same link, so it uses the link once in each direction. On a 1 Gbps full-duplex trunk the aggregate routed rate across all VLANs cannot exceed 1 Gbps in either direction.

SVI: routing happens inside the switch, so inter-VLAN traffic does not leave the box, and it scales with the switching hardware. Each VLAN needs one SVI, so the number of VLANs you route is the number of SVIs. Cisco's Catalyst 9300 Layer 3 subinterfaces chapter lists a maximum of 1000 SVI interfaces and says not to configure more than 4,000 Layer 3 interfaces in total (routed physical interfaces, SVIs and subinterfaces); limits differ by platform and release, so check the documentation for yours. For a campus with a few dozen VLANs per switch these ceilings are far away. An SVI needs ip routing globally.

Routed port: no VLAN and no gateway sharing; it is a plain IP interface and is the clean way to join a distribution switch to a core.

Worked example

PC 10.10.10.10 (VLAN 10) pings 10.10.20.20 (VLAN 20). With an SVI gateway: the PC ARPs for 10.10.10.1, sends the frame to the switch's MAC, the switch finds the destination subnet connected on interface vlan 20, ARPs for the host, and forwards in a new frame. With router-on-a-stick the same exchange crosses the trunk to the router and back. Trace it with illustrative values, VLAN 10 on 10.10.10.0/24, VLAN 20 on 10.10.20.0/24, router subinterfaces .10 (10.10.10.1) and .20 (10.10.20.1), PC1 10.10.10.10 and PC2 10.10.20.20:

1. PC1 ARPs for its gateway 10.10.10.1 in VLAN 10; the router's Gi0/0.10 answers with the router's MAC (call it R).
2. PC1 sends a frame with destination MAC R and destination IP 10.10.20.20. The switch adds the VLAN 10 tag and sends it up the trunk.
3. The router receives it on subinterface .10 (tag 10 matches encapsulation dot1Q 10), finds 10.10.20.0/24 connected on subinterface .20, and ARPs for 10.10.20.20 with tag 20.
4. It builds a new frame, source MAC R, destination MAC PC2, tag 20, and sends it out the same physical port.
5. The switch removes the tag and delivers it to PC2's VLAN 20 access port.
   The trunk carried the packet twice (in tagged 10, out tagged 20); with an SVI that hop never leaves the switch.

Trade-offs and pitfalls

Pick SVIs plus routed uplinks unless a router is all you have. Apply ACLs on the SVI or firewall, because routing between VLANs with no policy removes the segmentation. If the branch grows past what one link can carry, move the gateways to a multilayer switch.

Direct answer

Ethernet has no hop counter in its frame header, and a switch floods broadcasts, multicasts and frames to unknown destinations out of every port except the one they arrived on. With redundant links and no loop prevention, those frames circulate forever and multiply, which is a broadcast storm. STP (Spanning Tree Protocol) keeps the redundant cabling but logically blocks enough ports to leave a loop-free tree, and unblocks one if a forwarding link fails.

What happens on the wire without it

1. A host sends an ARP broadcast. Switch 1 floods it out every other port, including both links toward the other switches.
2. Each switch that receives a copy floods it out all its other ports, including the next loop link, so the copy comes back around.
3. Each pass duplicates the frame onto every extra link, so copies multiply. In a full mesh of four switches the starting broadcast leaves on 3 links (3 copies) and each copy is re-flooded on 2 other links, giving 3, 6, 12, 24, 48 ... 1536 copies after 10 hops (3 x 2^9).
4. Links and switch CPUs fill with duplicate frames, legitimate traffic is crowded out, and every hop is also delivered to hosts.
5. The MAC address table (source MAC to port mapping) keeps changing: the same source MAC is seen arriving on different ports, so unicast frames follow the flapping entries and arrive duplicated or in the wrong place.

With only two parallel links between two switches, each broadcast yields two copies that circle forever in opposite directions, and every new broadcast adds two more, so the loop is permanent even without exponential growth.

How STP keeps the redundancy usable

1. Switches exchange BPDUs (bridge protocol data units, small hello messages sent between switches). Each switch has a bridge ID: 2 bytes of priority (default 32768) followed by the switch's 6-byte MAC address, compared as one number. The switch with the lowest bridge ID becomes the root bridge, the reference point every other switch measures from.
2. Every other switch keeps the single lowest-cost path to the root: the port on that path is its root port. Each link between two switches (a segment) also gets one designated port, the end that is closer to the root, and that port forwards.
3. Every remaining port is blocked: it receives BPDUs but forwards no user traffic, which cuts every loop.

Worked example with illustrative numbers: three switches A, B and C in a triangle, all links 1 Gbps (path cost 20,000 each under Cisco's long cost method), all at default priority 32768, with MACs 0011.1111.1111 (A), 0022.2222.2222 (B) and 0033.3333.3333 (C). Priorities tie, so the lowest MAC decides and A is root. B reaches A directly at cost 20,000, or via C at 20,000 + 20,000 = 40,000, so B's root port is the direct link; C does the same. On the B-C segment both switches offer a cost of 20,000 to the root, another tie, so the lower bridge ID wins: B's end is designated and forwards, C's end is blocked. The A-B and A-C links forward and the B-C link is cut at C, so the triangle becomes a loop-free path.
4. BPDUs continue on blocked links. Cisco's defaults are a 2 second hello, 15 second forward delay and 20 second max age. If a forwarding link fails and the blocked port stops hearing a better BPDU, classic STP waits the max age (20 s, how long a switch keeps the last BPDU before declaring it stale) and then passes through listening (15 s, the port discards user traffic and only takes part in the election) and learning (15 s, it still discards user traffic but starts building its MAC table). Each of those 15 s stages is one forward delay, so the worst case is roughly 50 seconds (20 + 15 + 15). Rapid PVST+ (the default mode in Cisco's Catalyst 9300 guide) uses a proposal and agreement handshake, in which two neighbouring switches agree directly with each other that a link is safe to forward on, instead of waiting on those timers, so it recovers much faster.
5. To avoid wasting the blocked link, per-VLAN spanning tree (PVST+, one separate spanning tree per VLAN) elects a root per VLAN. Picture an access switch X with one uplink to switch 1 and one to switch 2, and switches 1 and 2 linked to each other. Make switch 1 root for VLAN 10 and switch 2 root for VLAN 20. For VLAN 10, X's best path to the root is the uplink to switch 1, so the uplink to switch 2 blocks for VLAN 10; for VLAN 20 it is the other way round. Each uplink forwards for one VLAN and blocks for the other, so both carry traffic.

Trade-offs and pitfalls

STP trades capacity for safety: blocked links carry nothing until a failure. Do not disable it to get the bandwidth back; use a port-channel (EtherChannel, several physical cables bundled and treated as one link) so parallel links are one forwarding port, or route at layer 3. Edge ports that could receive a rogue switch need BPDU guard, because an unmanaged switch cabled into two wall ports is the everyday way a loop appears.

Direct answer

A switch learns by reading the source MAC address of every frame it receives and recording "this address lives behind this port, in this VLAN" in its MAC address table (also called the CAM table, after content-addressable memory, the fast lookup hardware that holds it). It forwards by looking up the destination address. A known unicast destination goes out one port only. An unknown unicast destination, a broadcast, or a multicast (unless the switch filters multicast, for example with IGMP snooping) is flooded out every other port in the same VLAN. A MAC address is the 48-bit hardware address of a network card.

How learning and forwarding work, in order

1. A frame arrives on port Gi1/0/5 in VLAN 10 with source MAC aaaa.aaaa.aaaa. The switch adds or refreshes the entry (VLAN 10, aaaa.aaaa.aaaa, Gi1/0/5) and resets that entry's age timer.
2. It looks up the destination MAC in the same VLAN's table. The table is effectively per VLAN: the same MAC may appear in two VLANs on two ports.
3. Three outcomes:
   • Known, on a different port: forward out that one port only.
   • Known, on the port the frame arrived on: filter (drop). The destination is behind the same port, for example a hub or a downstream switch, and already saw the frame.
   • Unknown: flood out all ports in that VLAN except the one it arrived on (the ingress port, meaning the port a frame enters by).
4. Entries age out. The usual Cisco default is 300 seconds, set with mac address-table aging-time <seconds> (check the platform's command reference for the permitted range and for what a value of 0 does). Static entries (mac address-table static) never age.

Per-entry fields and special destinations

• Broadcast (destination ffff.ffff.ffff) is never learned as a source and is always flooded within the VLAN: every host must process it.
• Multicast, where the switch does no multicast filtering, is flooded within the VLAN, because no multicast address ever appears as a source. A switch can be given a feature to restrict it to interested receivers (IGMP snooping, where the switch listens to hosts' group-join messages), which is outside the table described here.

What flooding means for a large broadcast domain

A broadcast domain is every port that receives a broadcast sent by any one host, which on a switch is one VLAN. Flooding turns each unknown unicast, broadcast and multicast frame into N-1 copies (if the VLAN has N active ports, one copy leaves every port except the one the frame arrived on, so a 24-port VLAN produces 23 copies of every flooded frame). Every host in the VLAN spends CPU on broadcasts (ARP, DHCP discovery, some discovery protocols), and every switch-to-switch link carries all of it. If the topology has a layer 2 loop that spanning tree has not blocked, flooded frames circulate forever and multiply (a broadcast storm), and the MAC table thrashes because the same source address keeps appearing on different ports.

Worked example: unknown unicast and aging

Host A (aaaa.aaaa.aaaa, port 1) sends to host B (bbbb.bbbb.bbbb, port 2) on a fresh 4-port switch in one VLAN.

1. A's first frame is an ARP request to the broadcast address. The switch learns A on port 1, then floods to ports 2, 3 and 4.
2. B replies, unicast to A. The switch learns B on port 2 and, because A is known, sends the reply out port 1 only. Ports 3 and 4 never see it.
3. A now sends to B: both are known, so each frame goes out exactly one port.
4. If B stays silent for 300 seconds (the default aging time) its entry expires. A's next frame to B is an unknown unicast and floods again until B transmits something.

Mitigating excessive flooding

• Shrink the flood scope: keep broadcast domains small. One VLAN per function, with a layer 3 boundary between them, bounds the flood scope. A /24 (254 usable addresses) per VLAN is a common ceiling in campus designs because broadcast load grows with host count.
• Prevent loops: keep spanning tree healthy so flooded frames cannot circulate. On user ports enable PortFast (the port goes straight to forwarding, skipping the listening and learning states) together with BPDU guard (if the port ever receives a spanning-tree BPDU, meaning a switch has been plugged in, the switch shuts the port into the err-disabled state). A stray switch then cannot change the topology.
• Stop table flooding attacks: cap MAC addresses per access port. Port security, the switch feature that limits how many and which MAC addresses a port may learn, has a default limit of one secure MAC address per port, raised with switchport port-security maximum <value>. The default violation mode is shutdown (error-disabled), and restrict or protect modes drop offending frames instead. This also blunts a MAC flooding attack, where an attacker fills the table with fake sources so legitimate destinations become unknown and are flooded.
• Avoid asymmetric paths that leave a destination unlearned. A switch learns only from frames it sees B send. Suppose A sends to B through switch sw-1, but B's replies travel back over a different path that never passes through sw-1. sw-1 never sees B as a source, so it never learns B's port and floods every frame addressed to B, for as long as the asymmetry lasts. Fix the path so replies cross sw-1, or add a static entry for B.
• Size the aging time to the traffic. Lengthening it above 300 seconds reduces re-flooding for quiet hosts but slows stale-entry cleanup after a host moves.

Trade-offs and pitfalls

• Learning is from the source only. A host that never transmits is never learned and always gets flooded traffic.
• A host that moves ports is relearned on its next frame. If the old entry has not aged, traffic misdirects only until that frame arrives.
• A MAC table has finite capacity. When a table is full, behaviour on new sources depends on the platform, so check the datasheet rather than assuming.

Direct answer

A VLAN (virtual local area network) is a way to split one physical switch network into several separate logical networks. Each VLAN is its own broadcast domain: a broadcast sent by one machine (a frame addressed to every device, such as an ARP or DHCP request) is delivered only to ports in the same VLAN. Think of one open-plan office floor that gets partitioned into rooms: same building and same wiring, but a shout stays inside its room, and getting from one room to another requires walking through a door (a router or layer 3 switch).

What changes once the network is split

Broadcasts. On a flat network every ARP (Address Resolution Protocol: "who has this IP address, tell me your MAC address") and every DHCP (Dynamic Host Configuration Protocol: automatic IP address assignment) discover reaches every host. With VLANs those broadcasts stop at the VLAN edge. Each VLAN is normally one IP subnet, so ARP only ever resolves addresses inside the sender's own subnet, and traffic to anything else is sent to the default gateway's MAC address. (A subnet is a block of IP addresses that share a prefix, such as 10.10.10.0/24; the default gateway is the router address a host sends to when the destination is outside its own subnet.)

Security boundaries. A VLAN gives you separation at layer 2: a server in VLAN 10 cannot be reached by a direct frame from a laptop in VLAN 20. Traffic between VLANs has to be routed, and the routing point (a firewall, a router, or the switch's layer 3 interface) is where you apply access control lists (ACLs: permit/deny rules). The boundary is only as strong as that routing point: if the gateway routes everything between VLANs with no ACL, the VLANs separate broadcasts but not risk. A VLAN is a segmentation tool, not a firewall.

Day-to-day operations.

• Moving a person or server to another network is a port configuration change, not a recabling job.
• Every VLAN needs its own subnet, its own default gateway, and its own DHCP scope (the pool of addresses the DHCP server may hand out for that subnet). A DHCP server in another subnet is reached through a DHCP relay agent on the gateway, which forwards the broadcast as unicast to the server.
• Switches are joined by trunk links that carry many VLANs over one cable, each frame labelled with its VLAN number. A VLAN missing from a trunk is the classic "works on this switch, dead on that one" fault.
• Shared environments with several tenants (customers or business units on the same switches) usually start with one VLAN per tenant, paired with separate routing instances (VRFs, virtual routing and forwarding tables: each tenant gets its own private routing table on the same router) or separate firewall policy per tenant. For example, tenants A and B may both use 192.168.1.0/24, which is an overlapping address plan. Each tenant's VLAN lands in its own VRF, so 192.168.1.10 in tenant A and 192.168.1.10 in tenant B never collide, because the router looks up each in a different table, and any traffic between tenants is allowed only where you deliberately permit it.

Worked example

A site has 600 devices: 200 servers, 200 user devices (199 PCs plus the shared printer) and 200 IP phones. Flat, they all share one subnet. A subnet of 2^h addresses has 2^h - 2 usable addresses (minus the network and broadcast addresses), where h is the number of host bits, so it must satisfy 2^h - 2 >= 600. With h = 9, 2^9 - 2 = 510, which is too small; with h = 10, 2^10 - 2 = 1022, which fits. Host bits are 32 minus the prefix length, so h = 10 means a /22. Every broadcast then reaches the other 599 devices. Split into three VLANs of 200 devices each, each subnet needs 2^h - 2 >= 200: h = 7 gives 126 (too small), h = 8 gives 254, so a /24 per VLAN:

A broadcast now reaches 199 other devices instead of 599, and the printer in VLAN 20 no longer sees server chatter. (Addresses and counts here are illustrative.) The users-to-servers path crosses the gateway, where an ACL can permit only the application ports the users need.

Trade-offs and pitfalls

• More VLANs means more gateways, DHCP scopes and trunk allowed-lists to keep consistent. Name and document them.
• Do not treat a VLAN as a security control by itself. Leaving unused ports in a default VLAN, or trunking user VLANs carelessly, weakens the separation.
• Keep VLAN size sensible: a VLAN stretched across the whole campus enlarges the failure domain (a layer 2 loop affects every switch it touches).