Network Architecture and Topology Questions

Design of network topologies and segmentation for scale and reliability. Covers data center and backbone topologies such as spine leaf, hub and spoke, full and partial mesh, WAN options including MPLS and SD WAN, redundancy and failover strategies, segmentation and micro segmentation for security, and design considerations for global and regional network growth and performance.

HardTechnical

71 practiced

You must choose between building a private leased-fiber backbone and using internet-based overlays with SD-WAN for regional connectivity. Propose a cost-performance analysis framework that includes upfront capex/opex, per-Mbps bandwidth cost, expected traffic growth, SLA risk and cost of downtime, peering/transit considerations, and recommend criteria to decide when each approach is preferable.

Sample Answer

**Approach summary**Provide a TCO + risk model that compares private leased fiber vs. SD‑WAN overlays across cost, performance, and business impact over a 3–5 year horizon.

**Framework (step-by-step)**1. Scope & timeframe - Define sites, peak/avg traffic, planning horizon (e.g., 5 years), growth rate.

2. Upfront CapEx / OpEx - Leased fiber: construction/installation, CPE, mux/optical gear (CapEx); recurring lease, maintenance, power, repair SLAs (OpEx). - SD‑WAN: CPE appliances, orchestration licenses (CapEx); internet circuits, cloud appliance licenses, monitoring, vendor support (OpEx).

3. Per‑Mbps cost - Calculate $/Mbps/month = (annualized CapEx + annual OpEx) / average provisioned Mbps. Include headroom and burst pricing.

4. Traffic growth - Model traffic growth per site; compute upgrade cadence and incremental costs for both options.

5. Performance & SLA risk - Quantify expected latency, jitter, packet loss for both. Assign probability of outage and mean time to repair (MTTR). Map SLA violations to financial impact (revenue loss, SLA credits, remediation costs).

6. Cost of downtime - Multiply outage probability * downtime duration * cost per hour. Add expected remediation and reputational costs.

7. Peering / transit considerations - For SD‑WAN, include transit/peering costs, egress fees, and potential variability on carrier paths. For private fiber, include interconnection costs at POPs and cross‑connect fees.

8. Risk-adjusted TCO - Sum expected costs + downtime risk to produce net present value per option. Run sensitivity analysis on bandwidth growth, outage rates, and transit pricing.

**Decision criteria (recommendation)** - Choose private leased fiber when: consistent high bandwidth (>100–500 Mbps/site depending on scale), low latency/jitter needs (real‑time apps), predictable growth, and downtime cost is high such that reduced outage probability justifies premium. - Choose SD‑WAN when: variable/elastic bandwidth needs, cost per Mbps must be low up-front, multi‑provider resiliency and rapid provisioning are priorities, and workloads tolerate best‑effort internet with application‑aware routing. - Hybrid: use fiber for critical hub sites and SD‑WAN for branches; enforce SLA-sensitive traffic via private links and overflow or backup via SD‑WAN.

**Outputs to present** - TCO table (yearly), $/Mbps curves, outage cost comparison, sensitivity charts, and clear recommendation with break‑even bandwidth/uptime thresholds.

HardSystem Design

57 practiced

Design an active-active multi-region application network where application instances run in two regions simultaneously and user sessions require sticky behavior. Address session affinity, global server load balancing, database replication strategy (master-master vs active-passive), convergence and split-brain risk, and how to minimize user-perceived latency.

Sample Answer

**Clarify goals & constraints**- Active-active: two regions serve traffic concurrently; user sessions must be sticky.- Priorities: consistency vs latency, availability, and avoiding split‑brain.

**Global server load balancing & steering**- Use GSLB with multi-pronged approach: - BGP Anycast for low-latency TCP/UDP steering to nearest PoP. - DNS-based GSLB (geo-aware) as fallback to direct clients to preferred region. - Health checks (layer‑4 and application probes) and weighted failover to shift traffic instantly on regional failure.- Always authenticate and verify health before routing traffic.

**Session affinity**- Prefer stateless session tokens (JWT) so any region can resume session—ideal network-friendly approach.- If server-side state required, use a geographically distributed session store: - Primary pattern: local session store + asynchronous replication to remote (fast reads local). - Stronger pattern: active-active Redis with CRDTs or Redis Enterprise Multi‑Primary for cross-region writes.- Alternative: GSLB sticky by cookie/IP (last resort)—works but brittle over NAT/CGNAT and client mobility.

**Database replication strategy**- Recommend active-passive for stateful relational DBs unless application supports conflict resolution: - Active-passive (primary in Region A, replica in Region B with automated failover via quorum/leader election) minimizes split‑brain risk and simplifies consistency. - If active-active required, use distributed consensus (Raft/Paxos) or a database designed for multi-primary (e.g., CockroachDB, Yugabyte) to ensure transactional correctness and conflict resolution.- Replication over private backbone/MPLS or cloud provider backbone to reduce latency and improve security.

**Convergence & split‑brain mitigation**- Use quorum-based leader election and fencing (STONITH) for DB leaders.- Use synchronous replication only where latency acceptable; otherwise asynchronous with monitored lag plus safe read-after-write via session affinity or request routing.- Implement automated health orchestration (controller) that requires multi-signal confirmation before declaring a region dead. Use tie-breakers (e.g., cloud zone metadata) to avoid dual primaries.

**Minimizing user-perceived latency**- Localize reads: serve static and cacheable content from CDN/edge PoPs.- Use Anycast+regional load balancers so TCP handshake and SSL terminate close to user.- Optimize WAN: dedicated inter-region links, TCP BBR, MTU/path-MTU tuning, WAN acceleration for DB replication when synchronous needed.- Use read replicas in both regions for low-latency reads; route writes to leader or use local write with conflict resolution for multi-primary DBs.

**Network engineering specifics**- Protect control/replication plane over private peering or VPN with QoS and route preference to avoid flapping.- Monitor BGP/health state, replication lag, session store TTLs. Implement playbooks for failover and rollback.- Test regularly: chaos drills (region fail, network partitions) and measure RTO/RPO SLA.

Trade-offs: active-passive simplifies consistency and reduces split‑brain but increases failover cost; true active-active reduces failover latency but requires distributed DBs and stronger conflict resolution and network guarantees. Choose based on SLA and application tolerance for eventual consistency.

MediumSystem Design

62 practiced

Design a disaster recovery network failover strategy between primary and DR sites in different regions. Goals: RTO under 15 minutes for the application, minimal manual reconfiguration, and reliable validation via automated testing. Include routing failover approach (DNS vs BGP), tunnel pre-provisioning, data replication considerations, and a testing plan.

Sample Answer

**Clarify goals & constraints**- RTO < 15 min, minimal manual steps, automated validation, cross-region primary ↔ DR.

**High-level approach**- Active–passive with hot DR (pre-provisioned infra & tunnels), BGP for network-level failover, DNS low-TTL as application-level fallback. Automate runbook via orchestration (Terraform + Ansible + IaC pipelines).

**Routing failover: BGP vs DNS**- Primary: advertise prefixes from primary AS via BGP to cloud/on‑prem peers. DR: keep BGP sessions established but withdraw primary routes on failover and advertise DR prefixes. BGP gives sub-15min cutover and preserves TCP continuity where possible.- DNS: set low TTL (30–60s) and health-checked records (Route53 health checks / external monitor) as secondary. Use DNS only for global load balancing and as fallback for clients without BGP reachability.

**Tunnel pre-provisioning**- Pre-establish IPsec/DMVPN/CloudTransit tunnels between sites and peers; keep them up but idle. Use BGP over tunnels with next-hop tracking. Automate tunnel rekeying and monitoring; store keys in vault and automate rotation.

**Data replication**- Synchronous for critical DBs within latency bounds; otherwise asynchronous with write-ahead logs and point-in-time recovery. Use replication lag monitoring and automatic promotion scripts ensuring consistent application state. Ensure storage snapshots replicate to DR region regularly.

**Automation & orchestration**- Failover playbook that: validates DR health, withdraws primary BGP adverts, enables DR adverts, promotes DB, updates DNS if needed, and runs smoke tests. All steps executable via CI/CD.

**Testing plan**- Weekly automated DR drills in a staging window: simulate BGP withdraw, test tunnel failover, promote DB, run full integration tests (API, auth, data integrity). Metrics: total RTO measured, replication lag, packet loss, session interruption. Quarterly full failover with rollback validation.

**Trade-offs**- BGP complexity and coordination with providers vs faster network-level switch; DNS simpler but higher client dependency. Choose BGP primary + DNS as controlled fallback for fastest, reliable cutover.

HardTechnical

75 practiced

You cannot use AS-path prepending to influence inbound traffic from some upstream ISPs. Explain advanced BGP techniques you can use to influence inbound traffic without AS-path prepending, including communities, local preference manipulation via upstreams, selective prefix advertisement, MED adjustments, and Anycast. Describe the operational coordination that may be required with ISPs.

Sample Answer

**Overview / Goal**Explain ways to influence inbound BGP traffic when you cannot change AS‑path prepending. Focus: BGP communities, upstream LP changes, selective advertisement, MED, Anycast — plus required ISP coordination and operational considerations.

**Techniques**

- BGP Communities - Ask upstreams to honor well‑known or provider‑specific communities that set local preference or route acceptance (e.g., no‑export, or provider tags like “set LP 50”). - Example: tag prefixes with community X to make them less preferred at ISP A so traffic enters via ISP B.

- Local‑Preference via Upstreams - Coordinate with ISPs to set lower/higher LP inside their network for your prefixes. This is very effective because LP dominates AS‑path. - Use communities to request LP values per upstream/region.

- Selective Prefix Advertisement - Announce more specific / longer prefixes to preferred upstream(s) only; suppress them at others. More specifics win, steering ingress. - Be careful with RPKI, filtering and aggregation policies.

- MED (Multi‑Exit Discriminator) - If multiple connections to the same ISP (or same neighboring AS), request they honor MED to prefer specific entry points. MED is considered after LP, so requires upstream cooperation.

- Anycast - Deploy same prefixes from multiple locations. Incoming traffic goes to nearest/most preferred site by routing metrics. Combine with per‑site announcements and communities for regional steering. - Use Anycast with health checks and coordinated traffic management.

**Operational Coordination**- Document provider community values, filtering and max‑prefix policies.- Get written confirmation which communities and MED/LP actions each ISP honors.- Test in maintenance windows; monitor BGP table changes, traffic flows and reachability.- Implement gradual rollout, caps and alarms to detect route leaks or hijacks.- Maintain playbooks for rollback, DDoS impacts, and RPKI/IRR constraints.

**Trade‑offs & Risks**- Reliance on ISP policy — not standardized; may change.- MED ignored by many providers; LP overrides MED.- Selective deaggregation increases RIB/DFZ pressure.- Anycast adds operational complexity (stateful services, session affinity).

This approach combines technical mechanisms and proactive ISP coordination to achieve predictable inbound traffic engineering without AS‑path prepending.

HardTechnical

74 practiced

Design an EVPN-VXLAN fabric that must scale to 100,000 endpoints across multiple sites. Explain how you would address MAC and ARP scaling, control-plane scaling using route-reflectors, the role for Type-5 routes for external L3 reachability, overlay MTU concerns, and strategies to avoid broadcast storms or excessive flooding.

Sample Answer

**Situation & goals**Design an EVPN/VXLAN fabric to support ~100,000 endpoints across multiple sites with predictable control-plane scaling, minimal MAC/ARP flood, inter-site L3 reachability, and safe BUM behavior.

**MAC & ARP scaling**- Use EVPN Type‑2 MAC/IP advertisements so MAC learning is control-plane driven (no data‑plane flooding for learning).- Enable ARP/ND suppression (EVPN ARP-Proxy): edge VTEPs advertise host IP⇄MAC so ARP is answered locally—removes broadcast ARP storms.- Aggregate at the edge where possible: use L2 segment limits and per-VNI scaling plans (e.g., split large L2 domains into multiple VNIs or use routed access).

**Control-plane scaling (Route-Reflectors)**- Deploy dedicated BGP EVPN Route-Reflectors (RRs) as scale-out clusters (N+1 RRs; 3–5 minimum, more for 100k entries). Ensure RRs are high‑memory platforms.- Use RR client topology (leafs as clients) — avoid full-mesh. Use EVPN route-target constraints and RR‑cluster IDs to reduce churn.- Enable BGP bestpath and selective route reflection (RT‑based filtering) to limit routes per client. Monitor and autoscale RR instances.

**Type-5 routes for external L3 reachability**- Use EVPN Type‑5 (IP Prefix) to advertise L3 prefixes (L3VNI) across fabric for inter-subnet reachability without hairpinning to a central router.- At aggregation, inject and originate Type‑5 for external prefixes (e.g., VRF import/export via RT). Combine with L3 gateway (IRB) per VNI for local routing.

**Overlay MTU**- VXLAN overhead ~50 bytes (outer UDP/IP/ETH); require MTU >= 1600 recommended. Enable jumbo frames on physical underlay and hosts or use MSS clamping on virtual routers if hosts cannot be changed.- Validate path MTU across underlay, include tunnels, and monitor PMTU.

**Avoiding broadcast storms / excessive flooding**- Use ARP suppression + ingress replication or PIM multicast in underlay to eliminate BUM flooding.- Prefer ingress-replication where multicast is limited; scale with underlay bandwidth planning.- Implement control-plane rate limits, storm-control on access ports, and per-VNI limits.- Use split-horizon and EVPN ES‑filters for multihoming to prevent loops; enable graceful‑restart/fast‑convergence features.

**Operational notes**- Capacity-test RRs and VTEPs with 100k simulated endpoints; tune RIB/FIB memory.- Telemetry: monitor EVPN route counts, BUM bandwidth, ARP hit/miss, MTU drops.- Document VNI/RT plan, RR clusters, and upgrade paths.

This approach gives deterministic MAC/ARP behavior, scalable BGP control-plane via RRs, efficient L3 reachability with Type‑5, and safe BUM handling with ARP suppression + ingress replication.

Unlock Full Question Bank

Get access to hundreds of Network Architecture and Topology interview questions and detailed answers.

Join thousands of developers preparing for their dream job.