Comprehensive principles and practical patterns for designing reliable, scalable, and secure networks across enterprise, data center, and wide area environments. Covers topology choices such as spine and leaf, hub and spoke, star, mesh, ring, and hybrid topologies, and how those choices affect resilience, latency, and scalability. Includes hierarchical design patterns for access distribution and core layers, device roles such as switches, routers, and load balancers, and cloud and data center specific patterns. Details redundancy and high availability strategies including active active and active passive failover, redundancy sizing such as N plus one, failure modes, convergence behavior, and operational implications for service level agreements. Addresses capacity planning, performance trade offs, quality of service and traffic engineering, routing and switching fundamentals and routing protocol behavior, segmentation for performance and security including virtual local area networks and subnets, and placement of security controls. Emphasizes trade offs between bandwidth, latency, cost, complexity, manageability, and operational burden, and how to select topology and design patterns for different application, performance, and operational requirements.
HardTechnical
49 practiced
Design a BGP/MPLS architecture for a multi-homed edge that provides VPN services (VRF per customer), supports traffic engineering (RSVP-TE or SR-TE), and includes route reflectors for scale. Discuss control-plane scaling, route-leak prevention, RTBH for DDoS mitigation, and operational concerns for managing many VRFs.
Sample Answer
**Approach & high-level design**- Use MPLS L3VPN (RFC4364) at the multi-homed edge: CE — eBGP — PE (two active PEs per site) — MPLS core. Provide a VRF per customer with route-target (RT) tagging. Use iBGP to route-reflectors (RRs) in the control plane for scale.**Control-plane scaling**- Route Reflectors: deploy redundant RRs (active/active) per region; use BGP Add-Path to avoid path-churn and allow ECMP; partition RR clusters by tenant scale (per-100k prefixes threshold).- RIB / CPU: enable BGP flow-spec and prefix-limits, soft-reconfiguration offload, and RIB/FIB scale testing. Use incremental RIB and best-path selection optimizations on PEs.**Traffic engineering**- RSVP-TE for strict bandwidth reservations and hop-by-hop LSPs (use where legacy RSVP is required).- SR-TE (policy-based SID lists) for scalable, centralized steering (preferred): program via PCE/SDN controller for dynamic path computation and automation; advertise SIDs via BGP-LS for telemetry.**Route-leak prevention**- VRF import/export rules: enforce explicit RT import lists; use BGP route-target constraint and RP/RT filters on RRs so RRs do not reflect routes into wrong VRFs.- Use BGP communities and RT-constraints on RRs to avoid transitive leakage; implement RPKI/ROA where applicable.**RTBH for DDoS mitigation**- Use BGP RTBH: accept specially-tagged next-hops (blackhole community) from DDoS scrubbing controller; distribute to PEs and inject discard next-hops via static/Null0 in VRF/global depending on scope.- For granular mitigation, combine RTBH with flow-spec (if supported) at PEs for packet-rate / port-based drops; ensure auth between scrubbing controller and BGP speakers.**Operational concerns (many VRFs)**- Automation: provision VRFs via templates (Ansible/Netbox/Controller), automate RT and route-policy creation.- Monitoring: per-VRF telemetry (SNMP, gNMI, streaming telemetry), prefix counts, RIB/FIB usage, per-VRF throughput.- Backup & testing: dry-run config validation, staged rollouts. Limit per-device VRF count; consider VRF-Lite or segmentation in aggregation layer.- Documentation & runbooks for failover, RTBH playbooks, and TE policy updates.This design balances scale, TE flexibility (RSVP where needed, SR-TE for operational scale), and security (route-leak controls + RTBH/flow-spec), with automation and telemetry to manage many VRFs.
EasyTechnical
43 practiced
Describe redundancy strategies used in infrastructure: N+1, 2N, active-active, and active-passive. Provide practical examples for power, spine switches, load balancers, and firewalls. Explain how you would size redundancy to meet a 99.99% availability SLA given MTTR and MTBF assumptions, and discuss cost versus availability tradeoffs.
Sample Answer
**Overview of strategies**- N+1: one spare for N units (tolerates single failure). Common, cost-effective.- 2N: full duplicate capacity (hot spare for every component). Higher cost, near zero risk.- Active-active: both (or all) nodes share load and can failover seamlessly; capacity and performance benefit.- Active-passive: passive standby takes over on failure; simpler but may have failover delay.**Practical examples**- Power: N+1 UPS per rack; 2N means dual utility feeds + dual UPS + dual PDUs so every device has independent full path.- Spine switches: N+1 = extra spine switch in fabric; 2N = mirrored spine plane; active-active = ECMP across both spines; active-passive = one spine preferred, other only if primary fails (rare).- Load balancers: active-active pair sharing traffic (HA with session sync); active-passive pair with VIP failover (keepalived).- Firewalls: active-active (stateful clustering, scales throughput); active-passive (primary handles traffic, standby syncs state).**Sizing for 99.99% SLA**Use availability A = MTBF / (MTBF + MTTR).
text
A = MTBF / (MTBF + MTTR)
For single component, to reach 99.99% (≈52.6 minutes downtime/year) you must ensure MTTR and MTBF satisfy the formula. For redundant components (independent, identical), system availability for N+1 (two units) approximated:
text
A_sys = 1 - ( (1 - A)^2 )
Example: if single unit A = 0.999 (≈8.76 hours downtime/year), two in N+1 gives A_sys ≈ 0.999999 (much better). Calculate required redundancy level given measured MTBF/MTTR and choose N+1 or 2N/active-active accordingly.**Trade-offs**- N+1: good balance cost/availability; acceptable for many services.- 2N / active-active: higher CapEx/Opex but minimal risk and lower failover complexity.- Active-active reduces failover time and provides capacity but adds complexity (state sync, split-brain handling).Choose based on criticality, budget, and operational maturity—use N+1 for non-critical tiers, 2N/active-active for business-critical systems with strict SLAs.
HardTechnical
46 practiced
Design a CI/CD pipeline for network infrastructure changes: include version-controlled configurations, automated linting and semantic validation, staged deployment (lab -> canary -> production), orchestration tools (Ansible/Terraform/Nornir), automated rollback, and audit trails. Explain how to test BGP/OSPF and complex changes safely in the pipeline and how to implement canarying for routing changes.
Sample Answer
**Overview / Goals**Create a git-driven CI/CD pipeline that validates, stages, deploys, and audits network infra changes with safe rollback and routing canarying for BGP/OSPF.**Pipeline stages**- Git repo per environment + PR workflow; branch protection + signed commits.- CI: automated linting (pyang/napalm-validate/ansible-lint/custom-schema), semantic validation (closed-loop diff of intended config vs. current device state via NAPALM/Nornir).- Unit tests: molecule for Ansible roles, terraform plan + tflint + checkov.- Staged deploy: lab -> canary -> prod. Each deploy step is gated by automated tests and human approval (policy).**Orchestration**- Terraform for infra provisioning (virtual routers, BGP session templates).- Ansible/Nornir for config push and readbacks; use idempotent templates + transactional commits where supported (netconf/candidate).- Use CI runner agents in each environment to avoid network reachability issues.**Canarying & routing safety**- Implement routing canary by changing only a subset of routers (one leaf/ASBR), or announce prepended/no-export community. Validate reachability with active probes (iperf, BFD, traceroute) and control-plane checks (show ip bgp neighbors; route acceptance filters).- For BGP: run route-injection simulator in lab, verify prefix acceptance, MED/LOCAL_PREF behavior, and ebgp/ibgp attributes. Automate neighbor flap/regression tests.- For OSPF: verify LSDB consistency, adjacency formation, SPF run times, and route distribution with topology emulation.**Automated rollback & safety**- Perform staged “dry-run” (check-only) and transactional commits; after push, immediately run health checks. If any check fails or control-plane anomalies detected, trigger rollback playbook that re-applies previous config from git or withdraws canary announcements.- Implement circuit-breaker: if packet loss or route convergence > threshold, pause rollout.**Testing approach**- Lab: full-emulation (EVE-NG/Containerlab) with CI that runs BGP/OSPF scenario scripts.- Canary: small live subset with synthetic traffic and passive telemetry (sFlow/NetFlow), streaming telemetry (gNMI) and telemetry-based assertions.- Use BGP route collectors and route diff tools to detect unintended leaks.**Audit & Observability**- All changes logged to git with commit->CI run IDs. Store device diffs, test results, and change approvals in an audit DB (ELK or Splunk). Correlate with telemetry and incident tickets.- Generate automated change tickets and attach playbook IDs.**Example safety checks (automatable)**- Pre-check: validate policy and ACLs, collision detection.- Post-check: BGP peer up, expected prefixes present, RIB/FIB match, latency/jitter thresholds.Trade-offs: stricter canary reduces blast radius but increases time-to-rollout. Use feature flags (communities) to control route propagation without full config swaps.
EasyBehavioral
41 practiced
Tell me about a time you created network documentation and runbooks for a complex system handoff to operations. Include how you ensured clarity of diagrams, verification steps, rollback procedures, runbook testing, and how you trained the operations team. Use the STAR method in your answer.
Sample Answer
**Situation**I led the handoff of a multi-region Kubernetes + VPN networking deployment from engineering to operations after a three-month rollout.**Task**Create clear network documentation and operational runbooks so on-call operators could verify, troubleshoot, and safely roll back changes.**Action**- I produced layered diagrams (high-level service flow, mid-level VPC/subnet maps, and low-level firewall/NAT rules) with consistent symbols and a legend; exported SVGs and PDF printables.- Runbooks included step-by-step verification (connectivity tests: curl, traceroute, cluster pod-to-pod, BGP session status), expected outputs, and automation scripts for common checks.- Rollback procedures listed prerequisites, ordered steps, configuration checkpoints, and safety gates (feature-flag disable + staged traffic cutover).- I ran tabletop and live drills: simulated failure scenarios, executed rollbacks in a staging tenant, timed steps, and iterated the runbook based on gaps.- Training: two 90-minute hands-on sessions, recorded videos, and a one-week shadowing period where operators executed verification and recovery under my supervision.**Result**Post-handoff, MTTR for network incidents dropped 45%, operator confidence and checklist adherence improved, and the runbook became the standard for subsequent rollouts.
EasyTechnical
46 practiced
A production link shows high packet drops during peak hours. As a systems engineer outline a structured troubleshooting approach: list telemetry and data to gather (counters, flow samples, packet captures), tools you'd use, likely root causes to consider (buffer drops, duplex/MTU mismatch, microbursts), and immediate mitigations to restore service.
Sample Answer
**Situation & goal**Quickly identify cause of high packet drops during peak hours and restore service with minimal risk.**Telemetry & data to gather**- Interface counters (ifInErrors/ifOutErrors, drops, collisions) via SNMP/NetConf- Queue/TC/shaper stats, buffer utilization (ASIC/TCAM counters)- sFlow/NetFlow/IPFIX samples for flow-level loss and top talkers- Packet captures (SPAN/tap) at ingress/egress during peak- Host metrics: retransmits, RTOs, CPU, NIC stats (ethtool)- Time-series (Prometheus/Grafana) for correlation with load- Configuration: MTU, duplex, QoS, ACLs, firewall logs**Tools**- tcpdump/wireshark, tcptrace- sFlow/NetFlow collectors (nfdump, Kentik)- SNMP polling/NetBox, Grafana, Prometheus- Switch/router CLI, ethtool, iperf for test traffic**Likely root causes**- Buffer overflow / tail drops from sustained bursts- Microbursts exceeding port buffer or shaping- MTU or duplex mismatch causing fragmentation/errors- Misconfigured QoS or policing causing drops- Hardware/NIC driver issues or CPU overload- ACLs/firewall or asymmetric routing drops**Immediate mitigations**- Apply temporary rate-limit or traffic shaping to offending flows- Increase queue/buffer or adjust QoS to prioritize critical traffic- Move heavy backups to off-peak; BGP/traffic engineering reroute- Temporarily disable policing that’s dropping legitimate traffic- Verify MTU/duplex and restart affected interfaces as needed- If symptomatic of hardware, failover to spare path/device**Follow-up**Root-cause fix, post-mortem with captured data, implement monitoring alerts and capacity/ QoS changes to prevent recurrence.
Unlock Full Question Bank
Get access to hundreds of Network Architecture and Design interview questions and detailed answers.
