Network Design and Architecture Questions

Requirements & constraints:

• 100k RPS global sustained, 99.99% availability (~4.38 min downtime/year), three regions (active-active), tolerate single-region failure, regional latency targets <150ms for users in-region.

High-level approach:

• Active-active across three regions with traffic steered by global DNS+GSLB (DNS + Anycast + regional load balancers). Traffic splits based on geo-proximity and real-time region capacity. Each region sized to handle ~50% of normal load (N), with cross-region overflow capacity to survive one region failing.

Load distribution:

• Global layer: Anycasted edge (Cloud CDN/edge or BGP Anycast) + GSLB with health-aware latency-based routing.
• Regional layer: Public regional LBs (N LB nodes in AZs) → regional ingress (K8s ingress / API GW autoscaled) → stateless app pools → backing state tiers (sharded DB/cache).

Capacity planning (numbers & buffers):

• Target steady-state per-region: 40k RPS each (even split 100k/3 ≈ 33k, add 20% buffer → ~40k). If one region fails, remaining two must absorb 100k → 50k each.
• LB sizing: choose LBs that support >100k concurrent RPS aggregate; per-region deploy at least 3 LB instances each rated ≥20k RPS (N+1). Network throughput: estimate request size; e.g., avg response 50KB → 100k RPS ≈ 5 GB/s global. Plan NICs, routers, and firewall throughput accordingly.
• Compute: container pods sized to serve 200 RPS each → need 200 pods per region for 40k RPS; autoscale with headroom to 250 pods to absorb surge.
• State tiers: caches sized for miss-rate to keep DB QPS under limits. DB replicas per region sized for peak QPS; cross-region read replicas for failover.

Redundancy & fault domains:

• Multi-AZ per region (3 AZs). LB, ingress, pods, cache and DB replicas across AZs.
• Devices: routers/firewalls in HA pairs, use ECMP and BGP sessions, N+1 redundancy.
• Graceful degradation: circuit-breakers, rate-limits, feature flags to reduce load on backend systems.

Cross-region failover:

• GSLB monitors regional health (latency + app-level). On region degraded, advertise withdrawal via Anycast/BGP and shift traffic by updating DNS + traffic steering; TTL low (30–60s) for faster convergence.
• Warm standby capacity: regions keep autoscaling margins and pre-warmed caches; use traffic-scaling policies to ramp up compute quickly.
• Data consistency: use async replication with conflict resolution; for critical writes, route to primary region or use quorum-based multi-region writes where required.

Health checks and observability:

• Multi-layer health checks: network-level (ping/BGP), LB-level (TCP/HTTP), app-level (business-health endpoints checking DB/cache), synthetic user journeys from multiple locations.
• Metrics & alerts: RPS, latency P50/P95/P99, error rates, capacity utilization, instance start times. SLIs mapped to SLOs (99.99% availability).
• Distributed tracing and logs for root-cause.

Testing & validation:

• Capacity tests: ramp tests using distributed load generators from multiple regions to simulate 100k+ RPS with realistic traffic mix; run soak tests for hours/days.
• Failover tests: scheduled game-day exercises:
  • Region kill: withdraw BGP or shut down a region’s LBs to validate global failover and capacity absorption.
  • AZ failure: simulate AZ loss in region and verify intra-region redundancy.
  • Partial degradations: inject latency, DB failures, cache outages to validate graceful degradation.
• Chaos engineering: automated, controlled chaos runs with rollback playbooks.
• Observability validation: ensure alarms trigger and runbooks produce required run times.
• Post-test: capture metrics, validate SLOs, tune autoscale thresholds, and adjust capacity buffers.

Trade-offs:

• Active-active increases complexity (data consistency) but reduces failover time. Warm standby reduces cost vs full duplication but requires careful warm-up. DNS TTL impacts failover speed; combine DNS with Anycast for faster convergence.

This design gives capacity headroom, multi-layer redundancy, health-aware routing, and a test-driven validation plan to meet 100k RPS with 99.99% availability.

Requirements (clarify):

• Active/passive gateway for dual-stack clients (IPv4 + IPv6).
• Health checks for backend services; automatic failover if health fails.
• Optional BFD to detect link/node failure quickly.
• Prevent split‑brain and ensure safe recovery.

High-level design:

• Two or more gateway nodes running keepalived (same version), configured with VRRP for IPv4 and IPv6 addresses.
• Health probe scripts (track_script) check local backend (e.g., HTTP, TCP, BGP session) and adjust VRRP priority.
• Optional BFD between neighbors to accelerate failure detection (reduces VRRP timers).
• Monitoring/alerting integrated (Prometheus + blackbox/exporter), and a datastore (consul/etcd) or fencing for multi-site/quorum use.

Key keepalived config elements (examples):

vrrp_instance (core):

• state MASTER|BACKUP — initial role
• interface — physical interface
• virtual_router_id — VRID (same across nodes)
• priority — higher wins (adjusted by track_script)
• advert_int — advertisement interval (default 1s; lower with BFD)
• authentication — simple|AH (prevents rogue VRRP)
• virtual_ipaddress / virtual_ipaddress6 — IPv4/IPv6 addresses
• preempt/preempt_delay — control takeover timing
• garp_master_delay/notify — send gratuitous ARP or ND on failover

Health checks:

• track_script { script "/usr/local/bin/check-backend.sh"; interval 2; weight -50; }
• The script returns 0 on success; keepalived adjusts priority by weight on failure.
• Use HTTP/TCP/exec checks; for complex checks use systemd service that emits status to a local socket.

BFD integration (optional):

• Configure bfd peers on both routers, then in vrrp_instance: bfd yes; lower advert_int.
• BFD detects L2/L3 neighbor down in tens of ms; VRRP reacts and failover occurs faster.

Prevent split-brain:

• Use VRRP authentication (sha1/AH) to prevent fake adverts.
• Keep consistent and deterministic priorities (derived from role + node ID).
• Use interface tracking (ip link or neighbor) to lower priority on interface flaps.
• Use preempt_delay and nopreempt where appropriate to avoid churn.
• In multi-site, implement quorum/fencing: require an external lock (consul/etcd/STONITH) before a node assumes MASTER; or use unicast VRRP so only known peers participate.
• Configure gratuitous ARP/ND to quickly update L2 tables and avoid stale ARP entries.

Sample keepalived snippet (IPv4+IPv6 + track_script + BFD):

vrrp_instance VI_1 {
  state MASTER
  interface eth0
  virtual_router_id 51
  priority 200
  advert_int 1
  authentication {
    auth_type AH
    auth_pass <hexkey>
  }
  bfd {
    enable
    interval 50
    multiplier 3
  }
  track_script {
    chk_backend
  }
  virtual_ipaddress {
    192.0.2.10/32 dev eth0
  }
  virtual_ipaddress6 {
    2001:db8::10/128 dev eth0
  }
  notify_master "/usr/local/bin/on_master.sh"
  notify_backup "/usr/local/bin/on_backup.sh"
}
vrrp_script chk_backend {
  script "/usr/local/bin/check-backend.sh"
  interval 2
  weight -100
}

Why this works (reasoning):

• VRRP provides a single virtual address; keepalived track_script manipulates priority so the healthiest node becomes master.
• Authentication + unicast or BFD reduces false masters and speeds detection.
• GARP/ND and notify scripts handle network propagation and actions (route updates, firewall rules).

Validation tests (what to run and expected behavior):

1. Passive health-failure test:
   • Kill backend service on MASTER. Expect: track_script fails -> MASTER lowers priority -> BACKUP takes MASTER within advert_int*multiplier (or BFD detection) -> virtual IP moves -> client traffic restored. Validate with ping/traceroute and curl to service IP.
2. Node hard-fail test:
   • Power off MASTER. Expect quick failover; BFD lowers detection time. Verify VIP moves and no in-flight connection blackhole beyond expected window.
3. Interface flap:
   • Bring MASTER eth0 down/up. Expect priority drop and failover; when restored, preempt_delay prevents immediate takeover if configured.
4. Split-brain simulation:
   • Isolate nodes with network partition where both can see backend but not each other. Validate that quorum/fencing prevents both assuming MASTER (if quorum configured) or that authentication prevents bogus adverts. If unprotected, observe split-brain and adjust design (add quorum or fencing).
5. ARP/ND propagation:
   • After failover, check neighbor tables on adjacent switches and clients; verify gratuitous ARP/ND was sent and resolves quickly.
6. BFD stress:
   • Introduce flaps on physical link; validate BFD sessions go down and VRRP transitions quickly without oscillation. Monitor counters and use dampening (preempt_delay) to avoid thrash.
7. Recovery test:
   • Restore original MASTER; check preempt behavior (does it reclaim MASTER as desired). Validate notify scripts executed and routes updated gracefully.
8. Observability and alerting:
   • Verify metrics/alerts trigger on VRRP state changes, health check failures, BFD session state, and track_script events.

Metrics & SLOs:

• Target failover RTO (e.g., <500ms with BFD, <2s without).
• Monitor VRRP adverts, BFD sessions, track_script failure rate, and VIP move latency.

Operational best practices:

• Keep identical keepalived versions and synchronized clocks.
• Store configs in IaC (Ansible/Terraform) and test in staging.
• Use automated rollback and runbooks for failover/restore.
• Limit VIP count per node and use ARP/ND rate limits to avoid network storms.
• Regularly test failover in maintenance windows and run postmortems.

This design provides dual-stack high availability, fast detection with BFD, safe prevention of split‑brain via authentication and quorum/fencing, and clear tests to validate behavior.

Requirements & constraints:

• Critical service SLO: p99 latency < 5ms; availability 99.99%
• Bulk analytics: throughput-heavy, best-effort
• Multi-site and cloud hybrid, must be auditable & compliant

High-level segmentation:

• On-prem: separate VLANs + VRFs:
  • VLAN 10 / VRF svc-critical (service hosts, load balancers)
  • VLAN 20 / VRF analytics (data pipelines)
• Cloud: separate VPCs/subnets and routing tables; use VPC peering/Transit Gateway with route tables enforcing isolation
• Microsegmentation: Kubernetes network policies / security groups limit cross-traffic to only required ports

Traffic marking & QoS:

• End-to-end DSCP marking: critical traffic marked EF (DSCP 46) at ingress (app or sidecar) and preserved across network. Analytics marked CS1 or AF11 (lower priority).
• Edge devices (routers, NICs) trust DSCP only from trusted subnets; otherwise re-mark to default.

Shaping & policing:

• At egress on top-of-rack and edge routers: strict priority queue for EF traffic (low-latency queue), weighted queues for others.
• Policing to protect EF: token-bucket policer set slightly above expected peak critical throughput (e.g., CIR = 80% of provisioned capacity) to avoid starvation and prevent malicious bursts; excess EF can be dropped or downgraded to AF31.
• Traffic shaping for analytics: rate-limiters (tail-drop or ECN) and large-buffer pacing to maximize throughput without impacting EF queues.
• In Linux hosts or kube nodes use tc with clsact qdisc, fq_codel for analytics, and mqprio/ets for SR-IOV NICs.

Monitoring, enforcement & audit:

• Metrics: per-class latency, queue lengths, drops, policing events, bandwidth usage. Collect via Prometheus (node exporters, eBPF qdisc exporters), SNMP/telemetry (sFlow/IPFIX), and router APIs (gNMI).
• Traces: distributed traces instrumented to tag DSCP/class so requests can be correlated to network treatment.
• Alerts: p99 latency, queue drops, policing rate exceed alerts; integrate with PagerDuty.
• Auditing: log DSCP markings at edge, record config changes via IaC (GitOps), and periodically run active probes (synthetic traffic with marked DSCP) to verify end-to-end priority handling; store immutable logs for compliance.
• Testing: staged chaos (packet loss/latency injection, burst analytics) to validate SLOs. Include annually or on major change a compliance test report.

Trade-offs & governance:

• Strict priority risks starving lower classes — mitigate with policing and minimum bandwidth reservations.
• Require operational processes: change control for QoS, periodic reviews of DSCP mappings, and RBAC on network devices.

This design ensures critical traffic is segmented, marked, shaped and monitored end-to-end with measurable enforcement and auditability to meet latency SLOs.

Requirements clarification:

• 100+ sites (DCs + hybrid cloud), low-latency path selection, BGP policy control, secure private links, graceful failover, centralized control plane, per-site telemetry, and operational runbooks for routing changes. Target SLA: 99.99% for inter-site connectivity.

High-level architecture:

• Centralized SD‑WAN controller + orchestrator for policy, telemetry, and certificate management.
• Underlay: multiple transport links per site (MPLS, private dark fiber, Internet, cloud-direct connect).
• Overlay: encrypted tunnels (IPsec/DTLS/WireGuard) between site edges and a set of regional gateways; dynamic path selection handled by SD‑WAN control plane.
• Routing fabric: iBGP with hierarchical route reflectors (global RR + regional RRs) + per-site BGP sessions to local regional gateway.

Edge devices and placement:

• Physical CPE or virtual edge (x86 appliance / cloud VM) at each site supporting: BGP, IPsec/DTLS, BFD, QoS, telemetry agent, local policy engine.
• Regional gateways in each region (N+2 HA), act as egress/ingress points to underlay and peer with route reflectors.
• Route Reflectors and Controller in HA across cloud regions.

Routing design & BGP policies:

• Underlay and overlay separation: Underlay handles IP reachability; overlay advertises site prefixes.
• Use BFD for sub-100ms failure detection between adjacent edges and gateways.
• iBGP full-mesh avoided: use RRs. All edges peer to regional RR; RRs peer to global RR.
• Route policies:
  • Prefix tagging (community/origin) per site/class.
  • Local-pref to prefer low-latency regional paths; AS-path prepending for deprioritizing backup links.
  • MED used to influence cross-region inbound traffic.
  • Export filters to prevent leak of internal-only routes to cloud/public peers.
• ECMP + latency-aware path selection: controller collects active latency probes and adjusts local-pref or installs policy-based routing (PBR) to steer flows.

Encryption & secure links:

• IPsec with IKEv2, AES‑GCM-256, PFS; use DTLS for UDP-based overlays if needed; WireGuard optional for cloud-native sites.
• Certificate-based mutual authentication managed by controller (short-lived certs), automated rotation.

Graceful failover:

• Fast local detection: BFD triggers routing withdrawals and local-pref adjustments.
• Stateful application-aware failover: controller can signal per-flow steering; for TCP-heavy flows use graceful migration: replicate NAT/state where possible or allow session re-establishment on backup path.
• Staged failover: primary -> warm standby -> cold standby via incremental policy changes.

Monitoring & telemetry:

• Per-site exporters sending: BGP state, BFD, tunnel stats, latency/jitter probes, interface counters, CPU/mem, config drift.
• Central observability: Prometheus + Thanos for metrics, ELK/Opensearch for logs, Grafana dashboards, alerting via PagerDuty.
• Synthetic tests: continuous end-to-end latency and application-level checks between representative site pairs.
• Anomaly detection: ML-based baseline to alert on path regressions.

Automation & orchestration:

• Controller APIs + IaC (Terraform) for declarative configs; push via NETCONF/RESTCONF/gNMI or vendor APIs.
• CI/CD for network: pre-validate configs in staging emulation; automated rollback on health regressions.

Operational runbooks (routing change example: modify regional local-pref to reroute traffic):

1. Prepare:
   • Validate change in staging; prepare rollback config.
   • Notify stakeholders and schedule maintenance window if needed; set change ticket.
2. Pre-checks:
   • Verify current topology, BGP states, baseline telemetry (latency, packet-loss).
   • Ensure backups healthy and BFD sessions up.
3. Apply change (canary):
   • Push policy to 1–2 non-critical sites or regional gateway; wait 5–10 minutes.
   • Monitor: BGP convergence, flows, application performance, CPU/memory.
4. Validate:
   • If metrics within thresholds, progressively roll out region by region.
   • If anomaly, execute rollback: restore previous policy, clear caches, and monitor recovery.
5. Post-change:
   • Run full telemetry sweep, update runbook with lessons, close change ticket.
6. Automated safeguards:
   • Controller enforces rate-limited changes, automatic rollback on threshold breach (packet loss, RTT increase, BGP flap count).

Trade-offs & scalability:

• Using RRs + controller reduces BGP session scale; regional gateways limit blast radius.
• Centralized policy gives consistency but requires robust HA and strong security for controller.
• Latency-aware steering is reactive; consider hybrid approach: proactive measurements + historical performance models.

Why this suits SRE responsibilities:

• Provides programmable control for automation and CI/CD workflows, rich telemetry for SLO monitoring, and clear operational runbooks supporting safe changes and rapid incident response.