Resilience and Persistence Questions

Approach (one‑line)
As a network engineer I design alerting that shifts from noisy, static thresholds to context-aware signals during major incidents so the team sees only high‑value, actionable alerts.

Dynamic thresholds

• Use baseline/seasonal models (Prometheus with moving percentile rules, or ML on NetFlow) to set adaptive thresholds for latency, packet loss, and interface errors.
• Example: alert when 95th‑percentile latency > baseline + 3σ over 5m, not a fixed 100ms.

Incident‑mode alert silencing & escalation

• Implement an “incident mode” flag (in PagerDuty/ServiceNow) triggered by PagerDuty manual or automated incident detection (e.g., BGP route flap storm).
• On incident mode: automatically suppress lower‑severity alerts (info/warning) and group similar signals (alert deduplication by device/prefix). Keep only critical topology or traffic‑blackhole alerts and escalation to on‑call.

Layered notifications

• Tier alerts: P0 (network outage, control‑plane down) → immediate page/phone; P1 (major degradation) → page + channel; P2 (degraded but localized) → channel/email.
• Use routing‑aware enrichment: include affected prefixes, ASNs, circuit IDs in notification.

Post‑incident re‑tuning

• After action review: compare suppressed alerts vs. what helped. Update thresholds, add suppression rules for known benign storms (e.g., scheduled backup floods), and create proactive synthetic tests. Automate changelog of tuning with runbook updates.

Implementation notes

• Tools: Prometheus + Alertmanager, Grafana, NetFlow/IPFIX collector, SNMP traps, syslog, PagerDuty.
• Automate suppression and reversion via IaC (Terraform) and playbooks (Ansible).
• Maintain observability of suppression state and metrics to avoid blindspots.

This reduces noise, focuses responders on high‑impact signals, and closes the feedback loop so alerts stay actionable.

Situation & Task
At my last role we had intermittent application outages caused by flapping BGP sessions between two colo routers. Full mitigation required a maintenance window; I proposed a small experiment: implement a targeted health-check + automated partial failover to reduce customer impact while we diagnosed root cause.

Design & Action

• Added an active TCP health check probing the application VIP from two vantage points every 5s (vs existing 60s).
• Implemented a short Ansible playbook + automation in our controller to temporarily shift traffic to the secondary path if 3 consecutive probes failed (15s threshold).
• Rolled out to a single service cluster for 72 hours (canary).

Metrics & Measurement

• Measured: mean time to detect (MTTD), mean time to restore (MTTR), percent of user requests failing, and false-positive failovers.
• Collected probe logs, routing state, and application error rates.

Results & Iteration

• MTTD dropped from ~60s to ~15s; MTTR reduced 40%. False positives were 1 in 300k probes; I raised the threshold to 4 failures and added AS-path stability check to eliminate that.
• After two iterations deployed cluster-wide with runbook updates and on-call training. The change improved perceived availability and reduced incident pages during BGP instability.

Situation & goals
Design a resilient multi-region network for a global SaaS with 99.99% availability and low latency across three continents (NA, EU, APAC). Focus: redundancy, deterministic failover, optimal routing, data consistency choices, validation within budget and compliance.

High-level architecture

• Active-active deployment across three cloud regions with regional VPCs/VNets.
• Global tiered edge: Cloud CDN + Anycasted WAF at edge POPs; regional ingress via Global Load Balancer (GSLB) with health checks.
• Private inter-region backbone using cloud provider backbone (VPC peering/Transit Gateway/Cloud Interconnect) plus optional NVMe-over-Fabric for high-throughput links.

Redundancy & failover

• Dual AZs per region with active-active application and stateless services; stateful clusters (DB, cache) use regional leader + async replication.
• Multi-path routing: BGP with multiple upstreams in each region; ECMP for load sharing and fast reroute.
• Automated failover: health probes -> GSLB shifts traffic; local regional LB falls back to secondary AZ; for full-region failure, traffic diverted to nearest region with warmed-up capacity (traffic shift throttled by rate-limiting to avoid overload).

Routing strategies

• Use Anycast for edge endpoints to minimize latency and improve DDoS absorption.
• GSLB with latency-based and geo-aware policies + health-based weights.
• BGP communities to steer traffic during incidents; local preference adjustments for traffic engineering.
• Use TCP/QUIC optimizations and path MTU discovery; implement WAN optimization between regions if cross-region latency critical.

Data replication & consistency

• Categorize data: hot (user sessions, cache) -> global cache with CRDTs or active-active DB (when strong eventual consistency acceptable); transactional data -> primary region with synchronous local replication (AZ) and async cross-region replication for RTO/RPO trade-offs.
• Use change-data-capture (CDC) pipelines for cross-region eventual consistency and to rebuild replicas.
• For strict compliance (data residency), route writes and store data in designated regions; use encryption-at-rest and KMIP/HSM per region.

Validation & testing

• Chaos engineering: simulate AZ and whole-region failovers, BGP path flaps, and link saturation in staging and canary networks.
• Synthetic monitoring from global probes measuring latency, packet loss, and RTO; automated runbooks triggered by health anomalies.
• Capacity tests: ramp traffic to validate warm-up and autoscale thresholds; test DB failover and replication lag under load.

Budget & compliance trade-offs

• Use cloud managed services for backbone and GSLB to reduce operational cost; reserve capacity and use spot for non-critical workloads.
• For cost-sensitive regions, prefer async replication and CDN caching to reduce cross-region egress.
• Ensure compliance via regional isolation, per-region KMS, audit logs, and signed network configs; document data flows for audits.

Metrics & success criteria

• 99.99% availability SLA measured per region and globally, <100 ms median latency target per continent, RTO <5 min for AZ failover, RPO based on data tiering (near-zero for core transactional data, <1 min for critical replicas).

This design balances low latency, predictable failover, and compliance while controlling cost through managed services and tiered data strategies.

Overview
I’d implement a predictable, fair on-call rota that protects uptime and engineer wellbeing while keeping escalation clear and SLO-focused.

Shift pattern

• 12 engineers across timezones → 3 primary weekly shifts (UTC-aligned): Day (08–20), Night (20–08), and Overlap (12–00) with 4 people per shift; rotate every 4 weeks so no one permanently nights.
• Primary on-call (1 engineer per shift) + secondary backup from same shift; weekend coverage by rotating pairs so each engineer gets ~1 weekend/month.
• Max consecutive on-call weeks: 4, followed by 4 weeks off-call.

Escalation path

• Pager -> primary on-call (15 min) -> secondary (15 min) -> on-call lead/SME (30 min) -> engineering manager -> exec incident commander.
• Runbooks, runbook ownership and an on-call playbook in the runbook repository; automated triage (monitoring runbooks) to reduce noisy alerts.

Learning & allowances

• Guarantee 8 hours/week “learning time” for training, certifications, or project work while on-call; 16 hours/week when off-call.
• Quarterly “shadow” on-call for new hires: 2 weeks observing then paired with senior on-call.

Post-incident & mental health

• Mandatory post-incident 1:1 and blameless postmortem within 72 hours; action items assigned and tracked to closure.
• Recovery buffer: after Severity 1 incidents, affected on-call gets a recovery day (or two half-days) and reduced on-call load following week.
• Mental health: access to EAP, manager check-ins after major incidents, and opt-in peer support rotation.
• Metrics: track on-call load, SLOs, time-to-ack, and developer satisfaction quarterly; adjust rota based on data.

Situation: On-call as a senior network engineer for a cloud services platform, we received pager alerts: multiple availability zones reporting packet loss and customer VPN disconnects during a major marketing launch.

Task: Rapidly identify root cause, restore connectivity for highest-impact customers, and reprioritize planned maintenance tasks to limit downtime and protect SLAs.

Action: I triaged using monitoring (NetFlow, SNMP, cloud VPC flow logs) and confirmed a misconfigured BGP route advertisement from a partner causing flapping. Prioritization criteria I used: customer SLA impact, number of affected customers, potential for escalation (security/data loss), and rollback safety. Steps:

• Halted nonessential maintenance and diverted two engineers to mitigation.
• Implemented emergency BGP dampening and route filters to stop propagation.
• Restored VPN tunnels by pushing corrected route-maps and re-establishing peer sessions.
• Kept stakeholders informed with concise updates every 15 minutes via Slack + postmortem scheduled.

Result: Within 28 minutes we reduced packet loss to baseline and restored 95% of VPN sessions; no SLA breaches. Post-incident we documented the fix, added automated route validation, and adjusted runbooks.