Resilience and Persistence Questions

Situation / Goal
I treat handovers as the single source of truth so the next on-call can resume work quickly and recover if the incident recurs.

Handover contents (must-have)

• Incident summary: short description, start/end timestamps, affected sites/services/VLANs.
• Impact & scope: devices, interfaces, subnets, customer-facing services.
• Root cause (current understanding): e.g., misconfigured BGP policy, failing optical module, MTU mismatch.
• Actions taken: ordered timeline of commands run, config changes, ACLs applied, and exact CLI snippets or config diffs.
• Temporary workarounds and who approved them.
• Open tasks: immediate next steps, tickets, change windows required to complete fix.
• Evidence & diagnostics: key logs, traceroutes, show outputs, packet captures (links or file names).
• Runbook references: link to playbook and version used.
• Monitoring & thresholds: dashboards to watch, alert IDs, proposed alert tuning.
• Contacts & escalation: vendor, datacenter, senior on-call, expected SLA windows.
• Post-incident items: RCA owner, timeline for permanent remediation.

Format & delivery
I send a concise written handover in chat + update the incident ticket and attach runbook snippets; I also do a 5–10 minute verbal pass to highlight risks and answer questions.

Immediate (minutes–hours) — stabilize traffic

• Isolate and stop flapping source: identify peer(s) via BGP logs, route-refresh timestamps, and MRT dumps.
• Temporarily damp unstable prefixes with conservative BGP route-flap-damping (low penalty / short half-life) or apply max-prefix/peer prefix-limits to noisy peers.
• Preferentially pin critical prefixes: add static/aggregate routes or local RIB/operation policy to prefer stable next-hops (longer local-preference) so traffic avoids flapping paths.
• Contact upstream/peer NOC(s); coordinate graceful shutdown of problematic session if needed.

Mid-term (days–weeks) — policy & operational changes

• Tighten import/export filters: strict prefix-lists, ROA/RPKI validation, AS-PATH filtering to reject accidental announcements.
• Implement BGP PIC/fast-reroute and tuned timers (keepalive/hold) per peer class; avoid overly aggressive timers for stability-sensitive peers.
• Deploy BGP communities to control propagation and passive suppression of re-advertisements for unstable prefixes.
• Improve monitoring/alerting: MRT collection, BGPstream, prefix-origin change alerts, and automated correlation to detect flapping before impact.

Long-term (weeks–months) — architecture & resilience

• Multi-homing diversity: add geographically and topologically diverse peers and IXPs to reduce single-peer sensitivity.
• Route reflectors + Anycast for critical services to absorb churn; employ ECMP and traffic engineering (SR/MPLS) to spread load.
• Automated policy verification (config linter) and pre-deploy simulation of route changes.
• Invest in RPKI full-validation and origin enforcement across edge.

Trade-offs

• Faster convergence (aggressive timers, withdraws) can cause transient blackholes if peers oscillate; conservative timers improve reachability but increase failover time.
• Flap damping reduces control-plane churn but can blackhole legitimate prefixes if penalties are too strict.
• Anycast and ECMP improve reachability but add complexity in stateful services and debugging.

I would start with containment (pin critical prefixes), communicate with peers, then harden policies and expand topology diversity to reduce future sensitivity.

Overview / Goals
Run a blameless postmortem to learn, fix root causes, restore services faster, and improve resilience — not to assign blame.

Sections I include

• Summary & timeline — short incident summary, detection time, mitigation, recovery time
• Impact metrics — affected services, customers, SLO/SLA breaches, traffic dropped, packet loss %
• Timeline of events — precise timestamps, alerts, operator actions, config changes (who/what)
• Root cause analysis — causal chain (e.g., BGP flap due to misapplied ACL during maintenance → route withdrawal)
• Contributing factors — monitoring gaps, runbook ambiguity, automation failures
• Corrective actions — immediate mitigations, short-term fixes, long-term prevention
• Lessons learned & runbook updates
• Action items with owners, ETA, priority, verification steps

Classify & prioritize action items

• Severity (S1–S3) by customer impact and SLO breach
• Effort (low/med/high) and risk of rollout
• Prioritization matrix: high severity + low effort = P0; high severity + high effort = P1 with milestones; low severity + low effort = P2
• Example: rollback automation fix (P0, 2h), add BGP route validation (P1, 2 weeks), improve alert thresholds (P0, 4h)

Ensure follow-through, visibility, measurement

• Create tracked tickets in project tracker with owner, due date, test criteria, and link to postmortem
• Weekly review in ops meeting until all P0/P1 closed; monthly dashboard for long-term items
• Add verification steps: unit tests, staged rollout, canary, and post-deploy monitoring dashboards (packet loss, BGP convergence time, mean time to detect/repair)
• Update runbooks and run a tabletop drill within 30 days to validate changes
• Share postmortem and progress publicly to stakeholders with status and metrics against original SLOs

This approach keeps the process blameless, actionable, measurable, and network-focused.

Situation: On-call as a senior network engineer for a cloud services platform, we received pager alerts: multiple availability zones reporting packet loss and customer VPN disconnects during a major marketing launch.

Task: Rapidly identify root cause, restore connectivity for highest-impact customers, and reprioritize planned maintenance tasks to limit downtime and protect SLAs.

Action: I triaged using monitoring (NetFlow, SNMP, cloud VPC flow logs) and confirmed a misconfigured BGP route advertisement from a partner causing flapping. Prioritization criteria I used: customer SLA impact, number of affected customers, potential for escalation (security/data loss), and rollback safety. Steps:

• Halted nonessential maintenance and diverted two engineers to mitigation.
• Implemented emergency BGP dampening and route filters to stop propagation.
• Restored VPN tunnels by pushing corrected route-maps and re-establishing peer sessions.
• Kept stakeholders informed with concise updates every 15 minutes via Slack + postmortem scheduled.

Result: Within 28 minutes we reduced packet loss to baseline and restored 95% of VPN sessions; no SLA breaches. Post-incident we documented the fix, added automated route validation, and adjusted runbooks.

Situation & Task
During a scheduled firmware upgrade of core routers in a regional POP, monitoring 10 minutes in showed rising CRC errors and 15% packet loss on the primary aggregation link — an unplanned failure that risked service disruption if we continued the planned change.

Assessment & Risk Evaluation

• Immediate risks: network outage for 1k users, BGP flap propagation, impact to dependent services.
• Options: abort and roll back now (fast but leaves partial config state), pivot to maintenance-plane failover to standby aggregation (safer but requires route re-convergence), or pause and troubleshoot (time-consuming).
• Chosen: pivot to standby aggregation and hold firmware steps to avoid leaving devices half-updated.

Actions & Communication

• Notified stakeholders (NOC, app owners, change manager) within 3 minutes: described impact, chosen pivot, estimated 12–15 min for cutover and testing, and rollback criteria.
• Executed: updated ACL and static routes on edge devices to prefer standby path, activated pre-tested BFD timers to detect convergence quickly, monitored BGP sessions and latency.
• Kept a dedicated incident Slack channel and 1:1 phone with change manager for real-time updates.

Result & Follow-up

• Traffic shifted with <30s of packet-loss spike; services remained available. After stabilization, we completed firmware on a staged subset and rescheduled remaining upgrades. Post-incident: RCA identified a hardware interface degradation; replaced module and updated maintenance runbook to include immediate standby pivot procedure and automated telemetry alerts.