Situation: During a severe outage we must choose between a quick rollback that removes audit logs (hurting traceability) or preserving auditability but degrading latency for critical users.
Approach / Decision framework:
- Rapid fact-gathering (30–60s): current impact (users affected, revenue, safety), legal/regulatory constraints (retention, breach notification), and reversibility of each action. Use dashboards, on-call docs, and incident playbooks.
- Apply prioritized principles: safety & legal compliance first, then customer trust, then availability/SLOs. If removing audit logs would violate law, contract, or lead to unresolvable forensic gaps, it’s unacceptable.
- Quantify trade-offs quickly: estimate percentage of critical traffic impacted, expected latency increases vs. time to rollback, and expected duration of degraded state.
Who to involve (real-time):
- Incident Commander / SRE lead (decision authority)
- Service owners / engineering lead (technical feasibility)
- Legal / Compliance and Security (assess regulatory/forensic risk)
- Product / Customer Success (customer impact and RTO expectations)
- Exec on-call if business-critical or high public risk
Actionable options to consider in real time:
- Prefer solutions that preserve auditability: partial rollback (non-critical regions), route non-critical traffic to degraded paths, throttle background jobs, or enable read-only degraded mode.
- If rollback is the only quick path and is legal: implement mitigations to preserve traceability (temporarily mirror requests to an append-only store, enable packet captures, or create a tamper-evident log).
- If legal prohibits removing logs: accept temporary latency, optimize routes for critical users only, and communicate SLAs and mitigations to stakeholders.
Communication:
- Announce decision, rationale, and expected ETA to internal stakeholders and customer-facing teams within 5–10 minutes; prepare public status updates if user-facing.
Result & learnings:
- After stabilization, run a blameless postmortem to document decision rationale, update playbooks to include pre-approved mitigations, and add automated controls (feature-flag kill-switch that preserves audit trail) so future choices are faster and safer.
