Direct answer
A stopgap that is not tracked becomes permanent. So the moment service is restored I turn it into a recorded, owned, time-boxed risk: written down in the incident timeline, converted to a ticket with a named owner and a due date, communicated to the people who own the risk, and closed only when the stopgap is removed and someone independent confirms it.
Worked scenario (illustrative)
During an outage, to let a failover database replica connect, an engineer changed a firewall rule (the network filter that decides who may connect) so the database port (5432, the door clients use to talk to it) accepted connections from every address on the internet (written 0.0.0.0/0) instead of only the replica's own address (10.0.4.17). Service returned in 10 minutes. The gap: the database is now reachable from outside, by anyone who can guess or find it, instead of by one server.
Step by step, in order
- Document during the incident. In the incident timeline: exact change, who made it, time, and the gap it created. A one-line note beats memory later.
- Create a temporary-mitigation record the same day. Fields: what changed and where, why, what risk it adds (who could reach what), compensating controls now in place (a compensating control is a temporary safeguard that reduces the risk while the real fix is pending, such as extra logging or an alert on connections from unexpected addresses), owner, expiry date, and the permanent-fix ticket link. Filled in for this scenario:
text
Record: TM-2026-041
Change: Firewall rule db-failover-allow, port 5432, source 0.0.0.0/0 (was 10.0.4.17/32)
Why: Failover replica could not connect during the 2026-09-28 outage
Risk added: Anyone on the internet can attempt to log in to the production database
Compensating: Password-only logins disabled for non-replica users; alert on any 5432
connection from outside 10.0.0.0/8; connection log reviewed daily
Owner: Payments platform team lead
Expires: 2026-10-05
Permanent fix: SEC-2210 (restrict source to 10.0.4.17/32), due 2026-10-05
- Turn it into tracked work. One ticket for the permanent fix (restrict to the specific hosts), owned by the service team, with a due date (for example 5 business days, since this is a live exposure). Also add it as an action item in the postmortem (the blameless review after an incident: it asks what failed in the system, not who to blame).
- Communicate the risk. A short message to the service owner, the security team and the on-call manager: what is exposed, what compensates, when it ends, what triggers escalation. Plain wording, no blame. For example: "Since 14:10 today the production database port accepts connections from any internet address, a side effect of restoring failover. We have restricted logins and are alerting on outside connections. Fix is ticketed (SEC-2210), due Oct 5. If the alert fires, security will block the port immediately and we will take the failover replica offline until it is fixed."
- Make the expiry enforceable. Tag the change with an expiry date, and run an alert that fires when an expired tag is still present. A calendar reminder alone is not enough, because it depends on someone remembering. Concretely: put a label
expires=2026-10-05 on the firewall rule, and run a daily job that lists every rule whose expires date is earlier than today and pages the owner. On 2026-10-06 the broad rule shows up in that list if it still exists, without anyone having to recall it.
- Replace and verify. Apply the proper rule, remove the broad one, test the failover still works, and have someone other than the author confirm from outside that the port is no longer reachable. Attach the evidence and close both the record and the ticket.
What would change my call
If the permanent fix needs weeks (for example a network redesign), I stop calling it temporary: it becomes a formal, time-limited risk acceptance (a written, signed decision by the person who owns the risk that the organisation will live with it for a stated period), with stronger compensating controls and a renewal date.
Pitfalls
- Closing the incident and the ticket together, so nothing carries the exposure forward.
- Blame-heavy wording that discourages people from disclosing stopgaps next time.
- "Fixed" declared without confirming the old rule is gone.