Creativity and Unconventional Problem-Solving Questions

Situation: A core API that calculates user invoices started failing under peak load after a third-party pricing service degraded. The permanent fix (rewriting retry logic and batching) would take several days.

Task: Keep the billing service available with accurate-enough invoices while we implemented the permanent fix.

Action:

• Implemented a temporary fallback in the API: if the third-party call timed out or returned errors, the service used a cached prices snapshot (updated every 30m) and flagged invoices as “estimated.”
• Rolled the change behind a feature flag and enabled it for 5% of traffic first. Added a circuit breaker to stop calling the flaky service after 20% error rate.
• Built short-lived monitors and dashboards: error rates, estimated-invoice rate, divergence between cached vs live prices (synthetic checks every minute), latency, and customer-facing KPIs.
• Safety evaluation: ran canary tests against production-like test data, reviewed cost/accuracy trade-offs with product, and set strict SLAs for the fallback (max allowed estimation error).
• Risk mitigation: limited scope (5% → 25% → 100%), automatic rollback on threshold breaches, granular logs and alerting, and a remediation playbook for customer support (clear messaging and refund policy).
• Transition: while SREs implemented the permanent retry/batching fix, we maintained the gradual rollout. After the permanent fix passed end-to-end tests and 24h of stable canary metrics, we switched traffic back, removed the feature flag, and deleted cached fallback code in a tracked cleanup PR. Completed a postmortem documenting root cause, lessons, and updated runbooks.

Result: Availability stayed >99.9% during the incident window, customer impact was minimized, and the permanent fix was deployed without rollback.