Direct answer
Stabilize a flaky CI pipeline by deciding, dependency by dependency, whether to mock it, virtualize it, or keep a small number of real calls, rather than applying one blanket policy; the criteria are how flaky the dependency has actually been, how safety-critical it is, and how expensive it is to call for real.
Structured elaboration
A practical migration plan:
- Classify each external dependency by observed failure rate over the last few weeks of CI runs and by how central its real behavior is to what you're trying to validate.
- Mock-first for chronically flaky, low-risk dependencies: anything with a high enough failure rate that it's already causing "just re-run it" behavior, and where the business logic under test doesn't depend on the dependency's exact real-world quirks.
- Keep or add a small number of real integration tests for high-risk dependencies: run them less frequently (nightly, or gated on a separate slower pipeline) rather than on every commit, so their occasional real flakiness doesn't block every PR.
- Migrate incrementally, dependency by dependency, verifying after each migration that the pipeline's overall flakiness rate actually drops and that no regression slipped through because a mock is now hiding real behavior.
- Watch for the mix drifting the wrong way: track what fraction of tests exercise a real dependency over time; if it trends toward zero, the suite is losing its ability to catch real integration bugs, and if a chronically-mocked dependency's real API changes, nothing will tell you until it breaks in production.
Worked example
A pipeline has three external dependencies: a currency-conversion API (occasional slowness, low business risk, used in many tests), a fraud-check API (occasional slowness, high business risk), and an email-sending service (occasional slowness, low risk, used in far fewer tests). The right migration: mock currency conversion everywhere except in one or two integration tests that revalidate the response shape weekly; keep the fraud-check API real in a small, nightly-run suite specifically because its exact decision logic is what the business needs the tests to protect; fully mock the email service, since almost no test needs to verify that email actually sends, only that the code attempted to send it.
Trade-offs and pitfalls
The plan fails if it's applied as an all-or-nothing switch: flipping every test to mocks in one pass removes the flakiness immediately but can silently remove real-integration coverage for months before anyone notices a drift. Track the migration with a simple metric, like "count of tests exercising each real dependency," reviewed periodically, so the team can see the mix drifting and course-correct before it becomes invisible.
