Mocking, Stubbing, and Test Isolation Questions

Direct answer

Stabilize a flaky CI pipeline by deciding, dependency by dependency, whether to mock it, virtualize it, or keep a small number of real calls, rather than applying one blanket policy; the criteria are how flaky the dependency has actually been, how safety-critical it is, and how expensive it is to call for real.

Structured elaboration

A practical migration plan:

1. Classify each external dependency by observed failure rate over the last few weeks of CI runs and by how central its real behavior is to what you're trying to validate.
2. Mock-first for chronically flaky, low-risk dependencies: anything with a high enough failure rate that it's already causing "just re-run it" behavior, and where the business logic under test doesn't depend on the dependency's exact real-world quirks.
3. Keep or add a small number of real integration tests for high-risk dependencies: run them less frequently (nightly, or gated on a separate slower pipeline) rather than on every commit, so their occasional real flakiness doesn't block every PR.
4. Migrate incrementally, dependency by dependency, verifying after each migration that the pipeline's overall flakiness rate actually drops and that no regression slipped through because a mock is now hiding real behavior.
5. Watch for the mix drifting the wrong way: track what fraction of tests exercise a real dependency over time; if it trends toward zero, the suite is losing its ability to catch real integration bugs, and if a chronically-mocked dependency's real API changes, nothing will tell you until it breaks in production.

Worked example

A pipeline has three external dependencies: a currency-conversion API (occasional slowness, low business risk, used in many tests), a fraud-check API (occasional slowness, high business risk), and an email-sending service (occasional slowness, low risk, used in far fewer tests). The right migration: mock currency conversion everywhere except in one or two integration tests that revalidate the response shape weekly; keep the fraud-check API real in a small, nightly-run suite specifically because its exact decision logic is what the business needs the tests to protect; fully mock the email service, since almost no test needs to verify that email actually sends, only that the code attempted to send it.

Trade-offs and pitfalls

The plan fails if it's applied as an all-or-nothing switch: flipping every test to mocks in one pass removes the flakiness immediately but can silently remove real-integration coverage for months before anyone notices a drift. Track the migration with a simple metric, like "count of tests exercising each real dependency," reviewed periodically, so the team can see the mix drifting and course-correct before it becomes invisible.

Direct answer

For large-scale performance and load testing, mocking is acceptable only for components that are genuinely not part of what you're trying to measure (a notification service whose latency doesn't affect the checkout path under load, for example); for anything on the critical path whose real-world behavior under load is exactly the unknown you're testing, a mocked stand-in can produce a capacity number that doesn't hold up in production.

Structured elaboration

The core risk: a mock typically responds instantly and with unlimited concurrency, so a load test that mocks a real bottleneck (a database, a downstream service with its own capacity limits) will measure the capacity of everything EXCEPT that bottleneck, and can report a headline throughput number far higher than the system could actually sustain in production once the real dependency's own limits kick in.

A hybrid plan:

1. Identify the dependencies actually on the critical path for the metric you're measuring (throughput, p99 latency, error rate under load) and keep those real, at least in a representative capacity (a real database sized similarly to production, a real downstream service in a load-test environment with its own realistic scaling).
2. Mock dependencies genuinely off the critical path, or ones you've deliberately decided to measure independently in a separate, targeted load test.
3. Validate the mocked dependencies' assumed latency and error-rate profile against real production or staging data, so even a mocked component's behavior in the load test reflects a realistic response-time distribution rather than an unrealistic instant response.
4. Explicitly state, in the load test's own documentation or output, which dependencies were real and which were mocked, so anyone reading the resulting capacity numbers knows exactly what was and wasn't measured, and doesn't accidentally treat a partial measurement as a full-system capacity guarantee.

Worked example

Load-testing a checkout service, keep the real payment gateway's sandboxed load-test environment in the loop (since its actual latency and rate limits under load are part of what determines real checkout throughput), but mock the email-receipt service entirely (since its latency genuinely doesn't gate the checkout response, it's fired asynchronously after the user already sees success). Reporting "checkout throughput: 5,000 requests/second" without disclosing the email service was mocked is fine, since it's genuinely off the critical path; reporting the same number while the PAYMENT gateway was mocked would be actively misleading, since the real bottleneck was never exercised.

Trade-offs and pitfalls

Running the full system with every real dependency at true production scale is often prohibitively expensive or logistically impossible (rate limits on a payment provider, cost of provisioning production-scale infrastructure just for a test), which is exactly why the hybrid plan matters: it's not "mock everything for cost" or "use nothing but real dependencies for accuracy", it's a deliberate, documented choice per dependency based on whether it gates the metric being measured.