Distributed Systems and Microservices Testing Questions

Direct answer
Unit-test the retry/backoff decision logic in isolation (given a response code and attempt count, does it retry, and after how long), integration-test it against a virtualized CDN configured to return 5xx/timeouts, stale content, and truncated bodies on a controlled schedule to prove the real HTTP client actually honors each policy, add an end-to-end test proving the actual page/feature degrades gracefully (not just the raw client) when the CDN is persistently unavailable, and keep the whole suite fast by using a controllable clock instead of real sleeps for the backoff delays.

Structured elaboration

1. Unit level: the retry decision itself. Test the pure decision logic (given a status code, is this retryable; given an attempt number, what is the computed backoff delay) without any real network or real waiting involved. This is where you verify the actual backoff formula (exponential, with jitter) is implemented correctly, independent of anything about the CDN.
2. Integration level: the real client against a controlled CDN. Point the real HTTP client at a virtualized CDN endpoint configured to return a 5xx on the first N calls and succeed after that (or to hang past the client's timeout), and assert the real client's observed behavior (number of actual retries, whether it eventually succeeds or gives up) matches the intended policy. This catches the class of bug where the retry LOGIC is correct in isolation but is never actually wired to the real HTTP client, or is wired to the wrong exception type.
3. Cache staleness, as its own scenario, at the integration level. Configure the virtualized CDN to serve intentionally stale content (an old ETag) with a normal 200 status, and assert the client correctly detects and handles staleness according to whatever cache-validation policy is intended, rather than assuming any response with a 200 status is automatically fresh. This needs its own explicit test because a stale response is NOT distinguishable from a fresh one by status code alone.
4. Partial content, as its own scenario. Simulate a truncated or partial response (fewer bytes than the declared Content-Length, or a connection reset mid-transfer) and assert the client detects this as a failure worth retrying, not a silently-accepted partial success.
5. End-to-end level: the feature, not just the client. Drive the actual page-rendering or asset-loading feature that USES the CDN client (not the client in isolation) through a scenario where the CDN is persistently failing beyond the retry budget, and assert the feature itself degrades gracefully (serves a fallback asset, marks the response degraded) rather than the retry/backoff logic being correct in isolation while the caller has no idea what to do when it's finally exhausted.
6. Keeping the suite fast. Use a fake/controllable clock for the backoff delays (inject a clock dependency the test can fast-forward) instead of real sleep() calls; a retry test suite that actually waits out real exponential backoff delays becomes slow enough that people stop running it locally, which defeats its purpose.

Worked example

class FakeClock:
    def __init__(self):
        self.now = 0.0
        self.sleeps = []
    def sleep(self, seconds):
        self.sleeps.append(seconds)
        self.now += seconds  # advance instantly, no real waiting

def test_retry_uses_exponential_backoff_and_eventually_succeeds(virtualized_cdn):
    virtualized_cdn.configure_sequence([503, 503, 200])  # fails twice, then succeeds
    clock = FakeClock()
    client = CdnClient(base_url=virtualized_cdn.url, clock=clock, max_retries=3, base_delay_s=0.1)

    response = client.fetch("/asset.js")

    assert response.status == 200
    assert virtualized_cdn.call_count == 3
    assert clock.sleeps == [0.1, 0.2], f"expected exponential backoff [0.1, 0.2], got {clock.sleeps}"

def test_partial_content_is_treated_as_retryable_failure(virtualized_cdn):
    virtualized_cdn.configure_truncated_response(declared_length=1000, actual_bytes=400)
    client = CdnClient(base_url=virtualized_cdn.url, clock=FakeClock(), max_retries=1, base_delay_s=0.01)
    try:
        client.fetch("/asset.js")
        assert False, "a truncated response should not be silently accepted as success"
    except IncompleteContentError:
        pass

def test_cache_staleness_is_detected_not_assumed_fresh(virtualized_cdn):
    virtualized_cdn.configure_stale_response(served_etag="etag-stale-v1")
    client = CdnClient(base_url=virtualized_cdn.url, clock=FakeClock(), known_fresh_etag="etag-fresh-v2")
    resp, is_stale = client.fetch_and_validate_freshness("/asset.js")
    assert resp.status == 200, "a stale response is still a 200, which is exactly why staleness needs its own check"
    assert is_stale is True, "a 200 with an old ETag must be detected as stale, not assumed fresh"

def test_e2e_page_render_falls_back_when_cdn_exhausts_retries(virtualized_cdn, page_renderer):
    virtualized_cdn.configure_sequence([503, 503, 503, 503])  # more failures than max_retries
    result = page_renderer.render("/asset.js")
    assert result.degraded is True
    assert result.asset == FALLBACK_ASSET_BYTES, "the feature, not just the client, must fall back to a usable asset"

Trade-offs and pitfalls

• Testing the backoff formula in isolation and never against the real client leaves a real integration gap; always pair the unit-level formula test with at least one integration test proving the real client actually applies it.
• Cache-staleness bugs are frequently missed because most tests only configure a CDN to return either "clean success" or "clean failure," never "technically a 200 but stale"; make staleness a first-class scenario, not an afterthought.
• Stopping at the integration level (a correct client) without an end-to-end test can hide a bug in the CALLING feature: a client that correctly falls back internally is no guarantee the page-rendering code that calls it actually surfaces or uses that fallback correctly.
• A fake clock only speeds up the test suite if EVERY delay in the code path goes through it; a stray real time.sleep() left in the retry implementation (or in a library the client depends on) will silently reintroduce slowness the fake clock can't catch.

Direct answer
Keep the mocks for each partner (payment, warehouse) behind a shared virtualization layer configured from that partner's REAL, current API contract wherever one is published (an OpenAPI spec, a Pact contract), add a scheduled or CI-gated check that re-validates the mock's configured responses against that contract so drift is caught automatically rather than discovered when a real integration breaks, and design the CI test scenarios around the specific realistic failure modes (timeout, partial success, delayed webhook) as first-class, separately-asserted cases rather than only a single happy path.

Structured elaboration

1. Grounding mocks in the partner's real contract. Rather than hand-writing mock responses from memory (which drift from reality the moment the partner changes anything), derive the mock's response shapes from the partner's published contract (an OpenAPI schema, a Pact file if the partner supports consumer-driven contracts, or at minimum a checked-in sample of real captured responses reviewed periodically). This gives the mock a concrete, checkable source of truth to compare against over time.
2. Detecting drift automatically. Add a scheduled job (or a gated pre-merge check when the partner integration code changes) that validates the mock's current configured responses against the partner's current published contract or a fresh captured real response (obtained safely, for example via the partner's sandbox environment), and fails loudly if they've diverged, rather than relying on someone noticing a real production incident to reveal the drift.
3. Realistic failure modes as explicit, separate scenarios. For each partner, define and separately test: a timeout (the mock simulates the partner never responding within the caller's timeout window), a partial success (the mock simulates the order-processing pipeline getting a successful payment but the warehouse partner reporting only partial fulfillment), and a delayed webhook (the mock schedules a confirmation callback well after the initial synchronous response, mirroring how these partners actually behave in production).
4. Keeping mocks synchronized as a process, not just a mechanism. Assign clear ownership (whichever team owns the integration with a given partner also owns keeping that partner's mock definition current), and treat a mock update as part of the normal change process whenever the partner's integration code changes, not a separate, easily-forgotten housekeeping task.

Worked example
A drift-check sketch that validates a mock's configured response shape against a captured real contract, run on a schedule:

import jsonschema

def test_payment_gateway_mock_matches_current_contract(mock_config, partner_contract_schema):
    for route, behavior in mock_config["routes"].items():
        jsonschema.validate(instance=behavior["body"], schema=partner_contract_schema[route])

def test_order_pipeline_handles_partial_warehouse_fulfillment():
    payment_mock.configure_route("POST /v1/charges", status=201, body={"id": "ch_1", "status": "succeeded"})
    warehouse_mock.configure_route("POST /v1/fulfill", status=207, body={
        "order_id": "o1", "fulfilled_items": ["sku-1"], "backordered_items": ["sku-2"],
    })

    result = order_pipeline.process(sample_order_with_two_items())

    assert result.payment_status == "charged"
    assert result.fulfillment_status == "partial"
    assert "sku-2" in result.backordered_items, "a partial warehouse fulfillment must surface which items are backordered, not just a generic partial flag"

The first test is scheduled to run against a periodically-refreshed partner_contract_schema (pulled from the partner's published OpenAPI spec or a Pact broker), independent of the functional tests, specifically to catch drift even when nobody is actively changing the order-pipeline code.

Trade-offs and pitfalls

• If a partner has no published, checkable contract at all, the drift-detection mechanism has to fall back to periodically and manually comparing a captured real sandbox response against the mock, which is weaker and more likely to be neglected; push for at least a lightweight recorded-sample comparison even without a full formal contract.
• Testing only the happy path for each partner (successful payment, full fulfillment) misses exactly the failure modes that cause real production incidents; timeouts, partial success, and delayed webhooks should each have their own named test, not be treated as edge cases to add "later."
• Ownership diffusion (nobody clearly owns keeping a given partner's mock current) is the most common way this class of infrastructure quietly rots; make updating the mock an explicit, reviewed part of the same pull request that changes the partner integration code, not a separate backlog item.

Direct answer
Build the harness on tooling that injects faults at the network layer you actually need (packet-level shaping for latency and packet loss, service-mesh or proxy-level fault injection for HTTP-level errors and resets), justify the choice by which layer the fault needs to be believable at, measure that the fault genuinely landed by watching the affected connections directly (not just trusting the injection tool's own exit code), and assert correctness properties (does the system stay available in a degraded mode, does it preserve whatever consistency guarantee it claims) while keeping the blast radius to a single, clearly-scoped set of services.

Structured elaboration
Three design decisions carry this harness:

1. Tooling layer. Low-level network-shaping tools operate below the application (they manipulate the actual packets: dropping, delaying, corrupting), so they are the most realistic way to simulate a real network partition or lossy link, but they require access to the host/container network namespace and are coarser to target precisely at one service pair. Proxy or service-mesh-level fault injection operates at the application/HTTP layer (it can inject a 503, add artificial latency to one specific route, or reset a specific connection), which is easier to scope precisely to one dependency and easier to run in CI without host-level privileges, at the cost of being a slightly less faithful simulation of a true network-layer failure. Pick network-layer tooling when you specifically need to validate low-level behavior (TCP timeout handling, connection-reset recovery); pick proxy/mesh-level injection when you are testing application-level resilience logic (retries, circuit breakers, fallbacks) and want fine-grained, easily-automatable control.
2. Confirming the fault landed. Do not trust that calling the injection tool means the fault is active; independently observe an effect that proves it (a measured round-trip time above the injected floor, an actual dropped-packet counter increment, a proxy's own fault-injection metric). A test that injects a fault, asserts on the system's behavior, but never confirms the fault was really active can pass for the wrong reason (the fault silently failed to apply, and the system behaved correctly simply because nothing bad happened).
3. Blast-radius control. Scope the fault to a clearly bounded target (one service pair, one route, one percentage of traffic) using labels/selectors the tooling supports, run it first against a non-production, isolated environment, and have an automatic kill-switch (a maximum experiment duration, an automatic revert) so a misconfigured or unexpectedly severe fault cannot spread beyond the intended scope.

Worked example
A small harness abstraction that could be backed by either a network-shaping tool or a mesh-level fault-injection API, showing the "confirm it landed" discipline concretely:

class FaultInjector:
    """Backed by either a network-namespace tool or a service-mesh fault-injection
    API; the interface is deliberately the same so tests don't care which."""
    def inject_latency(self, source, target, delay_ms, duration_s):
        raise NotImplementedError
    def measured_effect(self, source, target):
        """Returns the actually-observed extra latency, independent of what was requested."""
        raise NotImplementedError

def test_partition_between_order_service_and_inventory_service():
    injector = MeshFaultInjector(namespace="test")
    injector.inject_latency("order-service", "inventory-service", delay_ms=2000, duration_s=30)

    observed_delay = injector.measured_effect("order-service", "inventory-service")
    assert observed_delay >= 1800, (
        f"fault injection did not actually land: measured only {observed_delay}ms of the requested 2000ms"
    )

    response = order_client.place_order(sample_order())
    assert response.status == "ACCEPTED_DEGRADED", (
        "order service should fall back to async inventory confirmation under high dependency latency, "
        f"got {response.status!r}"
    )
    assert response.availability_impact == "none", "the caller should never see an outage from this fault alone"

Trade-offs and pitfalls

• Network-namespace-level tools usually require elevated privileges and host access that a shared CI runner may not grant; service-mesh or proxy-level injection is more CI-friendly but only as faithful as the mesh's own fault-injection fidelity (it typically cannot simulate a true partition below the HTTP layer, such as a TCP-level black hole).
• A fault that is scoped too broadly (an entire percentage of ALL traffic instead of one route) risks the experiment itself becoming the incident; always start with a scoped, short-duration, single-route experiment and expand deliberately.
• The single most common false-positive in this class of test is skipping the "confirm the fault actually landed" step; a test environment where the injection silently no-ops (a misconfigured selector, an unsupported combination of options) will still report a passing test, for entirely the wrong reason.

Direct answer
Test the fallback at all three levels with a different focus at each: a unit test asserting the fallback logic itself returns cached data plus a soft-warning flag when the downstream call throws, an integration test proving the actual downstream failure (via a virtualized service) triggers that exact code path rather than a different error path, and an end-to-end test confirming the fallback response reaches the user quickly and with the soft-warning visibly attached, not silently swallowed somewhere in the middle.

Structured elaboration

1. Unit level. Mock the downstream client to throw the specific exception type a real outage would produce, and assert the service's fallback branch returns the cached value (from whatever cache/local store backs it) along with an explicit degraded: true (or equivalent) flag. Also test the CACHE-MISS case: what happens when the downstream is unreachable AND there is no cached data yet, since a fallback that assumes cached data always exists is a latent bug.
2. Integration level. Replace the real user-profile service with a virtualized stand-in configured to be unreachable (connection refused, or a timeout), and assert the calling service's actual configured client (its real timeout settings, its real retry policy, its real exception handling) correctly routes into the fallback branch, not into an unhandled exception or an overly-broad catch that also swallows unrelated bugs.
3. End-to-end level. Drive a real request through the full stack with the same virtualized downstream failure, and assert the user-facing response actually contains the cached data and the soft-warning is genuinely visible to the caller (present in the response payload the client-facing layer returns), and measure that the fallback path completes quickly (bounded by the downstream's configured timeout plus a small overhead), since a slow fallback defeats the purpose of having one.
4. Speed as an explicit assertion, not an assumption. Add a latency assertion at the end-to-end level specifically: the fallback response must return within a tight bound (for example, close to the configured downstream timeout, not several multiples of it), catching a regression where a well-intentioned retry-before-fallback change quietly makes the "fast fallback" slow again.

Worked example

# Unit level
def test_fallback_returns_cached_data_with_degraded_flag():
    cache = FakeCache({"user-1": {"name": "Ada"}})
    client = RaisingClient(DownstreamUnavailableError)
    service = ProfileService(downstream_client=client, cache=cache)

    result = service.get_profile("user-1")
    assert result.data == {"name": "Ada"}
    assert result.degraded is True

def test_fallback_with_no_cached_data_is_explicit_not_silent():
    cache = FakeCache({})
    client = RaisingClient(DownstreamUnavailableError)
    service = ProfileService(downstream_client=client, cache=cache)
    result = service.get_profile("user-unknown")
    assert result.data is None and result.degraded is True, (
        "a cache miss during an outage must be an explicit degraded-empty result, not a crash and not a silently empty-looking success"
    )

# Integration level, against a virtualized real downstream
def test_real_client_routes_into_fallback_on_downstream_timeout(virtualized_profile_service):
    virtualized_profile_service.configure_unreachable()
    service = ProfileService(downstream_client=RealHttpClient(virtualized_profile_service.url), cache=FakeCache({"user-1": {"name": "Ada"}}))
    result = service.get_profile("user-1")
    assert result.degraded is True

# End-to-end level, with a latency bound
def test_e2e_fallback_is_fast(live_stack, virtualized_profile_service):
    virtualized_profile_service.configure_unreachable()
    start = time.monotonic()
    response = live_stack.get("/profile/user-1")
    elapsed = time.monotonic() - start
    assert response.json()["degraded"] is True
    assert elapsed < 1.0, f"fallback took {elapsed:.2f}s, expected close to the configured downstream timeout"

Trade-offs and pitfalls

• Testing only the unit level (mocking the exception directly) can pass even if the real client never actually throws that exception under a real timeout (a misconfigured timeout, or an exception type the real HTTP client doesn't actually raise); the integration-level test against a virtualized real client is what catches that gap.
• A fallback that is correct but slow is a common, easy-to-miss regression; without an explicit latency assertion at the end-to-end level, a change that adds an extra retry "just to be safe" before falling back can silently reintroduce the original problem (slow responses under a downstream outage) that the fallback existed to prevent.
• The cache-miss case is frequently untested because it requires deliberately setting up an empty cache, which is easy to forget when every other test fixture happens to pre-populate one; make it an explicit, separate test rather than relying on it to show up as a side effect of another scenario.

Direct answer
Test each timing variation as its own explicit scenario rather than hoping one "happy path" test exercises them all: fast completion, slow completion, mid-work failure, duplicate start requests, and read-after-write lag on the status endpoint each need their own fault-injected or time-controlled test, because each exercises a different code path in the job-lifecycle state machine.

Structured elaboration
An async-job-with-polling API is really a small state machine (SUBMITTED -> RUNNING -> SUCCEEDED / FAILED), and the interesting bugs live at the TRANSITIONS, not the steady states. Structure the test suite around the transitions:

1. Fast completion. Submit, then immediately poll. The test must handle the (correct) possibility that the job already finished before the first poll returns; a test that assumes there is always at least one RUNNING observation is itself buggy.
2. Slow / long-running job. Use a controllable delay in the job's own logic (a test double, not a real long sleep) so the test can poll multiple times and assert the status stays RUNNING with a stable job_id, then eventually transitions.
3. Mid-work failure. Inject a failure partway through the job's work and assert the status becomes FAILED with an actionable error, and that (if applicable) any partial side effects are rolled back or marked so a client polling FAILED doesn't misread partial output as complete.
4. Duplicate start requests. Submit the same logical job twice (same idempotency key, if the API supports one) and assert you get back the SAME job_id rather than two independent jobs silently doing the work twice; if the API has no idempotency key, assert (and document) that duplicate submissions are the caller's responsibility, which is itself a finding worth surfacing rather than assuming.
5. Visibility lag. After the job actually finishes internally, assert that a poll immediately after may still show RUNNING for the read-after-write consistency window, and that a bounded poll (with backoff) eventually shows SUCCEEDED; do not assume the status flips synchronously with completion just because it usually does in a fast test environment.

Worked example

import time

def poll_until_terminal(client, job_id, timeout_s=5.0, interval_s=0.05):
    deadline = time.monotonic() + timeout_s
    last = None
    while time.monotonic() < deadline:
        last = client.get_status(job_id)
        if last in ("SUCCEEDED", "FAILED"):
            return last
        time.sleep(interval_s)
    raise AssertionError(f"job {job_id} still {last!r} after {timeout_s}s")

def test_mid_work_failure_reports_failed_not_stuck():
    client = JobClient(work_fn=raises_after_partial_progress)
    job_id = client.start()
    status = poll_until_terminal(client, job_id)
    assert status == "FAILED"
    detail = client.get_error_detail(job_id)
    assert detail is not None, "a FAILED job must carry an actionable error detail, not a bare failure"
    marked_as_partial = "partial" in detail.lower()
    rolled_back = client.get_side_effects(job_id) == []
    assert marked_as_partial or rolled_back, (
        "a job that fails mid-work must either roll back its partial side effects, or explicitly "
        "mark the error detail as partial, so a caller never mistakes partial output for a complete result"
    )

def test_duplicate_start_returns_same_job_id():
    client = JobClient(work_fn=lambda: None)
    idempotency_key = "submit-42"
    job_id_1 = client.start(idempotency_key=idempotency_key)
    job_id_2 = client.start(idempotency_key=idempotency_key)
    assert job_id_1 == job_id_2, "duplicate submission under the same key must not spawn a second job"

Trade-offs and pitfalls

• Do not use real time.sleep(N) to simulate a "slow job"; inject a controllable delay or hook into the job's own execution so the test suite doesn't become slow AND flaky at the same time.
• The duplicate-start test is only meaningful if the API design actually intends idempotent submission; if it doesn't, this test should assert the documented behavior (two independent jobs), not an aspirational one, or you will be "fixing" a correctly-behaving system.
• Visibility-lag testing is easy to skip because it rarely reproduces on a fast, quiet test environment; if the read path and write path for job status are backed by different stores or caches, add an explicit test that reads immediately after a status change and tolerates (and asserts you tolerate) a brief stale read, rather than assuming synchronous consistency because it happened to hold locally.
• The mid-work-failure assertion above is deliberately written as an explicit either/or (marked_as_partial or rolled_back) rather than a compound boolean expression: an earlier, more compact phrasing of this same check used operator precedence in a way that made it pass regardless of whether rollback actually happened, as long as the error message text didn't happen to contain the word "partial" -- a silent, vacuous pass. Prefer named intermediate booleans over compact and/or chains in any assertion that encodes more than one condition.