Distributed Systems and Microservices Testing Questions

Direct answer
Build the harness on tooling that injects faults at the network layer you actually need (packet-level shaping for latency and packet loss, service-mesh or proxy-level fault injection for HTTP-level errors and resets), justify the choice by which layer the fault needs to be believable at, measure that the fault genuinely landed by watching the affected connections directly (not just trusting the injection tool's own exit code), and assert correctness properties (does the system stay available in a degraded mode, does it preserve whatever consistency guarantee it claims) while keeping the blast radius to a single, clearly-scoped set of services.

Structured elaboration
Three design decisions carry this harness:

1. Tooling layer. Low-level network-shaping tools operate below the application (they manipulate the actual packets: dropping, delaying, corrupting), so they are the most realistic way to simulate a real network partition or lossy link, but they require access to the host/container network namespace and are coarser to target precisely at one service pair. Proxy or service-mesh-level fault injection operates at the application/HTTP layer (it can inject a 503, add artificial latency to one specific route, or reset a specific connection), which is easier to scope precisely to one dependency and easier to run in CI without host-level privileges, at the cost of being a slightly less faithful simulation of a true network-layer failure. Pick network-layer tooling when you specifically need to validate low-level behavior (TCP timeout handling, connection-reset recovery); pick proxy/mesh-level injection when you are testing application-level resilience logic (retries, circuit breakers, fallbacks) and want fine-grained, easily-automatable control.
2. Confirming the fault landed. Do not trust that calling the injection tool means the fault is active; independently observe an effect that proves it (a measured round-trip time above the injected floor, an actual dropped-packet counter increment, a proxy's own fault-injection metric). A test that injects a fault, asserts on the system's behavior, but never confirms the fault was really active can pass for the wrong reason (the fault silently failed to apply, and the system behaved correctly simply because nothing bad happened).
3. Blast-radius control. Scope the fault to a clearly bounded target (one service pair, one route, one percentage of traffic) using labels/selectors the tooling supports, run it first against a non-production, isolated environment, and have an automatic kill-switch (a maximum experiment duration, an automatic revert) so a misconfigured or unexpectedly severe fault cannot spread beyond the intended scope.

Worked example
A small harness abstraction that could be backed by either a network-shaping tool or a mesh-level fault-injection API, showing the "confirm it landed" discipline concretely:

class FaultInjector:
    """Backed by either a network-namespace tool or a service-mesh fault-injection
    API; the interface is deliberately the same so tests don't care which."""
    def inject_latency(self, source, target, delay_ms, duration_s):
        raise NotImplementedError
    def measured_effect(self, source, target):
        """Returns the actually-observed extra latency, independent of what was requested."""
        raise NotImplementedError

def test_partition_between_order_service_and_inventory_service():
    injector = MeshFaultInjector(namespace="test")
    injector.inject_latency("order-service", "inventory-service", delay_ms=2000, duration_s=30)

    observed_delay = injector.measured_effect("order-service", "inventory-service")
    assert observed_delay >= 1800, (
        f"fault injection did not actually land: measured only {observed_delay}ms of the requested 2000ms"
    )

    response = order_client.place_order(sample_order())
    assert response.status == "ACCEPTED_DEGRADED", (
        "order service should fall back to async inventory confirmation under high dependency latency, "
        f"got {response.status!r}"
    )
    assert response.availability_impact == "none", "the caller should never see an outage from this fault alone"

Trade-offs and pitfalls

• Network-namespace-level tools usually require elevated privileges and host access that a shared CI runner may not grant; service-mesh or proxy-level injection is more CI-friendly but only as faithful as the mesh's own fault-injection fidelity (it typically cannot simulate a true partition below the HTTP layer, such as a TCP-level black hole).
• A fault that is scoped too broadly (an entire percentage of ALL traffic instead of one route) risks the experiment itself becoming the incident; always start with a scoped, short-duration, single-route experiment and expand deliberately.
• The single most common false-positive in this class of test is skipping the "confirm the fault actually landed" step; a test environment where the injection silently no-ops (a misconfigured selector, an unsupported combination of options) will still report a passing test, for entirely the wrong reason.

Direct answer
Simulate the exact failure point (the downstream acknowledges receipt of the event, but the local database write behind it fails) as a deliberate, injectable fault, then assert three things: the operation is not silently lost (it is retried or dead-lettered, never dropped), a retry of the same event does not double-apply its effect (idempotency), and there is an observable signal (a metric, a log line, a trace tag) that proves the failure and its recovery actually happened, not just that the system eventually looked correct by coincidence.

Structured elaboration
This is a specific and common partial-failure shape: acknowledgment and persistence are two separate operations that are not atomic with each other, so a crash or bug between them leaves the system in an inconsistent state (the sender believes delivery succeeded; the receiver has no record of it). Test design breaks into:

1. Fault injection at the precise seam. Inject the fault BETWEEN acknowledgment and the database write, not before or after both. A test harness that can only kill the whole process (not target the specific gap) will miss this class of bug entirely, because it can accidentally always land on the safe side of the seam.
2. Recovery assertion. After the injected failure, the event must eventually still be applied: either the sender's at-least-once delivery redelivers it (because the sender's own ack was never actually confirmed, if you model it correctly), or a reconciliation/outbox-pattern process picks it up from a durable log. The test asserts the final state converges to correct, not just that no exception was thrown.
3. Idempotency under the redelivery the recovery path causes. Once recovery redelivers the event, a second application of the same event must not double-charge, double-count, or otherwise duplicate its effect. This needs its own explicit assertion (apply the event twice deliberately and assert state is identical to applying it once), it is not automatically covered by the recovery assertion above.
4. Observability proof. A counter or trace tag that increments specifically on "recovered from ack-without-persist" lets you assert programmatically that the FAILURE PATH was actually exercised, not just that the happy path also happens to produce a correct final state (a test that never actually triggers the bug it claims to test is a false-negative risk).

Worked example
A fault-injection point modeled as a hook the test controls, verifying both recovery and idempotency:

class OrderProcessor:
    def __init__(self, db, fault_hook=None):
        self.db = db
        self.fault_hook = fault_hook or (lambda: None)

    def handle_event(self, event):
        self.ack(event)              # step A: acknowledge
        self.fault_hook()             # test injects a crash HERE, between A and B
        self.db.persist(event)        # step B: persist

# Test: inject the crash between ack and persist, then simulate a redelivery
# (as an at-least-once source would do once it notices no downstream commit),
# then apply the SAME event again and assert no double effect.
def test_ack_without_persist_recovers_and_stays_idempotent():
    db = FakeIdempotentStore()
    crashed = {"value": False}
    def crash_once():
        if not crashed["value"]:
            crashed["value"] = True
            raise ConnectionError("simulated crash before persist")
    processor = OrderProcessor(db, fault_hook=crash_once)

    event = {"event_id": "evt-1", "order_id": "ord-1", "amount": 50}
    try:
        processor.handle_event(event)
    except ConnectionError:
        pass  # expected: this simulates the ack-then-crash window

    assert db.get("ord-1") is None, "nothing should be persisted yet"

    # redelivery: the source resends because it never saw a downstream commit
    processor.handle_event(event)
    assert db.get("ord-1") == {"amount": 50}

    # a THIRD delivery of the identical event (e.g. a slow duplicate arriving
    # late) must not double-apply
    processor.handle_event(event)
    assert db.get("ord-1") == {"amount": 50}, "replay must not double-apply"

FakeIdempotentStore here is assumed to dedupe on event_id; the test's real point is the sequencing (crash between ack and persist, then two more deliveries), not the store implementation.

Trade-offs and pitfalls

• If your real system's delivery guarantee does not actually redeliver on this failure mode (many systems only redeliver on a NETWORK failure of the ack itself, not a downstream persistence failure the sender can't see), this test will correctly show the event is permanently lost, which is the actual finding you want: it proves the sender's redelivery model does not cover this specific gap, and the fix belongs in the architecture, not the test.
• Testing this seam requires the ability to inject a fault at a precise point in the code path; if the two operations are buried inside a single opaque library call, you may need to add a seam (a hook, a small wrapper) purely for testability, which is a legitimate and common reason production code grows a small extension point.
• Do not stop at "the test passed once"; run the idempotent-replay assertion at least twice (as shown) since an off-by-one bug in a dedupe check often only shows up on the second or third redelivery, not the first.

Direct answer
Combine adversarial scheduling (deliberately interleaving lock-acquisition attempts and RPC calls to provoke a cycle) with resource-starvation tests (holding a lock or exhausting a pool just long enough to force contention) and a watchdog that asserts forward progress within a bounded time, so the test suite catches both a hard deadlock (nothing ever progresses) and a livelock (things keep happening, but nothing useful completes).

Structured elaboration

1. Provoking the condition deliberately. A deadlock or livelock is, by definition, rare under normal random timing; a test suite that just runs the system under light concurrent load will usually not hit it. Instead, script the specific interleaving that creates a lock-ordering cycle (service A acquires lock 1 then requests lock 2 from service B, while service B acquires lock 2 then requests lock 1 from service A) using controllable delays or an explicit test-only coordination hook that pauses each side at the critical moment.
2. Resource-starvation tests. Separately, saturate a shared resource (a connection pool, a thread pool, a rate-limited downstream) to just below its limit and add one more contender, and assert the system either queues it fairly and eventually serves it (progress, just delayed) or rejects it explicitly (a defined backpressure signal), rather than silently hanging forever.
3. Watchdog assertion. Wrap the scenario in a watchdog that asserts SOME forward-progress signal (a completed request, an incrementing counter, a released lock) occurs within a generous but bounded time window; a livelock is specifically the case where activity (retries, lock attempts) continues but the progress signal never fires, so "no timeout occurred" is not sufficient evidence of correctness, only "the progress signal fired" is.
4. Observability signals to add. Thread/goroutine dumps or stack traces on timeout (to see exactly where each participant is blocked), lock-acquisition and lock-wait-time metrics (a lock held far longer than its expected duration is a strong deadlock signal even before a full hang), and request-latency histograms (a livelock often shows as a spike in retries or a plateau in the completion rate, not literally zero throughput).

Worked example: a hard deadlock
A scripted two-service lock-ordering scenario with a watchdog assertion:

import threading

def test_lock_ordering_cycle_is_prevented_or_detected():
    lock1, lock2 = threading.Lock(), threading.Lock()
    progress = threading.Event()
    barrier = threading.Barrier(2)

    def service_a():
        with lock1:
            barrier.wait()          # deliberately synchronize both sides at the critical moment
            with lock2:
                progress.set()

    def service_b():
        with lock2:
            barrier.wait()
            with lock1:
                progress.set()

    # daemon=True is deliberate: this scenario deadlocks the two threads
    # PERMANENTLY (Python's plain threading.Lock has no built-in deadlock
    # detector, unlike a real database engine). Without daemon=True, the two
    # blocked threads never terminate, and the interpreter hangs forever at
    # exit waiting to join them, even after the watchdog assertion below has
    # already correctly failed and reported the deadlock.
    t1 = threading.Thread(target=service_a, daemon=True)
    t2 = threading.Thread(target=service_b, daemon=True)
    t1.start(); t2.start()

    made_progress = progress.wait(timeout=2.0)   # the watchdog
    assert made_progress, (
        "deadlock detected: neither side made progress within the watchdog timeout "
        "(this specific interleaving requires a consistent lock-ordering discipline, "
        "or a lock-timeout-and-retry strategy, to resolve)"
    )
    t1.join(timeout=0.1); t2.join(timeout=0.1)

This test is EXPECTED to fail against a naive implementation that acquires locks in inconsistent order (that is the point: it deliberately provokes the classic lock-ordering deadlock), and passing means the system under test has a real mitigation (consistent lock ordering, a timeout-and-retry, or a deadlock detector) in place.

Worked example: a livelock, the other half of this question
A hard deadlock and a livelock look identical from a bare "did it time out" check, so a suite that only has the example above has not actually tested for livelocks at all. Here two "polite" agents each back off the instant they detect contention, in perfect lockstep, so neither ever completes, yet both stay busy retrying: a deterministic, single-threaded simulation (rather than real OS threads, to avoid exactly the kind of real-timing race that makes ad hoc threaded livelock demos flaky) makes this reliably reproducible:

class Agent:
    def __init__(self, name, own_lock, other_lock):
        self.name = name
        self.own_lock = own_lock
        self.other_lock = other_lock
        self.holds_own = False
        self.retries = 0
        self.done = False

    def decide(self, snapshot):
        """Decide this round's action from a snapshot taken at the START of
        the round, so neither agent gets a sequencing advantage within the
        round -- a true simultaneous update, not a sequential one."""
        if self.done:
            return "noop"
        if not self.holds_own:
            return "take_own"
        if snapshot.get(self.other_lock) is None:
            return "take_other"
        return "back_off"

    def apply(self, action, held_by):
        if action == "take_own":
            held_by[self.own_lock] = self.name
            self.holds_own = True
        elif action == "take_other":
            held_by[self.other_lock] = self.name
            self.done = True
        elif action == "back_off":
            held_by[self.own_lock] = None
            self.holds_own = False
            self.retries += 1

def test_livelock_symmetric_polite_backoff_never_converges(max_steps=200):
    held_by = {"lock1": None, "lock2": None}
    a = Agent("a", "lock1", "lock2")
    b = Agent("b", "lock2", "lock1")
    for step in range(max_steps):
        if a.done or b.done:
            break
        snapshot = dict(held_by)
        action_a, action_b = a.decide(snapshot), b.decide(snapshot)
        a.apply(action_a, held_by)
        b.apply(action_b, held_by)   # both actions land together, from the same snapshot

    made_progress = a.done or b.done
    assert not made_progress, "expected the symmetric polite back-off pattern to livelock forever, not complete"
    assert a.retries > 10 and b.retries > 10, (
        f"expected substantial, ongoing retry activity from both sides (the defining signature of a "
        f"livelock, as opposed to a deadlock's total silence): got a={a.retries} b={b.retries}"
    )

Running this prints made_progress=False with a.retries=100 b.retries=100 after the full 200 simulated rounds: real, ongoing activity, and zero useful progress, which is exactly the property a watchdog based only on "did anything happen" would miss, and exactly why the observability signals in point 4 (a retry-rate or lock-attempt counter, not just a completion counter) matter for distinguishing this from a slow-but-healthy system.

Trade-offs and pitfalls

• A watchdog timeout that is too short will falsely flag a system that is merely slow (heavily contended but still making progress) as deadlocked; calibrate the timeout against the system's actual expected worst-case latency under contention, not an arbitrary round number.
• Deliberately scripted interleavings (using a barrier, as above) prove a SPECIFIC scenario is or isn't handled; they do not prove the absence of ALL possible deadlocks, which is a fundamentally harder problem. Pair scripted scenarios with production-side lock-wait-time monitoring as a complementary, ongoing detection mechanism.
• Distinguishing a livelock from legitimate-but-slow retry behavior requires a real progress signal, not just "activity is happening"; a system that keeps retrying a doomed operation forever looks identical to one that is slowly succeeding unless the test asserts on an actual completion signal, not just CPU or network activity.
• When a scenario is EXPECTED to deadlock or livelock permanently (as both examples above are, by design), make sure any threads used to model it are daemonized or otherwise bounded; a non-daemon thread that blocks forever will hang the test process itself at exit, turning a correctly-failing assertion into a CI job that never returns.

Direct answer
Strategy is layered: use service virtualization to make a downstream's flakiness controllable and repeatable in tests, inject latency and errors at that virtualized boundary to exercise the calling service's resilience logic directly, assert the resilience mechanisms (timeouts firing, circuit breakers opening) actually engage rather than merely "the call eventually returned something," and make every test failure carry enough context (which mechanism should have engaged and didn't) that a developer can act on it immediately.

Structured elaboration

1. Service virtualization as the control point. Replace the real downstream with a virtualized stand-in the test fully controls, so "the downstream is flaky" becomes a deterministic, repeatable test input (configure the virtualized service to return errors, hang, or respond slowly on command) rather than something you can only hope to reproduce against a real, genuinely-unreliable dependency.
2. Injecting latency and errors precisely. Configure the virtualized dependency to simulate the SPECIFIC failure shape you're testing for: a slow response just under the caller's timeout (does the caller still time out appropriately, or does it wait forever due to a misconfigured client timeout), a burst of errors (does the caller's circuit breaker actually open after the configured error threshold), and a full hang (does the caller's own timeout fire rather than blocking a thread/connection indefinitely).
3. Asserting the mechanism engaged, not just the outcome. It is not enough to assert "the caller returned an error instead of crashing"; assert the SPECIFIC mechanism did its job: the circuit breaker's state actually transitioned to open, a fallback path was actually invoked (not just that some response came back), and the caller stopped issuing new outbound calls to the failing dependency once the breaker opened (proving the breaker is actually protecting the failing dependency from further load, which is the entire point of a circuit breaker).
4. Making failures actionable. When a resilience-pattern test fails, its assertion message should name exactly what should have happened ("circuit breaker should have opened after 5 consecutive failures, but remained closed after 8") rather than a bare assert result == expected, since these tests exist specifically to catch subtle misconfigurations (a threshold set wrong, a timeout not actually wired to the client) that are otherwise invisible until a real incident.

Worked example

def test_circuit_breaker_opens_and_stops_calling_failing_dependency():
    virtual_downstream = VirtualizedService(name="inventory-service")
    virtual_downstream.configure_all_calls_fail()

    client = ResilientClient(virtual_downstream, breaker_threshold=5, breaker_cooldown_s=1.0)

    for _ in range(5):
        try:
            client.check_inventory("sku-1")
        except DependencyError:
            pass

    assert client.breaker_state == "OPEN", (
        f"expected circuit breaker OPEN after 5 consecutive failures, got {client.breaker_state!r}"
    )

    calls_before = virtual_downstream.call_count
    result = client.check_inventory("sku-2")  # breaker is open: should short-circuit
    assert virtual_downstream.call_count == calls_before, (
        "a call while the breaker is OPEN must not reach the failing dependency at all"
    )
    assert result.source == "fallback", f"expected a fallback response while open, got {result.source!r}"

Trade-offs and pitfalls

• Testing against a virtualized dependency proves the CALLER's resilience logic works given a controlled failure shape; it does not prove the real downstream fails in exactly that shape in production, so pair this with production observability (actual error rates, actual breaker-state transitions) to confirm the assumptions the tests encode still hold.
• The most common false pass in this area is asserting only the final returned value and never inspecting internal breaker/circuit state; a system that happens to return a correct-looking fallback response even with a broken (always-closed) circuit breaker will pass a shallow test while still hammering the failing dependency with load in production.
• Latency-injection tests are sensitive to the ACTUAL configured client timeout; if the test's injected delay is close to the timeout threshold, minor scheduling jitter in CI can make the test flaky in either direction. Inject a delay clearly and safely on one side of the threshold (well above or well below), not right at the boundary, unless you are specifically testing boundary behavior with a tolerant, retried assertion.

Direct answer
For stateful, long-running async workflows, the core testing move is to make the workflow's progress OBSERVABLE at every meaningful step (via checkpoints your test can poll) and to make the workflow's INPUTS deterministic (fixed IDs, fixed clocks, fixed random seeds), so a test can assert on a specific intermediate state rather than only the final one, and can reproduce a failure exactly when it finds one.

Structured elaboration
Break this into three concerns:

1. Orchestration and checkpoints. A long-running workflow (a saga, a multi-step approval process, an order-fulfillment pipeline) should emit a durable state transition at every step: PENDING -> VALIDATED -> CHARGED -> SHIPPED, each persisted with a timestamp and enough context to explain why it moved. Tests assert against these named states, not against a black-box "is it done yet." This also means a stuck workflow is diagnosable: the last recorded state tells you exactly which step never completed.
2. Determinism. Long-running workflows usually depend on time (SLAs, retry backoff, timeouts) and on identifiers generated during the run. For tests: inject a fake or controllable clock instead of time.now(), generate all IDs from a seeded generator so a failing run can be replayed exactly, and never depend on wall-clock sleeps to represent "time has passed" (advance a fake clock instead, where the workflow engine supports it, or use a real one only with generous, monitored timeouts).
3. Assertion strategy for flakiness. Async, multi-service workflows have many more places to be surprised by ordering than a single-service test. The two disciplines that most reduce flakiness are: (a) assert on the workflow's OWN recorded checkpoints (an internal source of truth) rather than inferring state indirectly from side effects in other services, and (b) use assert_eventually-style polling (see the eventual-consistency pattern) with a timeout keyed to the SLA the workflow is actually supposed to meet, not an arbitrary number.

Worked example
A workflow test harness sketch, showing checkpoint-driven assertions plus a seeded, deterministic ID generator:

import itertools
import time

class DeterministicIdGen:
    """Every test run with the same seed produces the same sequence of ids,
    so a failing run's ids can be grepped straight out of logs and replayed."""
    def __init__(self, seed=1):
        self._counter = itertools.count(seed)
    def next_id(self):
        return f"wf-{next(self._counter):06d}"

def run_workflow_and_wait_for(engine, checkpoint_store, workflow_id, expected_state, timeout_s=10.0):
    deadline = time.monotonic() + timeout_s
    last = None
    while time.monotonic() < deadline:
        last = checkpoint_store.latest_state(workflow_id)
        if last == expected_state:
            return last
        if last == "FAILED":
            raise AssertionError(f"workflow {workflow_id} failed before reaching {expected_state}")
        time.sleep(0.05)
    raise AssertionError(f"workflow {workflow_id} stuck at {last!r}, never reached {expected_state!r} within {timeout_s}s")

# usage
idgen = DeterministicIdGen(seed=42)
workflow_id = idgen.next_id()
engine.start(workflow_id, payload={...})
run_workflow_and_wait_for(engine, checkpoint_store, workflow_id, "VALIDATED")
run_workflow_and_wait_for(engine, checkpoint_store, workflow_id, "CHARGED")
final = run_workflow_and_wait_for(engine, checkpoint_store, workflow_id, "SHIPPED")
assert final == "SHIPPED"

Asserting each intermediate checkpoint in sequence (rather than only the terminal SHIPPED state) means a failure pinpoints the exact step that regressed, instead of forcing a manual investigation across every service in the pipeline.

Trade-offs and pitfalls

• Checkpoint-driven testing requires the workflow engine or the services themselves to actually persist intermediate state; if they don't, you either have to add that instrumentation (an investment worth making, since it also helps production debugging) or fall back to weaker, indirect assertions.
• A seeded ID generator only gives you reproducibility if every non-deterministic input (clock, randomness, network ordering) is also controlled; a workflow that is deterministic in its IDs but still races on message-processing order will still flake, just less often.
• Watch for tests that grow SLA-sized timeouts ("wait 30 seconds because the real SLA is 30 seconds") without also asserting the workflow reached each intermediate checkpoint at a reasonable pace; a workflow that is technically within SLA but spent 29 of those 30 seconds stuck can still hide a real regression from a single terminal-state assertion.