Distributed Systems and Microservices Testing Questions

Direct answer
Test each timing variation as its own explicit scenario rather than hoping one "happy path" test exercises them all: fast completion, slow completion, mid-work failure, duplicate start requests, and read-after-write lag on the status endpoint each need their own fault-injected or time-controlled test, because each exercises a different code path in the job-lifecycle state machine.

Structured elaboration
An async-job-with-polling API is really a small state machine (SUBMITTED -> RUNNING -> SUCCEEDED / FAILED), and the interesting bugs live at the TRANSITIONS, not the steady states. Structure the test suite around the transitions:

1. Fast completion. Submit, then immediately poll. The test must handle the (correct) possibility that the job already finished before the first poll returns; a test that assumes there is always at least one RUNNING observation is itself buggy.
2. Slow / long-running job. Use a controllable delay in the job's own logic (a test double, not a real long sleep) so the test can poll multiple times and assert the status stays RUNNING with a stable job_id, then eventually transitions.
3. Mid-work failure. Inject a failure partway through the job's work and assert the status becomes FAILED with an actionable error, and that (if applicable) any partial side effects are rolled back or marked so a client polling FAILED doesn't misread partial output as complete.
4. Duplicate start requests. Submit the same logical job twice (same idempotency key, if the API supports one) and assert you get back the SAME job_id rather than two independent jobs silently doing the work twice; if the API has no idempotency key, assert (and document) that duplicate submissions are the caller's responsibility, which is itself a finding worth surfacing rather than assuming.
5. Visibility lag. After the job actually finishes internally, assert that a poll immediately after may still show RUNNING for the read-after-write consistency window, and that a bounded poll (with backoff) eventually shows SUCCEEDED; do not assume the status flips synchronously with completion just because it usually does in a fast test environment.

Worked example

import time

def poll_until_terminal(client, job_id, timeout_s=5.0, interval_s=0.05):
    deadline = time.monotonic() + timeout_s
    last = None
    while time.monotonic() < deadline:
        last = client.get_status(job_id)
        if last in ("SUCCEEDED", "FAILED"):
            return last
        time.sleep(interval_s)
    raise AssertionError(f"job {job_id} still {last!r} after {timeout_s}s")

def test_mid_work_failure_reports_failed_not_stuck():
    client = JobClient(work_fn=raises_after_partial_progress)
    job_id = client.start()
    status = poll_until_terminal(client, job_id)
    assert status == "FAILED"
    detail = client.get_error_detail(job_id)
    assert detail is not None, "a FAILED job must carry an actionable error detail, not a bare failure"
    marked_as_partial = "partial" in detail.lower()
    rolled_back = client.get_side_effects(job_id) == []
    assert marked_as_partial or rolled_back, (
        "a job that fails mid-work must either roll back its partial side effects, or explicitly "
        "mark the error detail as partial, so a caller never mistakes partial output for a complete result"
    )

def test_duplicate_start_returns_same_job_id():
    client = JobClient(work_fn=lambda: None)
    idempotency_key = "submit-42"
    job_id_1 = client.start(idempotency_key=idempotency_key)
    job_id_2 = client.start(idempotency_key=idempotency_key)
    assert job_id_1 == job_id_2, "duplicate submission under the same key must not spawn a second job"

Trade-offs and pitfalls

• Do not use real time.sleep(N) to simulate a "slow job"; inject a controllable delay or hook into the job's own execution so the test suite doesn't become slow AND flaky at the same time.
• The duplicate-start test is only meaningful if the API design actually intends idempotent submission; if it doesn't, this test should assert the documented behavior (two independent jobs), not an aspirational one, or you will be "fixing" a correctly-behaving system.
• Visibility-lag testing is easy to skip because it rarely reproduces on a fast, quiet test environment; if the read path and write path for job status are backed by different stores or caches, add an explicit test that reads immediately after a status change and tolerates (and asserts you tolerate) a brief stale read, rather than assuming synchronous consistency because it happened to hold locally.
• The mid-work-failure assertion above is deliberately written as an explicit either/or (marked_as_partial or rolled_back) rather than a compound boolean expression: an earlier, more compact phrasing of this same check used operator precedence in a way that made it pass regardless of whether rollback actually happened, as long as the error message text didn't happen to contain the word "partial" -- a silent, vacuous pass. Prefer named intermediate booleans over compact and/or chains in any assertion that encodes more than one condition.

Direct answer
Orchestrate startup with explicit dependency ordering plus application-level readiness checks (not just process-alive or port-open checks), and treat "reports healthy" and "can actually serve a real request correctly" as two different things to verify, since a service can report healthy (its HTTP server is up) well before its own dependencies (a database connection pool, a cache warm-up, a schema migration) are actually ready to serve real traffic.

Structured elaboration

1. Orchestrating the environment. Whether using docker-compose or Kubernetes, express the dependency graph explicitly (service X depends on database Y and cache Z) so the orchestrator starts things in a sensible order and, more importantly, so the TEST SUITE knows which readiness checks to wait on before running anything against a given service.
2. Deciding when a dependency is truly ready, not just started. A container reporting "running" only means its process launched; a database container can be running for several seconds before it actually accepts connections, and an application container can be running before it finishes its own startup migrations or cache warm-up. The right readiness signal is an APPLICATION-LEVEL health check the service itself exposes (a /healthz endpoint that only returns 200 once its own dependencies are confirmed reachable and its own startup tasks are complete), not merely "the process is running" or "the port accepts a TCP connection."
3. Avoiding false positives from a service that reports healthy too early. A common bug is a health-check endpoint that returns 200 as soon as the HTTP server itself starts, before the service has actually verified its OWN downstream dependencies are reachable. Guard against this by making the health check itself verify real downstream connectivity (a lightweight ping to the database, a cache connectivity check) rather than a hardcoded 200, and by having the test harness's wait-loop perform a SEMANTIC check in addition to the reported health status wherever feasible (for example, issuing one real, cheap request the service can only answer correctly if it's truly ready, not just relying on the health endpoint's own self-report).
4. The wait strategy itself. Poll each service's readiness endpoint on an interval, with an overall timeout; only proceed to run the actual test suite once every dependency in the graph reports ready, and fail loudly (naming which service never became ready) rather than proceeding and getting a confusing cascade of unrelated test failures.

Worked example
A readiness-wait helper distinguishing "reports healthy" from "can actually serve a request," verified with a fake orchestration layer:

import time

def wait_for_ready(services, timeout_s=60, interval_s=1.0, semantic_check=None):
    deadline = time.monotonic() + timeout_s
    not_ready = set(services)
    while time.monotonic() < deadline and not_ready:
        for name in list(not_ready):
            if health_check(name) and (semantic_check is None or semantic_check(name)):
                not_ready.discard(name)
        if not_ready:
            time.sleep(interval_s)
    if not_ready:
        raise AssertionError(f"services never became ready within {timeout_s}s: {sorted(not_ready)}")

def test_environment_is_actually_ready_before_running_suite():
    services = ["postgres", "service-a", "service-b"]

    def semantic_check(name):
        # a health endpoint reporting 200 is necessary but not sufficient;
        # also confirm the service can answer one real, cheap query
        if name == "service-a":
            return service_a_client.ping_dependencies() is True
        return True

    wait_for_ready(services, timeout_s=30, semantic_check=semantic_check)

The semantic_check hook here is deliberately separate from health_check, so a service that reports 200 prematurely (before it has actually verified its own dependencies) still gets caught by the additional, real dependency-ping check.

Trade-offs and pitfalls

• Relying solely on a health endpoint that the service itself controls is only as trustworthy as that endpoint's own implementation; a health check that always returns 200 the moment the HTTP server binds its port is common, easy to write by accident, and defeats the whole purpose of readiness gating.
• A semantic check (an actual cheap request) adds a small amount of latency and coupling to the harness, but is often the only thing that reliably catches the "reports healthy but isn't really ready" class of bug; use it at least for the services most prone to this pattern (anything with its own downstream dependencies to warm up).
• An overall timeout that's too short makes the suite flaky on a slower CI runner; one that's too long means a genuinely broken environment wastes significant CI time before failing. Calibrate against measured real startup times, and fail with the specific service name(s) still not ready, never a generic "setup timed out."

Direct answer
Design test scenarios around each named failure mode as its own explicit test (duplicate delivery, broker-failure message loss, consumer crash-and-restart mid-processing, reordering, backpressure, visibility-timeout expiry, and poison messages), simulate each with a fault-injecting test harness rather than hoping production traffic happens to exercise them, assert application-level correctness (idempotent processing, no lost or double-applied effects) rather than only "no exception was thrown," generate deterministic test messages so a found failure can be reproduced exactly, and back every assertion with the same observability metrics (consumer lag, redelivery counts, DLQ depth) a real on-call engineer would use to verify recovery in production.

Structured elaboration
At-least-once delivery means the QUEUE promises a message is delivered at least once, but says nothing about exactly once, or in order, across failures; the application has to supply the missing guarantees itself, and each of the following needs its own test:

1. Duplicate delivery. Deliver the same message twice (same message ID) and assert the consumer's effect is applied exactly once (an idempotency-key-based dedupe check, or an operation that is naturally idempotent).

2. Broker-failure message loss. Simulate the broker failing after accepting a message but before it is durably committed (if the broker's own contract allows this window) and assert the PRODUCER side has its own confirmation/retry logic, so message loss at this layer is bounded by the producer's own retry, not silently absorbed.

3. Consumer crash-and-restart. Kill the consumer mid-processing (after it read the message but before it acknowledged) and assert the message is redelivered (since it was never acked) and reprocessing it produces the correct final state, not a partial or corrupted one.

4. Reordering. Deliver messages for the same logical entity out of their production order and assert the consumer either has an ordering-independent design (commutative updates) or explicitly detects and correctly handles the out-of-order case (a version check that rejects an older update arriving late).

5. Backpressure. Flood the consumer faster than it can process and assert the system degrades gracefully (bounded queue growth, load shedding, or backpressure signaled to the producer) rather than an unbounded memory blow-up or a silent message drop.

6. Visibility timeout expiry. Hold a message past its visibility timeout without acking it and assert it becomes available for redelivery to another consumer, and that BOTH the original (now-late) processing and the redelivered processing converge to the same correct idempotent result if the original consumer eventually also finishes.

7. Poison messages. Deliver a message the consumer can never successfully process (malformed payload, a bug that always throws) and assert it is moved to a dead-letter queue after a bounded number of retries, rather than blocking the queue for every other message behind it forever.

8. Observability metrics as verification evidence, not just test assertions. Beyond the pass/fail test assertions above, instrument the harness itself to emit the same signals you would want in production: a consumer-lag gauge (how far behind the latest offset each consumer is), a redelivery counter (how many times a given message id was redelivered), and a DLQ-depth gauge. Assert on these directly where relevant (for example, assert dlq_depth_metric.value() == 1 after the poison-message scenario, or assert redelivery_count_metric.value(message_id="m1") == 1 after the visibility-timeout scenario), so a test failure is corroborated by the same metrics an on-call engineer would look at in a real incident, and so a regression that silently stops emitting a metric (even while the underlying behavior is still correct) is itself caught.

Worked example

def test_poison_message_goes_to_dlq_without_blocking_the_queue():
    dlq = []
    dlq_depth_metric = Counter()
    redelivery_count_metric = Counter()
    queue = FakeAtLeastOnceQueue(max_retries=3, dead_letter_sink=lambda m: (dlq.append(m), dlq_depth_metric.inc()))
    processed_good = []

    def handler(message):
        if message.body == "POISON":
            raise ValueError("cannot process this payload, ever")
        processed_good.append(message.body)

    queue.enqueue(Message(id="m1", body="POISON"))
    queue.enqueue(Message(id="m2", body="good-payload"))

    queue.drain(handler)

    assert [m.id for m in dlq] == ["m1"], "poison message should land in the DLQ after exhausting retries"
    assert processed_good == ["good-payload"], "a poison message must not block processing of the message behind it"
    assert dlq_depth_metric.value() == 1, "the DLQ-depth metric must reflect the one poisoned message, corroborating the test assertion with the same signal on-call would see"

def test_redelivery_after_visibility_timeout_is_idempotent():
    store = IdempotentApplyStore()
    redelivery_count_metric = Counter()
    queue = FakeAtLeastOnceQueue(visibility_timeout_s=0.1, on_redeliver=lambda mid: redelivery_count_metric.inc(mid))
    queue.enqueue(Message(id="m1", body={"op": "credit", "account": "a1", "amount": 10}))

    first = queue.receive()
    time.sleep(0.15)
    second = queue.receive()

    store.apply(second.body, idempotency_key=second.id)
    store.apply(first.body, idempotency_key=first.id)

    assert store.balance("a1") == 10, "redelivery due to visibility-timeout expiry must not double-credit"
    assert redelivery_count_metric.value("m1") == 1, "the redelivery counter must show exactly one redelivery for m1, not zero (which would mean the scenario never actually fired) and not more than one"

Trade-offs and pitfalls

• Building a fake queue that faithfully reproduces visibility-timeout and redelivery semantics is itself nontrivial; where possible, run these tests against a real (local, disposable) instance of the actual message broker rather than a hand-rolled fake, to avoid the fake's own bugs masking or fabricating findings.
• Poison-message tests must assert BOTH halves: the poison message eventually stops retrying (lands in the DLQ), AND unrelated messages behind it are not blocked; a suite that only tests one half can pass while the other silently regresses.
• Testing reordering is easy to under-specify; be explicit about which entities' ordering matters (usually per-key, not global) and test out-of-order delivery specifically WITHIN one key's message stream, since that is the case a naive "just process messages as they arrive" consumer is most likely to get wrong.
• Asserting on a metric alongside a direct state assertion (as in the DLQ-depth and redelivery-count checks above) also catches a subtler regression: the underlying behavior staying correct while the metric silently stops being emitted, which would otherwise go unnoticed until an actual production incident where on-call has no signal to look at.

Direct answer
Combine adversarial scheduling (deliberately interleaving lock-acquisition attempts and RPC calls to provoke a cycle) with resource-starvation tests (holding a lock or exhausting a pool just long enough to force contention) and a watchdog that asserts forward progress within a bounded time, so the test suite catches both a hard deadlock (nothing ever progresses) and a livelock (things keep happening, but nothing useful completes).

Structured elaboration

1. Provoking the condition deliberately. A deadlock or livelock is, by definition, rare under normal random timing; a test suite that just runs the system under light concurrent load will usually not hit it. Instead, script the specific interleaving that creates a lock-ordering cycle (service A acquires lock 1 then requests lock 2 from service B, while service B acquires lock 2 then requests lock 1 from service A) using controllable delays or an explicit test-only coordination hook that pauses each side at the critical moment.
2. Resource-starvation tests. Separately, saturate a shared resource (a connection pool, a thread pool, a rate-limited downstream) to just below its limit and add one more contender, and assert the system either queues it fairly and eventually serves it (progress, just delayed) or rejects it explicitly (a defined backpressure signal), rather than silently hanging forever.
3. Watchdog assertion. Wrap the scenario in a watchdog that asserts SOME forward-progress signal (a completed request, an incrementing counter, a released lock) occurs within a generous but bounded time window; a livelock is specifically the case where activity (retries, lock attempts) continues but the progress signal never fires, so "no timeout occurred" is not sufficient evidence of correctness, only "the progress signal fired" is.
4. Observability signals to add. Thread/goroutine dumps or stack traces on timeout (to see exactly where each participant is blocked), lock-acquisition and lock-wait-time metrics (a lock held far longer than its expected duration is a strong deadlock signal even before a full hang), and request-latency histograms (a livelock often shows as a spike in retries or a plateau in the completion rate, not literally zero throughput).

Worked example: a hard deadlock
A scripted two-service lock-ordering scenario with a watchdog assertion:

import threading

def test_lock_ordering_cycle_is_prevented_or_detected():
    lock1, lock2 = threading.Lock(), threading.Lock()
    progress = threading.Event()
    barrier = threading.Barrier(2)

    def service_a():
        with lock1:
            barrier.wait()          # deliberately synchronize both sides at the critical moment
            with lock2:
                progress.set()

    def service_b():
        with lock2:
            barrier.wait()
            with lock1:
                progress.set()

    # daemon=True is deliberate: this scenario deadlocks the two threads
    # PERMANENTLY (Python's plain threading.Lock has no built-in deadlock
    # detector, unlike a real database engine). Without daemon=True, the two
    # blocked threads never terminate, and the interpreter hangs forever at
    # exit waiting to join them, even after the watchdog assertion below has
    # already correctly failed and reported the deadlock.
    t1 = threading.Thread(target=service_a, daemon=True)
    t2 = threading.Thread(target=service_b, daemon=True)
    t1.start(); t2.start()

    made_progress = progress.wait(timeout=2.0)   # the watchdog
    assert made_progress, (
        "deadlock detected: neither side made progress within the watchdog timeout "
        "(this specific interleaving requires a consistent lock-ordering discipline, "
        "or a lock-timeout-and-retry strategy, to resolve)"
    )
    t1.join(timeout=0.1); t2.join(timeout=0.1)

This test is EXPECTED to fail against a naive implementation that acquires locks in inconsistent order (that is the point: it deliberately provokes the classic lock-ordering deadlock), and passing means the system under test has a real mitigation (consistent lock ordering, a timeout-and-retry, or a deadlock detector) in place.

Worked example: a livelock, the other half of this question
A hard deadlock and a livelock look identical from a bare "did it time out" check, so a suite that only has the example above has not actually tested for livelocks at all. Here two "polite" agents each back off the instant they detect contention, in perfect lockstep, so neither ever completes, yet both stay busy retrying: a deterministic, single-threaded simulation (rather than real OS threads, to avoid exactly the kind of real-timing race that makes ad hoc threaded livelock demos flaky) makes this reliably reproducible:

class Agent:
    def __init__(self, name, own_lock, other_lock):
        self.name = name
        self.own_lock = own_lock
        self.other_lock = other_lock
        self.holds_own = False
        self.retries = 0
        self.done = False

    def decide(self, snapshot):
        """Decide this round's action from a snapshot taken at the START of
        the round, so neither agent gets a sequencing advantage within the
        round -- a true simultaneous update, not a sequential one."""
        if self.done:
            return "noop"
        if not self.holds_own:
            return "take_own"
        if snapshot.get(self.other_lock) is None:
            return "take_other"
        return "back_off"

    def apply(self, action, held_by):
        if action == "take_own":
            held_by[self.own_lock] = self.name
            self.holds_own = True
        elif action == "take_other":
            held_by[self.other_lock] = self.name
            self.done = True
        elif action == "back_off":
            held_by[self.own_lock] = None
            self.holds_own = False
            self.retries += 1

def test_livelock_symmetric_polite_backoff_never_converges(max_steps=200):
    held_by = {"lock1": None, "lock2": None}
    a = Agent("a", "lock1", "lock2")
    b = Agent("b", "lock2", "lock1")
    for step in range(max_steps):
        if a.done or b.done:
            break
        snapshot = dict(held_by)
        action_a, action_b = a.decide(snapshot), b.decide(snapshot)
        a.apply(action_a, held_by)
        b.apply(action_b, held_by)   # both actions land together, from the same snapshot

    made_progress = a.done or b.done
    assert not made_progress, "expected the symmetric polite back-off pattern to livelock forever, not complete"
    assert a.retries > 10 and b.retries > 10, (
        f"expected substantial, ongoing retry activity from both sides (the defining signature of a "
        f"livelock, as opposed to a deadlock's total silence): got a={a.retries} b={b.retries}"
    )

Running this prints made_progress=False with a.retries=100 b.retries=100 after the full 200 simulated rounds: real, ongoing activity, and zero useful progress, which is exactly the property a watchdog based only on "did anything happen" would miss, and exactly why the observability signals in point 4 (a retry-rate or lock-attempt counter, not just a completion counter) matter for distinguishing this from a slow-but-healthy system.

Trade-offs and pitfalls

• A watchdog timeout that is too short will falsely flag a system that is merely slow (heavily contended but still making progress) as deadlocked; calibrate the timeout against the system's actual expected worst-case latency under contention, not an arbitrary round number.
• Deliberately scripted interleavings (using a barrier, as above) prove a SPECIFIC scenario is or isn't handled; they do not prove the absence of ALL possible deadlocks, which is a fundamentally harder problem. Pair scripted scenarios with production-side lock-wait-time monitoring as a complementary, ongoing detection mechanism.
• Distinguishing a livelock from legitimate-but-slow retry behavior requires a real progress signal, not just "activity is happening"; a system that keeps retrying a doomed operation forever looks identical to one that is slowly succeeding unless the test asserts on an actual completion signal, not just CPU or network activity.
• When a scenario is EXPECTED to deadlock or livelock permanently (as both examples above are, by design), make sure any threads used to model it are daemonized or otherwise bounded; a non-daemon thread that blocks forever will hang the test process itself at exit, turning a correctly-failing assertion into a CI job that never returns.

Direct answer
For stateful, long-running async workflows, the core testing move is to make the workflow's progress OBSERVABLE at every meaningful step (via checkpoints your test can poll) and to make the workflow's INPUTS deterministic (fixed IDs, fixed clocks, fixed random seeds), so a test can assert on a specific intermediate state rather than only the final one, and can reproduce a failure exactly when it finds one.

Structured elaboration
Break this into three concerns:

1. Orchestration and checkpoints. A long-running workflow (a saga, a multi-step approval process, an order-fulfillment pipeline) should emit a durable state transition at every step: PENDING -> VALIDATED -> CHARGED -> SHIPPED, each persisted with a timestamp and enough context to explain why it moved. Tests assert against these named states, not against a black-box "is it done yet." This also means a stuck workflow is diagnosable: the last recorded state tells you exactly which step never completed.
2. Determinism. Long-running workflows usually depend on time (SLAs, retry backoff, timeouts) and on identifiers generated during the run. For tests: inject a fake or controllable clock instead of time.now(), generate all IDs from a seeded generator so a failing run can be replayed exactly, and never depend on wall-clock sleeps to represent "time has passed" (advance a fake clock instead, where the workflow engine supports it, or use a real one only with generous, monitored timeouts).
3. Assertion strategy for flakiness. Async, multi-service workflows have many more places to be surprised by ordering than a single-service test. The two disciplines that most reduce flakiness are: (a) assert on the workflow's OWN recorded checkpoints (an internal source of truth) rather than inferring state indirectly from side effects in other services, and (b) use assert_eventually-style polling (see the eventual-consistency pattern) with a timeout keyed to the SLA the workflow is actually supposed to meet, not an arbitrary number.

Worked example
A workflow test harness sketch, showing checkpoint-driven assertions plus a seeded, deterministic ID generator:

import itertools
import time

class DeterministicIdGen:
    """Every test run with the same seed produces the same sequence of ids,
    so a failing run's ids can be grepped straight out of logs and replayed."""
    def __init__(self, seed=1):
        self._counter = itertools.count(seed)
    def next_id(self):
        return f"wf-{next(self._counter):06d}"

def run_workflow_and_wait_for(engine, checkpoint_store, workflow_id, expected_state, timeout_s=10.0):
    deadline = time.monotonic() + timeout_s
    last = None
    while time.monotonic() < deadline:
        last = checkpoint_store.latest_state(workflow_id)
        if last == expected_state:
            return last
        if last == "FAILED":
            raise AssertionError(f"workflow {workflow_id} failed before reaching {expected_state}")
        time.sleep(0.05)
    raise AssertionError(f"workflow {workflow_id} stuck at {last!r}, never reached {expected_state!r} within {timeout_s}s")

# usage
idgen = DeterministicIdGen(seed=42)
workflow_id = idgen.next_id()
engine.start(workflow_id, payload={...})
run_workflow_and_wait_for(engine, checkpoint_store, workflow_id, "VALIDATED")
run_workflow_and_wait_for(engine, checkpoint_store, workflow_id, "CHARGED")
final = run_workflow_and_wait_for(engine, checkpoint_store, workflow_id, "SHIPPED")
assert final == "SHIPPED"

Asserting each intermediate checkpoint in sequence (rather than only the terminal SHIPPED state) means a failure pinpoints the exact step that regressed, instead of forcing a manual investigation across every service in the pipeline.

Trade-offs and pitfalls

• Checkpoint-driven testing requires the workflow engine or the services themselves to actually persist intermediate state; if they don't, you either have to add that instrumentation (an investment worth making, since it also helps production debugging) or fall back to weaker, indirect assertions.
• A seeded ID generator only gives you reproducibility if every non-deterministic input (clock, randomness, network ordering) is also controlled; a workflow that is deterministic in its IDs but still races on message-processing order will still flake, just less often.
• Watch for tests that grow SLA-sized timeouts ("wait 30 seconds because the real SLA is 30 seconds") without also asserting the workflow reached each intermediate checkpoint at a reasonable pace; a workflow that is technically within SLA but spent 29 of those 30 seconds stuck can still hide a real regression from a single terminal-state assertion.