Distributed Systems and Microservices Testing Questions

Direct answer
Encode the CRDT service's core guarantees (that all replicas converge to the same state given the same set of operations, regardless of delivery order) as explicit, checkable properties, generate randomized operation sequences and fault schedules (dropped messages, reordering, concurrent operations from multiple replicas) with a property-based testing library, and detect a violation inside an integration test by comparing replica states after settling rather than asserting a single hand-computed expected value.

Structured elaboration

1. Encoding the invariants as checkable properties. For a CRDT, the central property is: given any set of operations delivered to any two replicas in any order (as long as each replica eventually receives all of them), those two replicas must reach an identical state. This is checkable directly: apply the same multiset of operations to two model instances in two different orders and assert their resulting states are equal. A second, complementary property worth encoding: causally-related operations (an edit that logically depends on a prior edit) must never be observed "out of their causal order" at any replica, even though CONCURRENT (causally unrelated) operations may be observed in different orders at different replicas.
2. Generating randomized inputs and fault schedules. Use a property-based testing library (Hypothesis in Python, or an equivalent) to generate random sequences of operations across multiple simulated replicas, random delivery orders, and random dropped/duplicated/delayed deliveries, running each generated case against the properties above. Property-based tools also SHRINK a failing case automatically to the smallest example that still reproduces the violation, which is far more useful for debugging than a single large randomly-generated failure.
3. Capturing traces for evaluation. Record, for each generated case, the exact sequence of operations and the order they were delivered to each replica, so a failing case can be replayed exactly (property-based tools typically do this automatically via a seed, but re-log it independently as well, since a library-internal replay format can change between versions).
4. Detecting a violation inside an integration test. After all in-flight operations are delivered to all replicas (simulate quiescence explicitly, don't just wait a fixed time), compare every replica's state pairwise; any mismatch is a convergence violation and should immediately fail with the full operation trace attached.

Worked example

from hypothesis import given, strategies as st

def apply_in_order(ops, order):
    state = CRDTCounter()
    for i in order:
        state.apply(ops[i])
    return state.value()

@given(
    ops=st.lists(st.sampled_from(["inc", "dec"]), min_size=1, max_size=10),
    seed=st.integers(),
)
def test_crdt_counter_converges_regardless_of_delivery_order(ops, seed):
    import random
    order_a = list(range(len(ops)))
    order_b = list(range(len(ops)))
    random.Random(seed).shuffle(order_b)

    result_a = apply_in_order(ops, order_a)
    result_b = apply_in_order(ops, order_b)

    assert result_a == result_b, (
        f"replicas diverged: order_a -> {result_a}, order_b (seed={seed}) -> {result_b}, ops={ops}"
    )

A property-based framework running this will generate dozens of operation lists and delivery-order permutations automatically, and on any failure will shrink ops down to the smallest sequence that still reproduces the divergence, which is the actionable bug report a developer needs.

Trade-offs and pitfalls

• Property-based testing is only as good as the properties you encode; "converges to the same value" is the right property for a simple counter CRDT, but a richer CRDT (a collaborative text or JSON structure) needs a property that also checks the converged VALUE is semantically sensible, not merely that all replicas happen to agree on some value.
• Randomized generation finds bugs proportional to how much of the operation and fault space it actually explores; bound the generator's complexity (number of operations, number of replicas) to what your CI budget allows, and run a larger, slower sweep less frequently (nightly) rather than trying to maximize exploration on every commit.
• A property test that passes for hundreds of generated cases is still not a proof; treat it as strong empirical evidence, not a substitute for reasoning through the CRDT's merge function by hand for the specific operation types your system actually supports.

Direct answer
Orchestrate startup with explicit dependency ordering plus application-level readiness checks (not just process-alive or port-open checks), and treat "reports healthy" and "can actually serve a real request correctly" as two different things to verify, since a service can report healthy (its HTTP server is up) well before its own dependencies (a database connection pool, a cache warm-up, a schema migration) are actually ready to serve real traffic.

Structured elaboration

1. Orchestrating the environment. Whether using docker-compose or Kubernetes, express the dependency graph explicitly (service X depends on database Y and cache Z) so the orchestrator starts things in a sensible order and, more importantly, so the TEST SUITE knows which readiness checks to wait on before running anything against a given service.
2. Deciding when a dependency is truly ready, not just started. A container reporting "running" only means its process launched; a database container can be running for several seconds before it actually accepts connections, and an application container can be running before it finishes its own startup migrations or cache warm-up. The right readiness signal is an APPLICATION-LEVEL health check the service itself exposes (a /healthz endpoint that only returns 200 once its own dependencies are confirmed reachable and its own startup tasks are complete), not merely "the process is running" or "the port accepts a TCP connection."
3. Avoiding false positives from a service that reports healthy too early. A common bug is a health-check endpoint that returns 200 as soon as the HTTP server itself starts, before the service has actually verified its OWN downstream dependencies are reachable. Guard against this by making the health check itself verify real downstream connectivity (a lightweight ping to the database, a cache connectivity check) rather than a hardcoded 200, and by having the test harness's wait-loop perform a SEMANTIC check in addition to the reported health status wherever feasible (for example, issuing one real, cheap request the service can only answer correctly if it's truly ready, not just relying on the health endpoint's own self-report).
4. The wait strategy itself. Poll each service's readiness endpoint on an interval, with an overall timeout; only proceed to run the actual test suite once every dependency in the graph reports ready, and fail loudly (naming which service never became ready) rather than proceeding and getting a confusing cascade of unrelated test failures.

Worked example
A readiness-wait helper distinguishing "reports healthy" from "can actually serve a request," verified with a fake orchestration layer:

import time

def wait_for_ready(services, timeout_s=60, interval_s=1.0, semantic_check=None):
    deadline = time.monotonic() + timeout_s
    not_ready = set(services)
    while time.monotonic() < deadline and not_ready:
        for name in list(not_ready):
            if health_check(name) and (semantic_check is None or semantic_check(name)):
                not_ready.discard(name)
        if not_ready:
            time.sleep(interval_s)
    if not_ready:
        raise AssertionError(f"services never became ready within {timeout_s}s: {sorted(not_ready)}")

def test_environment_is_actually_ready_before_running_suite():
    services = ["postgres", "service-a", "service-b"]

    def semantic_check(name):
        # a health endpoint reporting 200 is necessary but not sufficient;
        # also confirm the service can answer one real, cheap query
        if name == "service-a":
            return service_a_client.ping_dependencies() is True
        return True

    wait_for_ready(services, timeout_s=30, semantic_check=semantic_check)

The semantic_check hook here is deliberately separate from health_check, so a service that reports 200 prematurely (before it has actually verified its own dependencies) still gets caught by the additional, real dependency-ping check.

Trade-offs and pitfalls

• Relying solely on a health endpoint that the service itself controls is only as trustworthy as that endpoint's own implementation; a health check that always returns 200 the moment the HTTP server binds its port is common, easy to write by accident, and defeats the whole purpose of readiness gating.
• A semantic check (an actual cheap request) adds a small amount of latency and coupling to the harness, but is often the only thing that reliably catches the "reports healthy but isn't really ready" class of bug; use it at least for the services most prone to this pattern (anything with its own downstream dependencies to warm up).
• An overall timeout that's too short makes the suite flaky on a slower CI runner; one that's too long means a genuinely broken environment wastes significant CI time before failing. Calibrate against measured real startup times, and fail with the specific service name(s) still not ready, never a generic "setup timed out."

Direct answer
Combine synthetic traffic generation that deliberately hits every named edge case (oversized files, partial uploads, simulated network partitions between regions, non-ASCII locale encodings, and artificially-induced GC pauses) with mocks for the pieces that are unsafe or impractical to exercise for real, per-region integration tests run against real regional deployments, a small end-to-end tier confirming the whole pipeline works together, and canary checks that specifically assert on these edge cases, not just generic health, so a regression in one language's implementation or one region's specific configuration is caught before it reaches all three regions.

Structured elaboration

1. Synthetic traffic covering each named edge case, independently. File-size limits: generate uploads at, just under, and just over the configured limit, asserting the correct accept/reject boundary. Partial uploads: deliberately truncate a connection mid-upload and assert the service correctly detects and rejects (or resumes, if resumable uploads are supported) rather than silently accepting a corrupt file. Locale encodings: include filenames and metadata with non-ASCII characters (accented characters, right-to-left scripts, emoji) and assert correct storage and retrieval across both the Java and Node implementations, since encoding handling is exactly the kind of behavior that can subtly differ between language runtimes. GC pauses: this is harder to synthesize directly; instead, add a test that artificially induces GC pressure (allocate and hold a large amount of memory during the test) and assert the service's request-handling behavior (timeouts, retries) degrades gracefully rather than silently corrupting an in-flight upload.
2. Network partitions across regions. Use proxy- or mesh-level fault injection between region-to-region calls to simulate one region becoming unreachable from another, and assert the system's documented behavior for that case (does the affected region degrade gracefully, queue for later reconciliation, or reject outright) rather than an undefined hang.
3. Mocks for the pieces that are hard or unsafe to exercise for real. Mock the storage backend (or a downstream virus-scanning/transformation service) to inject deterministic failures for the file-size and partial-upload edge cases on demand, rather than depending on real storage misbehaving on command; and mock any third-party locale/encoding library at the boundary so a test can isolate whether an encoding bug lives in the Java implementation, the Node implementation, or a shared downstream dependency both call into, instead of only being able to say "it broke somewhere."
4. Per-region integration tests, not just one canonical region. Run the same test suite against each of the three real regional deployments (not only one "reference" region), since a region-specific configuration difference (a different storage backend version, a different network topology) can behave differently even with identical application code.
5. A small end-to-end tier on top of the rest. Reserve a handful of true end-to-end tests, exercising the full upload-to-availability flow (upload, validation, transformation, storage, retrieval) through a real client against a real regional deployment, for the properties that only emerge when the whole pipeline runs together, such as confirming a file with a non-ASCII name and a near-limit size is actually retrievable end-to-end in each region. Keep this tier deliberately small given its cost and slowness; it sits above the synthetic-traffic and per-region integration layers to catch integration gaps between stages, not to duplicate what those layers already cover.
6. Canary-rollout checks tied to these specific edge cases. As part of a canary deployment, run a lightweight, fast subset of these edge-case tests (not the full suite) against the canary specifically, and gate promotion on them passing, so a regression introduced by a new deploy is caught in one region's canary before it reaches the other two regions and the rest of that region's traffic.
7. Automating all of this without excessive cost. Reserve the full, expensive edge-case matrix (all combinations of file size, encoding, region, language) for a scheduled, less-frequent run (nightly or pre-release), and run a smaller, representative, fast subset on every commit and every canary promotion, to keep everyday CI feedback fast while still getting full coverage on a regular cadence.

Worked example

import pytest

@pytest.mark.parametrize("filename", [
    "plain-ascii.txt",
    "café-résumé.txt",         # accented Latin characters
    "文件.txt",                 # non-Latin script
    "\U0001F600-emoji.txt",    # emoji in the filename
])
def test_upload_handles_locale_encodings_across_both_language_implementations(filename, java_impl_client, node_impl_client):
    for client in (java_impl_client, node_impl_client):
        response = client.upload(filename=filename, content=b"test-data")
        assert response.status == "accepted"
        retrieved = client.get_metadata(response.upload_id)
        assert retrieved.filename == filename, (
            f"{client.impl_name}: filename round-trip mismatch for {filename!r}, got {retrieved.filename!r}"
        )

def test_partition_between_region_a_and_region_b_degrades_per_spec(fault_injector):
    fault_injector.partition("region-a", "region-b", duration_s=10)
    response = region_a_client.upload(sample_file())
    assert response.status == "accepted_pending_replication", (
        "an upload during a cross-region partition should be accepted locally and queued for "
        "replication once healed, per the documented degraded-mode behavior"
    )

def test_storage_backend_failure_is_mocked_for_partial_upload_case(mocked_storage_backend):
    # mocking the storage backend lets this failure be triggered on demand, rather than
    # depending on real storage misbehaving at the exact moment the test runs
    mocked_storage_backend.fail_next_write(error="connection_reset")
    response = region_a_client.upload(sample_file())
    assert response.status == "rejected_incomplete_upload"

Trade-offs and pitfalls

• Testing every combination of edge case, region, and language implementation is combinatorially expensive; be deliberate about which combinations are actually likely to differ (language-specific behavior mostly matters for encoding/serialization edge cases, region-specific behavior mostly matters for network/partition scenarios) rather than testing the full cross-product on every commit.
• Artificially-induced GC pauses are inherently harder to make deterministic than most fault types here; treat GC-pressure tests as a probabilistic signal (run several times, look for a pattern) rather than a single deterministic pass/fail assertion.
• A partition test that only checks the LOCAL region's immediate response can miss a bug in the eventual reconciliation once the partition heals; pair the immediate-response assertion with a follow-up check, after healing, that replication actually completed correctly.
• Mocking the storage backend makes failure injection reliable, but a mock that doesn't match the real backend's actual failure shapes (error codes, partial-write semantics) can pass against the mock while a real integration issue goes uncaught; periodically validate the mock's failure behavior against the real backend's documented contract, applying the same discipline to every other mocked third-party dependency in the suite, such as the locale/encoding library used for the encoding tests above.

Direct answer
Split the test strategy along the same line the system itself splits on: the transfer's ledger-entry correctness (must be strongly correct, tested with strict invariant checks and no tolerance for a wrong number) versus the settlement projection (allowed to lag, tested with eventual-consistency techniques), and specifically design adversarial scenarios (concurrent transfers, a mid-transfer failover, a network partition during a transfer) whose PASS condition is "no double-spend and no lost funds," not merely "no exception was thrown."

Structured elaboration
A payments system with this shape has two correctness domains that need different test techniques:

1. Strongly-correct transfer core. Test with a test oracle that tracks the INVARIANT directly: total funds in the system before a batch of concurrent operations must equal total funds after, regardless of how many transfers ran, how many failed partway, or how many retried. Concretely: run N concurrent transfer attempts (some deliberately racing on the same source account) against a fresh, known starting balance, then assert sum(all account balances) == starting_total no matter which subset of transfers actually succeeded. This is a stronger and more useful check than asserting any one transfer's specific outcome, because it catches double-spend and lost-funds bugs regardless of which individual transfer triggered them.
2. Eventually-consistent settlement. Test settlement the same way you would test any eventual-consistency path (poll-with-timeout on the settlement projection, assert it eventually reflects the ledger), but ADDITIONALLY assert the ledger itself never depends on settlement having happened; a slow or failed settlement update must never be able to corrupt or roll back an already-committed ledger entry.
3. Adversarial fault scenarios, each with the invariant check as the pass condition rather than a specific expected trace: concurrent transfers on the same account (race for the same funds), a failover of the primary ledger service mid-transfer (does a client retry after failover risk applying the transfer twice), and a network partition isolating the ledger from the settlement pipeline (does settlement silently drop the update, or correctly queue and catch up).

Worked example
A conservation-of-funds oracle run against concurrent transfer attempts, the strongest and most general check for this domain:

import concurrent.futures

def test_no_double_spend_under_concurrent_transfers():
    ledger = Ledger(initial_balances={"acct-A": 100, "acct-B": 100, "acct-C": 100})
    starting_total = sum(ledger.balances().values())

    # Ten concurrent attempts to move funds, several racing on the same
    # source account with retries enabled (simulating client-side retry on
    # ambiguous failure), using idempotency keys per logical transfer.
    def attempt_transfer(i):
        key = f"transfer-{i % 4}"  # some keys reused on purpose: simulated retries
        try:
            ledger.transfer("acct-A", "acct-B", amount=10, idempotency_key=key)
        except InsufficientFundsError:
            pass  # a legitimate outcome when funds run out; not a bug

    with concurrent.futures.ThreadPoolExecutor(max_workers=10) as pool:
        list(pool.map(attempt_transfer, range(10)))

    ending_total = sum(ledger.balances().values())
    assert ending_total == starting_total, (
        f"funds were created or destroyed: started with {starting_total}, ended with {ending_total}"
    )
    for acct, balance in ledger.balances().items():
        assert balance >= 0, f"{acct} went negative: {balance}"

The test never asserts a specific final balance for a specific account (that would depend on which racing attempts happened to win, which is not the property under test); it asserts the domain invariant (conservation of funds, no negative balances) that must hold regardless of the race's outcome.

Trade-offs and pitfalls

• A conservation-of-funds oracle is powerful precisely because it doesn't need to predict WHICH transfer wins a race, but it also can't tell you which specific bug caused a violation; pair it with detailed logging of every attempted transfer so a failure is debuggable, not just detectable.
• It is tempting to test settlement eventual-consistency and ledger strong-consistency as if they were the same concern; keeping them as clearly separate test suites, with separate tolerance for flakiness (zero tolerance on the ledger invariant, a bounded window on settlement), prevents a genuinely broken ledger invariant from being masked by "it's probably still catching up."
• Failover-mid-transfer testing requires an environment that can actually simulate a mid-operation failover (not just a full outage before or after); if your test infrastructure cannot inject a failure at that precision, the conservation-of-funds oracle test above still catches most double-spend classes even without a dedicated failover harness, which is a reasonable fallback if true chaos-level fault injection is not yet available.

Direct answer
Combine three layers: deterministic unit tests against the lock service's own state-machine logic (no network involved, pure logic), property or linearizability tests that generate many concurrent lock-acquisition sequences and check the results are consistent with SOME valid serial order, and fault injection (partitions, forced leader re-election) layered on top of the property tests so the harness catches ordering bugs that only appear when a leader change happens mid-lock-acquisition.

Structured elaboration

1. Deterministic unit tests. Test the lock service's core decision logic (given a lease request, an existing lease, and a current term/epoch, what should happen) as pure functions wherever possible, with no real network or real etcd involved. These catch straightforward logic bugs cheaply and fast, and they are the foundation the more expensive tests below build on.
2. Linearizability / property tests. Generate many random sequences of concurrent lock acquire/release/renew calls against a real (or realistically faithful) instance of the lock service, record the real-time order operations were issued and completed in, and check the recorded history against a LINEARIZABILITY CHECKER: does there exist some valid sequential ordering of these operations, consistent with real-time constraints, that a correct mutual-exclusion lock could have produced? This is a fundamentally different and stronger check than "did two clients ever both think they held the lock at the same time in a single run," because it can catch subtler violations across many randomized runs rather than relying on one lucky (or unlucky) manual scenario.
3. Fault injection layered on top. Run the SAME property-based concurrent-acquisition generator while also injecting network partitions and forced leader re-elections at random points. The interesting bugs live specifically at the leader-transition boundary: does a lease granted by the OLD leader remain valid and exclusive after a new leader takes over, or can a naive implementation allow the new leader to grant the same lease to a second client before it learns about the old leader's grant?
4. Subtle ordering/timing bugs. These typically surface as: two clients briefly believing they both hold the lock (a genuine mutual-exclusion violation), or a lease expiring and being re-granted to someone else while the original holder is still unaware and still acting as if it holds the lock (a fencing-token problem). Test explicitly for the second case by having the "stale" holder attempt a guarded write using its (now-expired) lease token, and assert the guarded resource rejects it.

Worked example
A minimal linearizability-style checker applied to a recorded history of lock operations (illustrating the technique on a small case, not a production-grade checker):

def is_linearizable_mutex_history(history):
    """history: list of (client, op, start_time, end_time, result) for acquire/release.
    Returns False if any two ACQUIRE operations, both reporting success, have
    overlapping [start_time, end_time] intervals with no release between them."""
    successful_acquires = [h for h in history if h.op == "acquire" and h.result == "granted"]
    for i, a in enumerate(successful_acquires):
        for b in successful_acquires[i + 1:]:
            overlap = a.start_time < b.end_time and b.start_time < a.end_time
            if overlap:
                return False, (a, b)
    return True, None

def test_no_two_clients_hold_lock_concurrently_across_leader_change():
    history = []
    cluster = FakeEtcdLockCluster(replicas=3)
    clients = [LockClient(cluster) for _ in range(5)]

    def worker(client, idx):
        result = client.acquire("resource-1", ttl_s=1.0)
        history.append(LockEvent(client=idx, op="acquire", start_time=result.start, end_time=result.end, result=result.status))
        if result.status == "granted":
            client.release("resource-1")

    cluster.schedule_leader_reelection(after_ms=20)
    run_concurrently([lambda i=i: worker(clients[i], i) for i in range(5)])

    ok, violation = is_linearizable_mutex_history(history)
    assert ok, f"mutual exclusion violated: {violation}"

Trade-offs and pitfalls

• A real linearizability checker (such as Jepsen's Knossos, or an equivalent) is considerably more rigorous than the illustrative overlap-check above, which only catches the simplest violation shape; for a production-grade harness, use an established checker rather than hand-rolling one, since the general problem is subtle and easy to get wrong in exactly the way you are trying to test for.
• Randomized concurrent property tests need MANY runs (often hundreds or thousands of seeds) to have a meaningful chance of hitting a rare timing window; budget CI time accordingly, or run a large randomized sweep nightly and a smaller fixed regression set on every commit.
• Testing only "do two clients ever both hold the lock" misses the fencing-token class of bug (a client that believes it still holds an expired lease and acts on stale authority); always add an explicit test where a lease holder is forced to lag past its TTL and then attempts a guarded action.