API and Contract Testing Questions

Direct answer

Consumer-driven contract testing (CDC) verifies that a service (the provider) still honors the expectations of the services that call it (the consumers) without spinning up the whole system. Each consumer records what it expects from the provider as a machine-readable "contract." The provider then replays that contract against its own code, in its own pipeline, to confirm it still satisfies it. Nobody has to run both services at once.

Structured elaboration

Why this exists. A full integration test where every dependent service is actually running catches real bugs, but it's slow, flaky, and gets worse as the number of services grows. Contract testing answers a narrower, cheaper question: does the provider still return what this specific consumer needs? That narrower question is fast to check and can run on every commit.

Division of responsibility.

• The consumer writes a test against a mock of the provider that asserts what it needs (a field, a status code, a shape). Running that test generates the contract as a byproduct, so the contract is always in sync with what the consumer's code actually depends on, not with what someone remembers to document.
• The provider takes that contract and replays it against its real implementation: it sets up whatever preconditions the contract's scenario needs (a "provider state," like "a user with ID 42 exists"), serves the request the contract describes, and checks the response matches.
• CI wiring on both sides is what makes this useful: the consumer's pipeline publishes new contracts when they change, and the provider's pipeline pulls and verifies against the latest contracts before it's allowed to deploy. A provider that would break a consumer fails its own build, not the consumer's.

Typical infrastructure. A contract-testing tool (Pact is the best known) plus a broker: a small service that stores contracts, tracks which provider version has verified which consumer version, and can answer "is it safe for me to deploy this provider version given what's currently in production?"

Benefits. Fast (no real network calls, no shared environment), and it fails on the side that actually caused the problem, since the provider's own CI is where a broken contract shows up.

Trade-offs and pitfalls

Contract tests are not a substitute for integration or end-to-end tests. They confirm the pieces individually honor their agreements; they don't prove the whole system behaves correctly when wired together, and they say nothing about things a contract can't easily express, like timing, ordering across multiple calls, or business-logic correctness. Common early-adoption pitfalls: contracts that are too strict (asserting on fields the consumer doesn't actually use, so unrelated provider changes fail verification for no reason) and contracts that are too loose (missing something the consumer genuinely depends on, so a real break slips through). A useful discipline is to only assert on what the consumer's code actually reads, nothing more.

Consumer-driven vs. provider-driven. In the consumer-driven model described above, the consumer's own tests generate the contract, so it's automatically grounded in real usage. A provider-driven model runs the other way: the provider maintains its own suite of contract tests representing what it believes its consumers need, without a consumer necessarily being involved. That's easier to set up when consumers can't or won't participate, but the contract can silently drift out of sync with what a consumer actually needs, since nothing forces it to be re-derived from real consumer code. Consumer-driven is preferable whenever you can get consumer teams to participate; provider-driven is a fallback when a consumer is external, unresponsive, or a system you don't control.

Direct answer

Generating the signature is a few lines: HMAC (hash-based message authentication code) SHA-256 over the timestamp and raw body, using the shared secret. Verifying it correctly on the receiver side is the part that actually matters, and it has three genuinely separate checks: the signature is valid, the timestamp is fresh, and this exact request hasn't been processed before.

Structured elaboration

Signing (sender side, what the Postman pre-request script does). The vendor's convention here is standard: concatenate the timestamp and the raw request body with a separator, then HMAC-SHA256 that combined string with the shared secret, and send the result as a header alongside the timestamp itself. The receiver has to sign the exact same bytes the same way to check it, so the pre-request script and the receiver's verification logic must agree on the signed-content format down to the separator character.

Verifying (receiver side), three checks, each catching a different failure:

1. Timestamp freshness. Reject anything outside a tolerance window (a few minutes is typical) before doing anything else. This is what limits how long a captured request stays replayable even before the replay check runs, and it's cheap to check first so an obviously stale request doesn't cost a cryptographic comparison.
2. Signature validation. Recompute the expected signature from the secret, the timestamp, and the body, and compare it to the header using a constant-time comparison, never a plain string equality, which can leak timing information about how many leading bytes matched and make the secret guessable byte by byte over many attempts.
3. Replay protection. Even a validly-signed, fresh request should only be accepted once. Track signatures (or a vendor-supplied event ID, if one exists) already processed, and reject a repeat.

Worked example

The pre-request script that generates the signature, using CryptoJS, the crypto library Postman's sandbox exposes as a global:

const secret = 'test-webhook-secret-shared-with-vendor';
const timestamp = Math.floor(Date.now() / 1000).toString();
const body = pm.request.body.raw;
const signedContent = timestamp + '.' + body;
const signature = CryptoJS.HmacSHA256(signedContent, secret).toString(CryptoJS.enc.Hex);
pm.request.headers.upsert({ key: 'X-Webhook-Timestamp', value: timestamp });
pm.request.headers.upsert({ key: 'X-Webhook-Signature', value: signature });

And the receiver-side verification, in Python:

import hmac, hashlib, time

def verify_webhook(secret, timestamp, body, signature_header, seen_signatures, max_age_seconds=300):
    ts = int(timestamp)
    if abs(time.time() - ts) > max_age_seconds:
        return False, "stale timestamp"
    signed_content = timestamp.encode() + b"." + body
    expected = hmac.new(secret.encode(), signed_content, hashlib.sha256).hexdigest()
    if not hmac.compare_digest(expected, signature_header):
        return False, "signature mismatch"
    if signature_header in seen_signatures:
        return False, "replay detected"
    seen_signatures.add(signature_header)
    return True, "accepted"

Verified end to end with a minimal receiver, not just the isolated function above. Wiring verify_webhook into a small Flask endpoint and driving it with Flask's test client (real HTTP request/response objects, no mocking) confirms the three cases the tests need to cover:

from flask import Flask, request, jsonify

SECRET = "test-webhook-secret-shared-with-vendor"
seen_signatures = set()
app = Flask(__name__)

@app.route("/webhook", methods=["POST"])
def webhook():
    timestamp = request.headers.get("X-Webhook-Timestamp", "")
    signature = request.headers.get("X-Webhook-Signature", "")
    body = request.get_data()
    ok, reason = verify_webhook(SECRET, timestamp, body, signature, seen_signatures)
    if ok:
        return jsonify({"status": reason}), 200
    if reason == "replay detected":
        return jsonify({"error": reason}), 409
    return jsonify({"error": reason}), 400

Driving it with three real round trips through app.test_client():

fresh first-time request -> 200 {'status': 'accepted'}
replayed request          -> 409 {'error': 'replay detected'}
stale (1hr old) request   -> 400 {'error': 'stale timestamp'}

Exactly the three outcomes the acceptance criteria need: a fresh, correctly-signed request accepted, the identical request replayed and rejected as a duplicate, and a correctly-signed but hour-old request rejected as stale.

Trade-offs and pitfalls

A real gotcha found while building this, worth knowing before you hit it live. The obvious "modernization" of the pre-request script, replacing the bare CryptoJS global with const CryptoJS = require('crypto-js'), actually breaks in the current Postman sandbox: CryptoJS is already bound as a global, and redeclaring it throws SyntaxError: Identifier 'CryptoJS' has already been declared. The bare global form is deprecated (Postman's own console warns about it) but is still the one that actually works today; don't "fix" a deprecation warning by introducing a naming collision that breaks the script outright, verify a replacement actually runs before trusting a deprecation notice's suggested fix.

The signed-content format (timestamp, a separator, then the raw body, in that exact order and byte form) has to match the vendor's real convention exactly; a mismatch anywhere (a different separator, a parsed-and-re-serialized body instead of the raw bytes, a different byte encoding) makes every signature fail to verify even though the logic is otherwise correct, so the first thing to check against a real vendor's docs, not assume, is the exact signed-content construction.

Direct answer

A solid test-data strategy for API and service tests rests on four decisions made deliberately rather than by default: generate-on-the-fly versus fixed fixtures, synthetic versus production-derived data, how tests stay isolated from each other's data, and what actually needs cleanup versus what can be safely shared.

Structured elaboration

Generated on the fly vs. fixed fixtures. A fixture generated fresh for each test (a new user created at the start of the test, torn down at the end) gives strong isolation, no test can be affected by another test's leftover state, at the cost of some setup time per test. A fixed, shared fixture (a small set of known reference records seeded once) is faster to use repeatedly but risks one test's assumptions about that shared data becoming invalid if another test modifies it. The practical split: generate fresh data for anything a test needs to MUTATE, and use shared fixtures only for read-only reference data nothing is expected to change.

Synthetic vs. closer-to-production data. Purely synthetic data (templated names, generated emails) is simplest and carries no privacy risk, but can miss bugs that only show up with realistic value distributions or realistic correlations between fields. Production-derived data (masked or anonymized) is more realistic but requires real discipline around what's actually safe to use in a test environment, and is usually reserved for a smaller number of higher-fidelity tests (load testing, a final pre-release check) rather than the bulk of the day-to-day suite, where synthetic data's speed and simplicity matter more than its lower realism.

Isolation between tests. However the data is generated, tests running concurrently need to not collide: namespacing (a unique prefix or identifier per test run), using ephemeral per-test-run resources (a fresh database schema, a dedicated test namespace), or simply ensuring every generated record's identifiers are unique enough that two tests' data can coexist without either noticing the other.

Cleanup. Data a test creates should be removed when the test finishes, via the test's own teardown when things go well, and via a backstop (a scheduled sweep for anything matching a test-data naming pattern and older than some threshold) for the cases where a test crashes before its own teardown runs. Shared, read-only reference data doesn't need this kind of cleanup at all, that's part of why the mutate-vs-read-only split above matters: it determines which data even needs a cleanup story.

Trade-offs and pitfalls

The mistake that causes the most downstream pain is treating "test data strategy" as one uniform decision applied to everything, generate everything fresh, or share everything, rather than recognizing that different data has different needs. A shared reference fixture that's occasionally, accidentally mutated by a test that was supposed to only read it is one of the more confusing categories of flaky test to debug, since the failure shows up in a DIFFERENT, unrelated test that happened to run afterward and inherited the corrupted shared state, not in the test that actually caused the problem.

Direct answer

Below is a reusable HTTP GET function with retry and exponential backoff for transient server errors and network errors, with configurable attempt count and base delay.

Structured elaboration

The function distinguishes what's worth retrying (a timeout, a connection error, a 5xx that's likely transient) from what isn't (a 4xx, which means the request itself is wrong and retrying an unmodified request will just fail the same way again), and backs off exponentially between attempts so a struggling server isn't hit with an immediate retry storm.

Worked example

import time
import requests


def resilient_get(url, max_attempts=5, initial_backoff=0.5, backoff_factor=2,
                   retry_statuses=(500, 502, 503, 504)):
    last_exception = None
    delay = initial_backoff

    for attempt in range(1, max_attempts + 1):
        try:
            resp = requests.get(url, timeout=5)
            if resp.status_code not in retry_statuses:
                return resp  # success, or a non-retryable error (e.g. 4xx): return as-is
            last_exception = None
        except (requests.ConnectionError, requests.Timeout) as e:
            last_exception = e
            resp = None

        if attempt == max_attempts:
            if last_exception:
                raise last_exception
            return resp  # exhausted retries on a retryable status; return the last response

        time.sleep(delay)
        delay *= backoff_factor

    raise RuntimeError("unreachable")  # defensive; loop always returns or raises above

Thread-safety. As written, resilient_get has no shared mutable state at all: delay, attempt, and last_exception are all local to each call, so many test threads calling it concurrently don't interact with each other in any way. The one thing worth being deliberate about in concurrent use is the underlying requests session: this version uses the module-level requests.get, which creates a new connection per call and is safe under concurrency but doesn't reuse connections. If you switch to a shared requests.Session() for connection pooling (a reasonable optimization under high concurrency), the Session object itself needs to be either one per thread or explicitly documented as thread-safe for your use case, since requests.Session is not guaranteed thread-safe for concurrent use by requests' own documentation.

Verified with a fixture that fails twice with a 503 and then succeeds:

call_count = [0]

def flaky_get(url, timeout):
    call_count[0] += 1
    class FakeResp:
        status_code = 503 if call_count[0] <= 2 else 200
    return FakeResp()

requests.get = flaky_get  # monkeypatched into requests.get for this test

resp = resilient_get("http://fake/x", initial_backoff=0.01)
print(f"attempts made: {call_count[0]}")
print(f"final status_code: {resp.status_code}")

Running resilient_get against this fixture (with initial_backoff=0.01 to keep the test fast) returns a 200 after exactly 3 attempts, confirming the retry loop and the eventual-success path both work:

attempts made: 3
final status_code: 200

Trade-offs and pitfalls

A 4xx status code deliberately does NOT trigger a retry in this implementation: retrying an unmodified request that the server has already rejected as invalid wastes time and, in the worst case, can look like an attempted abuse pattern to the server (repeated requests to an endpoint that keeps rejecting them). If a caller genuinely wants to retry a 429 (rate-limited) specifically, that status needs to be added to retry_statuses deliberately, and ideally the delay should respect a Retry-After header if the server provides one, rather than blindly following the function's own generic backoff schedule.

Direct answer

Below is a pytest suite for a GET-by-id endpoint using a fixture to create the test data it needs, parametrized cases for an existing and a non-existing ID, and teardown that removes what it created.

Structured elaboration

The fixture is what makes this test independent of any pre-existing state: rather than assuming a user with a known ID already exists somewhere, it creates one at the start of the test and cleans it up afterward, so the test can run repeatedly, in any order, against a fresh environment.

Worked example

import pytest
import requests

BASE_URL = "http://localhost:5000"  # fixture server for this example


@pytest.fixture
def existing_user():
    resp = requests.post(f"{BASE_URL}/users", json={"name": "Ada", "email": "ada@example.com"})
    assert resp.status_code == 201
    user = resp.json()
    yield user
    requests.delete(f"{BASE_URL}/users/{user['id']}")  # teardown


@pytest.mark.parametrize("use_existing_id,expected_status", [
    (True, 200),
    (False, 404),
])
def test_get_user_by_id(existing_user, use_existing_id, expected_status):
    user_id = existing_user["id"] if use_existing_id else 999999
    resp = requests.get(f"{BASE_URL}/users/{user_id}")

    assert resp.status_code == expected_status, (
        f"expected {expected_status} for user_id={user_id}, got {resp.status_code}"
    )
    assert resp.elapsed.total_seconds() < 2, "response took unexpectedly long"

    if expected_status == 200:
        body = resp.json()
        assert body["id"] == user_id
        assert body["name"] == existing_user["name"]
        assert body["email"] == existing_user["email"]

Executed against a small local Flask fixture implementing /users create/get/delete with an in-memory store:

test_get_user_by_id[True-200] PASSED
test_get_user_by_id[False-404] PASSED
2 passed in 0.14s

Beyond the status code. For the 200 case, the test asserts the returned body actually reflects the SAME resource the fixture created, name and email matching, not just that SOME 200 body came back. This catches a bug class the status code alone would miss entirely: an endpoint that returns 200 with the wrong resource (an off-by-one in an internal lookup, for instance).

Trade-offs and pitfalls

The fixture's teardown running via yield (rather than a separate finalizer or manual cleanup call) means the DELETE happens even if the test body raises an exception partway through, which matters for keeping the test environment clean across a full CI run: a test that fails on an assertion shouldn't also leave orphaned data behind for the next test to trip over. It's worth explicitly testing that the teardown itself doesn't silently swallow a failure, if the DELETE call fails, that failure should be visible (in CI logs, if not as a hard test failure) rather than disappearing quietly.