Test Levels and the Test Pyramid Questions

Contract tests exist to solve a problem neither end-to-end nor mocked-integration tests solve well: proving your service and the gateway agree on the SHAPE of their interaction, without needing the real gateway running and without silently drifting out of sync with what the gateway actually does.

Where contract tests sit, and what they uniquely solve

A contract test sits at the seam between integration and end-to-end: like an integration test, it runs fast and needs no live external dependency; like an end-to-end test, it is checking something about the REAL interface, not a hand-written assumption about it. Concretely, a contract test verifies that a captured or agreed-upon schema (which fields exist, their types, allowed values) for the gateway's request and response still holds, and critically, this contract can be verified independently on each side: your service checks it against the contract, and (in a full consumer-driven setup) the gateway's own team checks their real implementation against that same contract, so drift is caught the moment either side changes, at unit-test speed, without needing both systems running together in the same test.

Why (B), a hand-mocked integration test, is not equivalent

An integration test that mocks the gateway's responses is only as good as the assumptions baked into the mock: if the team writes the mock once and the gateway later changes its real response shape, the mocked test keeps passing forever, having silently drifted from reality. This is exactly the gap a contract test closes, since a contract test is verified against an actual agreed specification (or a captured real response), not an assumption the test author wrote down once and never revisited.

Evaluating the three approaches for a payment flow

• (A) End-to-end against the sandbox: highest realism (a real request genuinely reaches something resembling the gateway), but slowest and most fragile, since it depends on the sandbox's availability, network conditions, and test-account state, none of which your team controls.
• (B) Mocked integration tests: fast and reliable to run, but only as trustworthy as the mock's freshness, with the drift risk described above.
• (C) Contract tests: fast like (B), but without the drift risk, since the contract itself is the source of truth both sides verify against, rather than an assumption one side wrote down.

A worked, executable contract test

CONSUMER_CONTRACT = {
    "required_fields": {"transaction_id": str, "status": str, "amount_cents": int},
    "allowed_status_values": {"succeeded", "declined", "pending"},
}

def verify_contract(response_body, contract):
    violations = []
    for field, expected_type in contract["required_fields"].items():
        if field not in response_body:
            violations.append(f"missing required field: {field}")
        elif not isinstance(response_body[field], expected_type):
            violations.append(f"field '{field}' has wrong type")
    if "status" in response_body and response_body["status"] not in contract["allowed_status_values"]:
        violations.append(f"unexpected status value: {response_body['status']!r}")
    return violations

Run against the CURRENT provider shape ({"transaction_id": "txn_abc123", "status": "succeeded", "amount_cents": 4999}), this correctly returns zero violations. To prove the check is real and not a no-op, it was also run against two realistic breaking changes: a provider release that renames amount_cents to amount (returned violation: missing required field: amount_cents), and a provider that introduces a new status value the contract never declared, such as "requires_action" (returned violation: unexpected status value: 'requires_action'). Both breaking changes were caught, at zero network cost and in well under a millisecond, exactly the class of drift a stale hand-written mock would miss silently.

Recommendation: what runs on every pull request versus nightly

Run the contract test (C) on every pull request: it's fast, deterministic, and catches the highest-value class of bug (a broken assumption about the gateway's real shape) at the lowest cost. Run a smaller number of mocked integration tests (B) on every pull request too, for the request/response HANDLING logic the contract test doesn't cover (how your code reacts to a decline, a timeout, a malformed amount). Reserve the sandbox end-to-end test (A) for nightly, since it is the slowest and least reliable of the three, and its unique value, proving the real network path and real gateway behavior work together, does not need to be re-proven on every single commit.

Trade-offs and pitfalls

A contract test is only as good as how the contract itself stays current: without a process (ideally automated, via a shared contract broker - a service, such as a Pact Broker, where consumer teams publish the contracts they depend on and provider teams look up every contract they need to satisfy) for the gateway team to verify their real implementation against the same contract your consumer test uses, a "contract" test degrades back into the same drift risk as a hand-mocked test, just with extra ceremony. The value of contract testing comes specifically from BOTH sides verifying against a shared source of truth, not from the format of the test itself.

Empirically validating a pyramid ratio means treating it as a genuine hypothesis, not just an assertion, and the choice of WHAT metric you measure changes whether that validation is even practically feasible.

Experiment design: control versus experiment groups

Since you can't run two versions of the same team simultaneously, the practical design is a staggered rollout: apply the current ratio (control) to one set of comparable services or feature teams, and the proposed 70/20/10 ratio (experiment) to a matched set of comparable services or teams, matched as closely as possible on size, domain complexity, and current release cadence, since an unmatched comparison would confound the ratio's effect with pre-existing differences between the groups.

Metrics, duration, and required sample size, computed two ways

Option A: a binary per-release "escaped defect" metric. Suppose your baseline escaped-defect rate is 8% of releases, and you want to detect whether the new ratio meaningfully reduces it to 4%. In plain language before the numbers: alpha is the false-positive risk you're willing to accept (5% here, meaning a 5% chance of concluding the new ratio helped when it actually didn't), power is the chance of correctly detecting a real effect if one truly exists (80% here), and Cohen's d (used below in Option B) is a standardized way to measure how big the gap between two groups is relative to how spread out the underlying data is. A standard two-proportion power calculation (alpha=0.05, power=0.80) gives:

from scipy import stats
import math

def sample_size_two_proportions(p1, p2, alpha=0.05, power=0.8):
    z_alpha = stats.norm.ppf(1 - alpha / 2)
    z_beta = stats.norm.ppf(power)
    p_bar = (p1 + p2) / 2
    numerator = (z_alpha * math.sqrt(2 * p_bar * (1 - p_bar)) +
                 z_beta * math.sqrt(p1 * (1 - p1) + p2 * (1 - p2))) ** 2
    return math.ceil(numerator / (p1 - p2) ** 2)

n = sample_size_two_proportions(0.08, 0.04)

Executed result: n = 553 releases per group.

Rather than trusting that formula blindly, validate it with a Monte Carlo simulation that draws releases as random pass/fail outcomes at the two true rates and runs the actual two-proportion z-test at n=553, tallying how often it correctly rejects the null:

import numpy as np

rng = np.random.default_rng(12345)

def simulate_power(n, p1, p2, alpha=0.05, trials=2000):
    rejections = 0
    for _ in range(trials):
        x1 = rng.binomial(n, p1)
        x2 = rng.binomial(n, p2)
        phat1, phat2 = x1 / n, x2 / n
        p_pool = (x1 + x2) / (2 * n)
        se = math.sqrt(p_pool * (1 - p_pool) * (2 / n))
        z = (phat1 - phat2) / se
        if abs(z) > stats.norm.ppf(1 - alpha / 2):
            rejections += 1
    return rejections / trials

empirical_power = simulate_power(553, 0.08, 0.04)

Executed result: empirical power = 0.8145 over 2,000 trials (seed 12345), consistent with the target of 0.80 and validating the formula's answer rather than trusting it blindly. 553 releases per group is not practically achievable for most teams within any reasonable timeframe, which is itself an important, honest finding: a binary per-release metric is usually the wrong choice for this experiment.

Option B: a continuous, higher-frequency metric. Using weekly escaped-defect COUNT instead of a binary per-release outcome (baseline mean 3.0/week, target mean 1.8/week, standard deviation 2.0, a standardized effect size of Cohen's d = 0.6), the same normal-approximation approach applied to a continuous outcome gives:

def sample_size_continuous(d, alpha=0.05, power=0.8):
    z_alpha = stats.norm.ppf(1 - alpha / 2)
    z_beta = stats.norm.ppf(power)
    n = 2 * ((z_alpha + z_beta) ** 2) / (d ** 2)
    return math.ceil(n)

n_b = sample_size_continuous(0.6)

Executed result: n = 44 weeks per group, roughly a year total once both groups run concurrently over the same calendar period (see confounding-variable handling below), a dramatically more feasible design than Option A purely because of the metric choice, not the underlying effect size.

Success criteria

Pre-register the specific metric (weekly escaped-defect count, per Option B) and the specific improvement threshold (a reduction from a mean of 3.0 to 1.8 per week or better) BEFORE the experiment starts, along with the significance threshold (p < 0.05) and the practical-significance bar (the observed reduction must also be large enough to justify the ratio change's real cost, not just statistically distinguishable from zero).

Confounding variables

The biggest confounds in a real organization are: team composition changes during the measurement window (a team gaining or losing senior engineers independent of the ratio change), product complexity changes (a team shipping a harder feature set during the experiment than during the baseline period), and seasonal effects (release cadence and defect rates both often shift around major company events or holidays). Mitigate by choosing matched comparison groups from teams with stable composition over the measurement window, running both groups over the SAME calendar period rather than sequentially (so seasonal effects hit both equally), and tracking a secondary complexity metric (such as story points shipped) to confirm the two groups' workload stayed comparable throughout.

Trade-offs and pitfalls

The core lesson from the computation above generalizes: teams that try to empirically validate a testing-strategy change using a binary, rare-event, per-release metric are very often choosing an infeasible measurement design without realizing it, since a several-hundred-releases-per-group requirement is invisible until you actually run the power calculation. Doing that calculation FIRST, before committing to an experiment design, is what separates a real empirical validation from an experiment that will never reach a conclusive answer.

An 80%-UI-test regression suite is an inverted pyramid: the CI cost and flakiness live disproportionately at the most expensive, least precise level. The goal of a migration plan here is not "delete the UI tests," it is "prove the same business coverage more cheaply, then retire the UI test only once its replacement is proven equivalent."

1. Inventory

Catalog every UI test by what it actually verifies, not by its name: for each test, identify the underlying business assertion (for example, "a discount code reduces the order total correctly") separately from the UI mechanics used to exercise it (clicking through a cart page). Many UI tests will turn out to duplicate the same handful of business assertions through slightly different click paths, which is valuable information for step 2.

2. Selection criteria for migration candidates

Prioritize migrating a UI test to the API level when: (a) its business assertion does not depend on rendering, layout, or client-side interaction behavior itself, meaning the same assertion can be verified by calling the API directly; (b) it is one of several UI tests covering the same underlying business rule, since only one of them needs to stay at the UI level to prove the flow renders correctly, while the rest can move down; (c) it is currently a source of flakiness (timing-dependent, brittle selectors), since those are exactly the tests whose UI framing is adding risk without adding proportional confidence. Leave at the UI level anything whose actual subject IS the rendering or interaction behavior itself (does the button visibly disable during submission, does a validation message appear in the right place).

3. Incremental rollout strategy

Migrate in small batches grouped by business area (checkout, account management), running the new API-level test and the old UI test IN PARALLEL for one full release cycle before retiring the UI test, so you have a real comparison window rather than trusting the migration on faith. Start with the batch identified as most duplicative and most flaky in the inventory, since that batch gives the fastest CI-time win with the least coverage risk.

4. Metrics to track parity

Track, per migrated batch: the number of distinct production defects each UI test has caught historically (from incident postmortems or bug trackers) against whether the new API-level test would have caught the same defects if replayed against the historical bug; overall CI wall-clock time before and after; and flakiness rate (failures that resolve on rerun with no code change) before and after. A drop in caught-defect equivalence for a batch is the signal to keep more of that batch's UI coverage rather than fully retiring it.

5. Risk mitigation during the transition

Never retire a UI test until its replacement has run in parallel for a full cycle with no coverage gap identified; keep a small, deliberately curated UI layer for the handful of assertions that are genuinely about rendering and interaction, since no amount of API-level testing can verify those; and treat the migration as reversible, keeping the retired UI tests in version control (not deleted) for one additional cycle in case a gap surfaces late.

Trade-offs and pitfalls

The main pitfall is treating "80% UI tests" as inherently wrong without checking what those tests actually verify: if a genuinely large share of your business coverage requires rendering and interaction assertions (a highly visual, interaction-heavy product), a smaller UI share than 80% might still be too aggressive a cut. The inventory step exists precisely to avoid migrating tests whose real subject the API level cannot see.

Five test types map to three CI/CD stages, based on how much confidence each buys versus how much time it costs.

Where each type executes, and why

To place the top two rows precisely: a component test differs from a unit test by including a piece's real in-process collaborators instead of mocking everything, and differs from an integration test by still faking anything external like a database or network call.

The underlying trade-off, made explicit

Unit and component tests buy fast, precise confidence about logic, which is why they are the safe types to let block every merge: a false positive is rare and a true positive is almost always worth stopping the merge for. Integration and UI tests buy broader, more realistic confidence, but at meaningfully higher cost and with real risk of flakiness producing false positives, so only a small, carefully curated slice of them should be allowed to block a merge; the rest should run on a slower cadence where a failure gets investigated without holding up unrelated work. Performance tests answer a different question entirely (capacity, not correctness) and are noisy enough commit-to-commit that gating a merge on them directly would produce too many false alarms; they belong in a monitored, threshold-based alerting flow instead.

Trade-offs and pitfalls

The main pitfall is over-blocking: putting the full UI or performance suite in the merge-blocking path "to be safe" reliably backfires, because the resulting slow, occasionally-flaky gate trains developers to rerun blindly or bypass it, which defeats the entire purpose of having a gate. The discipline is choosing a SMALL, high-confidence subset for the blocking path and trusting the rest of the suite, running on a faster feedback loop than "never," to catch what the blocking subset misses.

A unit test exercises a single function or class in complete isolation: every dependency is faked, stubbed, or simply absent, so the test runs in microseconds and its failure points at exactly one piece of logic. An integration test exercises how two or more real components work together, most often your code against a real (or near-real) database, queue, or external service, so it catches wiring and serialization bugs a unit test cannot see. An end-to-end test drives the system the way a real client would, through its actual entry point (an HTTP call, a UI click), with nothing faked, so it is the only level that proves the whole assembled system actually works.

What each level covers, with two concrete examples per level

The QA-engineer angle on unit tests is collaborative, not just "who writes them": developers usually author the unit tests since they know the implementation, but QA should read them during review to spot missing edge cases the implementer didn't think of, and QA is often the one who notices a bug that unit tests theoretically should have caught but didn't (a coverage gap, not a process failure). "System testing" is a related but distinct idea: it validates the whole assembled system as one unit against requirements, similar in spirit to end-to-end testing, but typically owned by QA and run just before release with attention to environment parity and realistic test data, whereas end-to-end testing is often owned by whoever automates the user-facing flow and runs continuously.

The concrete tools differ by stack but the pattern holds everywhere: JUnit or pytest for the unit layer, pytest combined with testcontainers (spinning up a real, disposable database or service in a container) for the integration layer, and Selenium or Playwright for the end-to-end layer, with the same mock-vs-real-service decision applying at the integration boundary regardless of which tools you pick: mock a dependency when you're testing YOUR handling logic, use the real (or containerized) dependency when you're testing that the wiring itself is correct. For a payments microservice specifically, this maps onto where each level runs in the deployment pipeline: unit tests run locally on every save and in CI on every commit; integration tests run in CI against a containerized database and a sandboxed payment gateway; end-to-end tests run in a staging environment before a production deploy, and a small smoke subset may re-run immediately after reaching production to confirm the live deploy itself is healthy. This progression, more tests locally and in CI, fewer in staging, fewer still in production, is also how the pyramid should guide the ALLOCATION of engineering effort: invest the majority of new test-writing time at the level closest to the developer's own commit, not because the higher levels don't matter, but because that's where a fixed hour of effort buys the most coverage per dollar of CI time and developer attention.

Worked example: the same business rule, tested three ways

The clearest way to see the boundary is to test the identical rule at all three levels and watch what each level can and cannot catch.

# calculate_discount is pure logic: no I/O, so it belongs at the unit level.
def calculate_discount(price: float, tier: str) -> float:
    if price < 0:
        raise ValueError("price must be non-negative")
    rate = {"standard": 0.0, "silver": 0.05, "gold": 0.15}.get(tier)
    if rate is None:
        raise ValueError(f"unknown tier: {tier}")
    return round(price * (1 - rate), 2)

# UNIT TEST: no database, no network. Executed directly.
def test_calculate_discount_unit():
    assert calculate_discount(100, "standard") == 100.0
    assert calculate_discount(100, "silver") == 95.0
    assert calculate_discount(100, "gold") == 85.0

Executed output: UNIT level: 4/4 assertions passed (pure function, no I/O, <1ms) (all four, including the negative-price ValueError case).

import sqlite3

class OrderRepository:
    def __init__(self, conn):
        self.conn = conn
        self.conn.execute(
            "CREATE TABLE IF NOT EXISTS orders (id INTEGER PRIMARY KEY, price REAL, tier TEXT, total REAL)"
        )
    def save(self, price, tier):
        total = calculate_discount(price, tier)
        cur = self.conn.execute(
            "INSERT INTO orders (price, tier, total) VALUES (?, ?, ?)", (price, tier, total)
        )
        self.conn.commit()
        return cur.lastrowid

# INTEGRATION TEST: a REAL SQLite database, catching serialization/wiring the unit test cannot see.
def test_order_repository_integration():
    conn = sqlite3.connect(":memory:")
    repo = OrderRepository(conn)
    order_id = repo.save(200, "gold")
    row = conn.execute("SELECT total FROM orders WHERE id = ?", (order_id,)).fetchone()
    assert row[0] == 170.0

Executed output: INTEGRATION level: repository round-trip through real SQLite passed: {'id': 1, 'price': 200.0, 'tier': 'gold', 'total': 170.0}.

For the end-to-end level, the same repository was wired behind a real HTTP handler and hit with an actual socket-level POST followed by a GET, using Python's built-in http.server and urllib.request, no mocking anywhere in the path:

import json, threading, urllib.request, urllib.error
from http.server import BaseHTTPRequestHandler, HTTPServer

class Handler(BaseHTTPRequestHandler):
    def log_message(self, format, *args):
        pass
    def do_POST(self):
        length = int(self.headers.get("Content-Length", 0))
        body = json.loads(self.rfile.read(length))
        try:
            order_id = repo.save(body["price"], body["tier"])
            total = calculate_discount(body["price"], body["tier"])
        except ValueError as e:
            self.send_response(400)
            self.end_headers()
            self.wfile.write(json.dumps({"error": str(e)}).encode())
            return
        self.send_response(201)
        self.end_headers()
        self.wfile.write(json.dumps({"id": order_id, "total": total}).encode())
    def do_GET(self):
        order_id = int(self.path.rsplit("/", 1)[-1])
        row = conn.execute("SELECT id, total FROM orders WHERE id = ?", (order_id,)).fetchone()
        self.send_response(200)
        self.end_headers()
        self.wfile.write(json.dumps({"id": row[0], "total": row[1]}).encode())

# E2E TEST: real HTTP socket, real server thread, nothing faked.
def test_checkout_end_to_end():
    server = HTTPServer(("127.0.0.1", 0), Handler)
    port = server.server_address[1]
    threading.Thread(target=server.serve_forever, daemon=True).start()
    req = urllib.request.Request(
        f"http://127.0.0.1:{port}/orders",
        data=json.dumps({"price": 50, "tier": "silver"}).encode(),
        headers={"Content-Type": "application/json"}, method="POST",
    )
    created = json.loads(urllib.request.urlopen(req).read())
    fetched = json.loads(urllib.request.urlopen(f"http://127.0.0.1:{port}/orders/{created['id']}").read())
    assert fetched == {"id": created["id"], "total": 47.5}
    bad = urllib.request.Request(
        f"http://127.0.0.1:{port}/orders",
        data=json.dumps({"price": 50, "tier": "platinum"}).encode(),
        headers={"Content-Type": "application/json"}, method="POST",
    )
    try:
        urllib.request.urlopen(bad)
        assert False, "expected HTTPError"
    except urllib.error.HTTPError as e:
        assert e.code == 400
    server.shutdown()

Executed output: E2E level (valid order): POST+GET over real HTTP socket returned {'id': 1, 'total': 47.5} and E2E level (invalid tier): POST over real HTTP socket returned status 400. That is precisely the trade-off: the unit test told us the discount math is right in under a millisecond; the end-to-end test told us the whole pipe, JSON serialization, routing, and the network stack included, actually delivers that correct math to a real client, at the cost of running a live server and a real socket for the one test.

Trade-offs and pitfalls

The pyramid shape follows directly from this example: you want most of your assertions at the level that is cheapest to run and most precise about what broke, which is the unit level, and you want just enough integration and end-to-end coverage to prove the pieces are wired correctly, because that proof is expensive and comes with flakiness risk (a slow database, a stalled network call, a race in the test server) that a pure function can never have. A common pitfall is over-mocking at the unit level: if you replace so many collaborators that the "unit" test no longer exercises real logic, it stops earning its speed advantage and becomes a maintenance burden that breaks on every refactor without ever catching a real bug. The opposite pitfall is under-investing in unit tests and leaning on end-to-end tests to catch logic bugs, which works but means every logic bug takes minutes instead of milliseconds to surface, and a flaky end-to-end suite starts to be ignored by the team, which is worse than no suite at all.