Test Levels and the Test Pyramid Questions

Five test types map to three CI/CD stages, based on how much confidence each buys versus how much time it costs.

Where each type executes, and why

To place the top two rows precisely: a component test differs from a unit test by including a piece's real in-process collaborators instead of mocking everything, and differs from an integration test by still faking anything external like a database or network call.

The underlying trade-off, made explicit

Unit and component tests buy fast, precise confidence about logic, which is why they are the safe types to let block every merge: a false positive is rare and a true positive is almost always worth stopping the merge for. Integration and UI tests buy broader, more realistic confidence, but at meaningfully higher cost and with real risk of flakiness producing false positives, so only a small, carefully curated slice of them should be allowed to block a merge; the rest should run on a slower cadence where a failure gets investigated without holding up unrelated work. Performance tests answer a different question entirely (capacity, not correctness) and are noisy enough commit-to-commit that gating a merge on them directly would produce too many false alarms; they belong in a monitored, threshold-based alerting flow instead.

Trade-offs and pitfalls

The main pitfall is over-blocking: putting the full UI or performance suite in the merge-blocking path "to be safe" reliably backfires, because the resulting slow, occasionally-flaky gate trains developers to rerun blindly or bypass it, which defeats the entire purpose of having a gate. The discipline is choosing a SMALL, high-confidence subset for the blocking path and trusting the rest of the suite, running on a faster feedback loop than "never," to catch what the blocking subset misses.

With limited CI minutes, the right rule is: run what's cheap enough to not slow anyone down on every commit, defer what's expensive but low-risk-of-being-wrong-right-now to nightly, and gate what's slow but release-critical to the release pipeline.

What runs where, by level

• Every commit / pull request: the full unit-test suite (should be fast enough, seconds to low minutes, that nobody thinks twice about running it) plus a small, curated smoke set of the highest-value integration and end-to-end tests covering your one or two most critical journeys (login, checkout). Rationale: a developer needs fast feedback on the logic they just touched, and a small smoke layer catches the worst wiring regressions before they even reach a shared branch.
• Nightly: the full integration suite and the full end-to-end suite, run against a shared or staging-like environment. Rationale: these are too slow to run on every commit without destroying developer velocity, but running them nightly still catches integration regressions within a day, which is an acceptable latency for bugs that are rarer than pure logic bugs.
• Pre-release / release gate: a final full end-to-end pass plus any slow, environment-heavy tests (performance baselines, cross-browser matrices) that are too expensive to run even nightly. Rationale: this is the last checkpoint before real users are affected, so it's worth paying maximum test cost here even though it's not worth paying it on every commit.

A proposed ratio and CI-time budget for a typical SaaS application

A reasonable starting ratio is roughly 70% unit, 20% integration, 10% end-to-end by test count, with a target CI-time budget per layer on a pull request of: unit tests under 2 minutes total, the curated smoke slice of integration/end-to-end tests under 5 minutes total, so the whole PR check stays under about 7 minutes, comfortably inside the roughly ten-minute mark where most teams report developers start context-switching away and waiting on results loses its value.

Tactics to enforce those CI-time targets without silently losing coverage

• A fast smoke suite: a small, deliberately hand-picked subset of integration/end-to-end tests covering your highest-risk journeys, run on every PR instead of the full suite, so PR feedback stays fast while still catching the worst regressions immediately.
• Test selection: running only the tests whose code path plausibly touches the files changed in a given commit, rather than the entire suite, to cut PR time without reducing what eventually gets run before release.
• Test-impact analysis: a more precise, automated version of test selection that uses a dependency map (which tests exercise which source files, including transitive dependencies) to compute the minimal correct test set for a given diff, rather than relying on manual tagging.

Applied together, these tactics let a team keep the FULL suite's coverage intact (nothing is deleted, nothing stops running entirely) while making sure only the necessary fraction of it runs on the expensive, time-constrained PR path.

Trade-offs and pitfalls

The common failure mode is letting the "nightly" bucket become a dumping ground: tests get moved there because they're slow or flaky, not because nightly is genuinely the right cadence for their risk level, and regressions caught only nightly then sit unnoticed for a full day while more commits pile on top, making the eventual fix harder to isolate. Treat the nightly tier as a deliberate risk-and-cost decision per test, not a place to hide problems.

A test suite should contain far more unit tests than end-to-end tests because the two levels trade off cost against realism in opposite directions, and the cheap-and-precise end of that trade dominates for the vast majority of the bugs you actually need to catch.

Five reasons, each with a concrete illustration

1. Cost of running. A unit test for a pure function runs in well under a millisecond and needs no external process. An end-to-end test for the same behavior needs a running server, a database, and often a browser or HTTP client, and commonly takes seconds. Run a suite of 2,000 tests: at unit-test speed that finishes in well under a minute; at end-to-end speed the same count could take hours, which no team can afford to run on every commit.
2. Feedback speed. A developer who breaks a function wants to know within seconds, while still holding the change in their head. A unit test gives that immediately; an end-to-end test, queued behind environment provisioning and a full pipeline run, might report the failure twenty minutes later, by which point the developer has moved on to something else and the context-switch cost to fix it is much higher.
3. Maintainability. A unit test breaks only when the specific function's contract changes. An end-to-end test walks through many screens or endpoints, so it is coupled to all of them at once: a single unrelated UI change (a renamed button, a reordered field) can break dozens of end-to-end tests that were never testing that button in the first place, creating maintenance work with zero corresponding increase in confidence.
4. Diagnostic precision (technical). When a unit test fails, the failure message names the exact function and assertion that broke. When an end-to-end test fails, you know only that somewhere across dozens of components something is wrong, and someone has to spend real time narrowing that down, effectively re-deriving the information a unit test would have handed you directly.
5. Organizational scaling. As a codebase and team grow, the number of possible logic paths grows roughly with the code, while the number of realistic whole-system journeys grows much more slowly (most new logic is a variation inside an existing journey, not a brand-new one). Unit tests scale naturally with that logic growth; trying to scale end-to-end tests at the same rate produces a suite that is mostly redundant coverage of the same few journeys, wasting CI time without adding proportional confidence.

Trade-offs and pitfalls

None of this means end-to-end tests are dispensable: they are the only level that proves the pieces are actually wired together correctly for a real user, which is exactly the class of bug the other four reasons cannot catch. The pitfall is treating "more unit tests" as license to skip end-to-end coverage of your critical paths entirely; the healthy pattern is a small, curated set of end-to-end tests covering the handful of journeys that matter most (checkout, login), backed by a much larger base of fast unit tests covering the underlying logic.

A unit test exercises a single function or class in complete isolation: every dependency is faked, stubbed, or simply absent, so the test runs in microseconds and its failure points at exactly one piece of logic. An integration test exercises how two or more real components work together, most often your code against a real (or near-real) database, queue, or external service, so it catches wiring and serialization bugs a unit test cannot see. An end-to-end test drives the system the way a real client would, through its actual entry point (an HTTP call, a UI click), with nothing faked, so it is the only level that proves the whole assembled system actually works.

What each level covers, with two concrete examples per level

The QA-engineer angle on unit tests is collaborative, not just "who writes them": developers usually author the unit tests since they know the implementation, but QA should read them during review to spot missing edge cases the implementer didn't think of, and QA is often the one who notices a bug that unit tests theoretically should have caught but didn't (a coverage gap, not a process failure). "System testing" is a related but distinct idea: it validates the whole assembled system as one unit against requirements, similar in spirit to end-to-end testing, but typically owned by QA and run just before release with attention to environment parity and realistic test data, whereas end-to-end testing is often owned by whoever automates the user-facing flow and runs continuously.

The concrete tools differ by stack but the pattern holds everywhere: JUnit or pytest for the unit layer, pytest combined with testcontainers (spinning up a real, disposable database or service in a container) for the integration layer, and Selenium or Playwright for the end-to-end layer, with the same mock-vs-real-service decision applying at the integration boundary regardless of which tools you pick: mock a dependency when you're testing YOUR handling logic, use the real (or containerized) dependency when you're testing that the wiring itself is correct. For a payments microservice specifically, this maps onto where each level runs in the deployment pipeline: unit tests run locally on every save and in CI on every commit; integration tests run in CI against a containerized database and a sandboxed payment gateway; end-to-end tests run in a staging environment before a production deploy, and a small smoke subset may re-run immediately after reaching production to confirm the live deploy itself is healthy. This progression, more tests locally and in CI, fewer in staging, fewer still in production, is also how the pyramid should guide the ALLOCATION of engineering effort: invest the majority of new test-writing time at the level closest to the developer's own commit, not because the higher levels don't matter, but because that's where a fixed hour of effort buys the most coverage per dollar of CI time and developer attention.

Worked example: the same business rule, tested three ways

The clearest way to see the boundary is to test the identical rule at all three levels and watch what each level can and cannot catch.

# calculate_discount is pure logic: no I/O, so it belongs at the unit level.
def calculate_discount(price: float, tier: str) -> float:
    if price < 0:
        raise ValueError("price must be non-negative")
    rate = {"standard": 0.0, "silver": 0.05, "gold": 0.15}.get(tier)
    if rate is None:
        raise ValueError(f"unknown tier: {tier}")
    return round(price * (1 - rate), 2)

# UNIT TEST: no database, no network. Executed directly.
def test_calculate_discount_unit():
    assert calculate_discount(100, "standard") == 100.0
    assert calculate_discount(100, "silver") == 95.0
    assert calculate_discount(100, "gold") == 85.0

Executed output: UNIT level: 4/4 assertions passed (pure function, no I/O, <1ms) (all four, including the negative-price ValueError case).

import sqlite3

class OrderRepository:
    def __init__(self, conn):
        self.conn = conn
        self.conn.execute(
            "CREATE TABLE IF NOT EXISTS orders (id INTEGER PRIMARY KEY, price REAL, tier TEXT, total REAL)"
        )
    def save(self, price, tier):
        total = calculate_discount(price, tier)
        cur = self.conn.execute(
            "INSERT INTO orders (price, tier, total) VALUES (?, ?, ?)", (price, tier, total)
        )
        self.conn.commit()
        return cur.lastrowid

# INTEGRATION TEST: a REAL SQLite database, catching serialization/wiring the unit test cannot see.
def test_order_repository_integration():
    conn = sqlite3.connect(":memory:")
    repo = OrderRepository(conn)
    order_id = repo.save(200, "gold")
    row = conn.execute("SELECT total FROM orders WHERE id = ?", (order_id,)).fetchone()
    assert row[0] == 170.0

Executed output: INTEGRATION level: repository round-trip through real SQLite passed: {'id': 1, 'price': 200.0, 'tier': 'gold', 'total': 170.0}.

For the end-to-end level, the same repository was wired behind a real HTTP handler and hit with an actual socket-level POST followed by a GET, using Python's built-in http.server and urllib.request, no mocking anywhere in the path:

import json, threading, urllib.request, urllib.error
from http.server import BaseHTTPRequestHandler, HTTPServer

class Handler(BaseHTTPRequestHandler):
    def log_message(self, format, *args):
        pass
    def do_POST(self):
        length = int(self.headers.get("Content-Length", 0))
        body = json.loads(self.rfile.read(length))
        try:
            order_id = repo.save(body["price"], body["tier"])
            total = calculate_discount(body["price"], body["tier"])
        except ValueError as e:
            self.send_response(400)
            self.end_headers()
            self.wfile.write(json.dumps({"error": str(e)}).encode())
            return
        self.send_response(201)
        self.end_headers()
        self.wfile.write(json.dumps({"id": order_id, "total": total}).encode())
    def do_GET(self):
        order_id = int(self.path.rsplit("/", 1)[-1])
        row = conn.execute("SELECT id, total FROM orders WHERE id = ?", (order_id,)).fetchone()
        self.send_response(200)
        self.end_headers()
        self.wfile.write(json.dumps({"id": row[0], "total": row[1]}).encode())

# E2E TEST: real HTTP socket, real server thread, nothing faked.
def test_checkout_end_to_end():
    server = HTTPServer(("127.0.0.1", 0), Handler)
    port = server.server_address[1]
    threading.Thread(target=server.serve_forever, daemon=True).start()
    req = urllib.request.Request(
        f"http://127.0.0.1:{port}/orders",
        data=json.dumps({"price": 50, "tier": "silver"}).encode(),
        headers={"Content-Type": "application/json"}, method="POST",
    )
    created = json.loads(urllib.request.urlopen(req).read())
    fetched = json.loads(urllib.request.urlopen(f"http://127.0.0.1:{port}/orders/{created['id']}").read())
    assert fetched == {"id": created["id"], "total": 47.5}
    bad = urllib.request.Request(
        f"http://127.0.0.1:{port}/orders",
        data=json.dumps({"price": 50, "tier": "platinum"}).encode(),
        headers={"Content-Type": "application/json"}, method="POST",
    )
    try:
        urllib.request.urlopen(bad)
        assert False, "expected HTTPError"
    except urllib.error.HTTPError as e:
        assert e.code == 400
    server.shutdown()

Executed output: E2E level (valid order): POST+GET over real HTTP socket returned {'id': 1, 'total': 47.5} and E2E level (invalid tier): POST over real HTTP socket returned status 400. That is precisely the trade-off: the unit test told us the discount math is right in under a millisecond; the end-to-end test told us the whole pipe, JSON serialization, routing, and the network stack included, actually delivers that correct math to a real client, at the cost of running a live server and a real socket for the one test.

Trade-offs and pitfalls

The pyramid shape follows directly from this example: you want most of your assertions at the level that is cheapest to run and most precise about what broke, which is the unit level, and you want just enough integration and end-to-end coverage to prove the pieces are wired correctly, because that proof is expensive and comes with flakiness risk (a slow database, a stalled network call, a race in the test server) that a pure function can never have. A common pitfall is over-mocking at the unit level: if you replace so many collaborators that the "unit" test no longer exercises real logic, it stops earning its speed advantage and becomes a maintenance burden that breaks on every refactor without ever catching a real bug. The opposite pitfall is under-investing in unit tests and leaning on end-to-end tests to catch logic bugs, which works but means every logic bug takes minutes instead of milliseconds to surface, and a flaky end-to-end suite starts to be ignored by the team, which is worse than no suite at all.

A "70/20/10" (or similarly-shaped) ratio is a useful DEFAULT, not a target to hit for its own sake: it encodes the assumption that most of a typical system's defect risk lives in logic a unit test can isolate cheaply, a smaller amount lives in how components wire together, and only a small remainder needs the expensive proof that the whole assembled system works.

What the ratio assumes

1. Most bugs are logic bugs, not integration bugs. If your system's core complexity is business logic (pricing rules, calculations, state machines), this holds well, and a heavy unit-test base pays off directly.
2. Integration points are relatively few and stable. The ratio assumes there aren't so many service-to-service or component-to-component seams that integration testing alone would need to be a much larger share to give adequate confidence.
3. The team can afford SOME slow, broad tests, but not many. The "10%" isn't zero: it assumes a small curated end-to-end layer is enough to catch whole-system wiring problems, which is only true if your riskiest journeys are few in number.
4. Cost scales the way the model assumes. The whole justification for weighting the base so heavily rests on unit tests being drastically cheaper than integration and end-to-end tests; if that cost gap narrows (fast, hermetic integration tests via lightweight containers, for instance - hermetic meaning self-contained: no real network calls or shared external state, so the same test run always gets the same result), the "right" ratio shifts too.

When to deviate, and why

• A thin orchestration service whose logic is almost entirely "call service A, then call service B" has very little unit-testable logic of its own; here the risk genuinely concentrates at the integration boundary, so a heavier integration-test share (closer to something like 40/50/10) reflects reality better than forcing a 70% unit-test floor onto code that barely has any unit-testable branches.
• A frontend-heavy, interaction-driven product where most of the risk is "does clicking through this actually work for a user" may reasonably lean toward more integration-style component tests (a component test renders one UI component together with its real child components but fakes the network or backend, which is what separates it from a unit test, which isolates everything) - closer to the testing-trophy shape (an alternative to the pyramid that keeps a small unit-test base but makes these broader, more realistic tests the largest layer) - than a strict 70/20/10 pyramid, because the thing most likely to break is how components interact on screen, not isolated pure functions.
• A system with very few, very high-stakes end-to-end journeys (payment settlement, safety-critical workflows) may justify a larger-than-10% end-to-end share for those specific journeys, even while the rest of the system keeps the standard ratio, because the cost of an undetected wiring bug there is disproportionately high.

Trade-offs and pitfalls

The most common misuse of this heuristic is treating the numbers as a scorecard: chasing "70% unit tests" by writing large numbers of low-value unit tests for trivial getters, while under-investing in the harder work of a few well-chosen integration and end-to-end tests for the journeys that actually carry business risk. The ratio should be a lagging description of where your test investment naturally lands once you've tested the RIGHT things at each level, not a quota to satisfy directly.