Test Automation Scripting Questions

Direct answer

The weakness is that the test asserts an immediate count delta (len(after) == len(before) + 1) right after the POST returns, with no allowance for the possibility that the created user is not yet visible in a subsequent GET; if the system has any eventual-consistency lag between accepting the write and reflecting it in reads (an async index, a read replica, a cache), this test can fail even when the system is working exactly as designed, or worse, can PASS for the wrong reason if some unrelated user happened to be created by another test at the same moment. Fix it by polling for the SPECIFIC created resource to appear, with a bounded timeout, instead of asserting an instantaneous count.

Structured elaboration

Two separate problems live in that one assertion. First, timing: api_client.post(...) returning does not guarantee the write is immediately visible to a subsequent GET; many real systems accept a write synchronously but propagate it to whatever the read path queries (a search index, a cache, a read replica) asynchronously, which is a completely normal and correct design, not a bug, so a test that assumes synchronous visibility is testing an assumption the system never promised. Second, specificity: counting total users before and after is fragile even ignoring timing, since it says nothing about WHICH user appeared; if the count happens to be off by one for a completely unrelated reason (test isolation failure, a concurrent test running against the same environment), the assertion could pass or fail for reasons that have nothing to do with whether bob@example.com specifically was created.

The fix addresses both: poll the GET endpoint for the SPECIFIC email just created (not just any count increase) for a bounded time, and only fail if it never appears within that window, which correctly tolerates real, bounded eventual-consistency lag while still catching a genuine failure to create the user at all.

Worked example

import time

def test_create_user_robust(api_client, target_email='bob@example.com', timeout=1.0, poll=0.02):
    before = api_client.get('/users').json()
    api_client.post('/users', json={'email': target_email})
    deadline = time.time() + timeout
    while time.time() < deadline:
        after = api_client.get('/users').json()
        if any(u.get('email') == target_email for u in after):
            assert len(after) >= len(before) + 1
            return True
        time.sleep(poll)
    raise AssertionError(f"{target_email} never became visible within {timeout}s")


import threading

class FakeAPIClient:
    """Simulates a system with a 0.05s asynchronous write-visibility delay: POST is
    accepted immediately, but the new record only appears in GET results after a short
    async propagation delay (like an async index or read replica)."""
    def __init__(self, delay=0.05):
        self.delay = delay
        self._visible = []
        self._lock = threading.Lock()

    def get(self, path):
        class R:
            def __init__(self, data): self._data = data
            def json(self): return self._data
        with self._lock:
            return R(list(self._visible))

    def post(self, path, json):
        def make_visible():
            time.sleep(self.delay)
            with self._lock:
                self._visible.append(json)
        threading.Thread(target=make_visible).start()

def weak_test(api_client, target_email):
    before = api_client.get('/users').json()
    api_client.post('/users', json={'email': target_email})
    after = api_client.get('/users').json()
    assert len(after) == len(before) + 1

N = 5
weak_failures = 0
for i in range(N):
    client = FakeAPIClient(delay=0.05)
    try:
        weak_test(client, target_email=f"bob{i}@example.com")
    except AssertionError:
        weak_failures += 1
    time.sleep(0.1)  # let the background thread finish before the next run

robust_failures = 0
for i in range(N):
    client = FakeAPIClient(delay=0.05)
    try:
        test_create_user_robust(client, target_email=f"bob{i}@example.com")
    except AssertionError:
        robust_failures += 1

print(f"weak (count-based, no wait) assertion failed {weak_failures}/{N} runs due to the visibility race")
print(f"robust (poll-for-visibility) assertion failed {robust_failures}/{N} runs")

Running it:

weak (count-based, no wait) assertion failed 5/5 runs due to the visibility race
robust (poll-for-visibility) assertion failed 0/5 runs

The original weak assertion failed on every single run because it never allows for the visibility lag; the fixed version, polling for the specific user with a timeout, passed on every run.

Trade-offs and pitfalls

The most common mistake in "fixing" this is simply adding a fixed time.sleep(1) before the GET, which trades a fast, flaky test for a slow, still-technically-flaky test (it works until the real lag exceeds whatever sleep duration was guessed, and it wastes a full second on every run even when the system responds in milliseconds); polling with a bounded timeout gets the correctness of waiting without paying the worst-case delay every single time. A second pitfall is polling forever with no timeout at all, which turns a genuine backend outage (the user is NEVER created) into a hung test rather than a clear, timely failure; the timeout is what preserves the test's ability to catch a real bug instead of just tolerating a slow one.

Direct answer

Two test cases execute: test_email_validator[a@b.com-True] and test_email_validator[bad-email-False]. The db fixture is scope='module', so setup_database() runs exactly once before the first test in the module needs it, and teardown_database() runs exactly once after the last test in the module finishes, not once per parametrized case. If teardown raises, pytest reports it as a separate error attached to the fixture's finalization, and it does not retroactively fail the tests that already passed using that fixture.

Structured elaboration

The two mechanics stack independently: parametrize controls how many times the test FUNCTION body runs (two, one per row), and fixture scope controls how many times the DEPENDENCY setup runs relative to that (once for the whole module, regardless of how many parametrized calls consume it). A common misreading is to assume scope='module' means "once per module invocation of the test," conflating it with the test count; it actually means "once for the lifetime of the module's collection," so both parametrized rows share the exact same fixture instance and see any state it set up.

On teardown-raises-an-exception: pytest treats a fixture's teardown as running during test session teardown for that fixture's scope, generator-based (yield) fixtures resume after the yield at that point. If teardown_database() raises, pytest reports it as its own failure entry (historically shown as an "error" associated with the fixture's finalizer, distinct from a test failure), and any test that already completed and reported PASS keeps that result: teardown failing does not retroactively flip an already-recorded PASS to a FAIL. It does mean the module's teardown did not complete cleanly, which is exactly the scenario that matters in a real suite that reuses the same database resource for other modules.

Worked example

import pytest

calls = []

def setup_database():
    calls.append('setup')

def teardown_database():
    calls.append('teardown')

def validate_email(email):
    return '@' in email and '.' in email.split('@')[-1]

@pytest.fixture(scope='module')
def db():
    setup_database()
    yield
    teardown_database()

@pytest.mark.parametrize('email,valid', [
    ('a@b.com', True),
    ('bad-email', False)
])
def test_email_validator(db, email, valid):
    assert validate_email(email) == valid
    calls.append(f'test-ran:{email}')

def test_calls_trace():
    print("call trace so far:", calls)
    assert calls.count('setup') == 1
    assert calls.count('teardown') == 0  # module fixture has not torn down yet; it fires at end of module

Running it:

$ pytest test_fixture_scope.py -v -s
test_fixture_scope.py::test_email_validator[a@b.com-True] PASSED
test_fixture_scope.py::test_email_validator[bad-email-False] PASSED
test_fixture_scope.py::test_calls_trace call trace so far: ['setup', 'test-ran:a@b.com', 'test-ran:bad-email']
PASSED

3 passed in 0.02s

The trace confirms setup fired exactly once before both parametrized cases ran, and teardown had not fired by the time the third test (added to observe the trace) executed, since the module fixture tears down only after the last test in the module completes.

Trade-offs and pitfalls

The most common misdiagnosis here is answering "two setups, two teardowns, one per parametrized case" by confusing parametrize with fixture instantiation count; they are orthogonal mechanisms. A second pitfall specific to module scope: because the fixture instance is shared across the two rows, any accidental mutation of shared state by one row's test body (for example, if the fixture returned a live database connection and one test left a row inserted) can leak into the next parametrized case, which is a realistic source of order-dependent flakiness. Finally, a teardown exception being reported separately from the test results means a suite can show "2 passed" while ending in a non-zero overall result once you count the fixture error, which surprises engineers used to reading only the pass/fail tally and skipping the errors section of the report.

Direct answer

A test fixture is the setup an automated test needs before it can run and the cleanup it needs afterward, whether that is a database connection, a browser instance, or a mock server; most test frameworks let you scope a fixture to run once per test, once per class of tests, or once per whole test suite/module, and choosing the right scope is a real trade-off between speed and isolation.

Structured elaboration

Common resources fixtures prepare and clean up: a database connection or transaction (opened before, rolled back or closed after), a browser/WebDriver session (launched before, quit()'d after), a mock server standing in for a third-party dependency (started before, stopped after), or simply test data seeded into a shared environment.

Scope is the key decision. Method-level (per-test) fixtures give the strongest isolation: every test gets a completely fresh resource, so nothing one test does can leak into another, at the cost of paying the setup cost (launching a browser, opening a connection) on every single test. Suite/module-level fixtures amortize that cost across many tests, which is much faster, but every test sharing that fixture must be written defensively enough not to depend on, or corrupt, state left behind by the tests that ran before it, across pytest specifically, scope='function' is the default (per-test), scope='class' shares across one test class, and scope='module'/scope='session' share across a whole file or the entire run respectively, in increasing order of both speed gained and isolation risk taken on.

Across the multiple frameworks a real team might use, the same idea appears under different names: pytest's @pytest.fixture, JUnit's @BeforeEach/@BeforeAll (paired with @AfterEach/@AfterAll), and TestNG's @BeforeMethod/@BeforeClass/@BeforeSuite, all express the same setup-scope-teardown shape.

A frequent real need is sharing a single WebDriver or database connection safely across many tests for speed: the safe pattern is a session- or module-scoped fixture for the EXPENSIVE, STATELESS part (launching the browser process itself, opening the connection pool) combined with a function-scoped fixture for anything STATEFUL that must not leak (navigating to a fresh page and clearing cookies before each test, or wrapping each test's database work in its own transaction that rolls back at the end), rather than sharing the stateful parts directly.

Worked example

import pytest

@pytest.fixture(scope="session")
def browser_process():
    # expensive, stateless: launch once for the whole run
    driver = launch_browser()
    yield driver
    driver.quit()

@pytest.fixture(scope="function")
def clean_page(browser_process):
    # cheap, stateful: reset to a known state before EVERY test
    browser_process.delete_all_cookies()
    browser_process.get("about:blank")
    yield browser_process

Here browser_process pays the expensive browser-launch cost once per test session, while clean_page guarantees every individual test starts from a clean slate (no leftover cookies or navigation state), combining the speed of session scope with the isolation function scope would otherwise provide alone.

Trade-offs and pitfalls

The most common mistake is sharing a session-scoped fixture that ALSO carries mutable state (a logged-in browser session, an open database transaction with uncommitted writes) directly across tests: the first test to run can leave the shared resource in a state the next test silently depends on, which produces order-dependent flakiness that only shows up when tests run in a different order or in parallel. Function scope avoids this entirely at the cost of speed; the combined pattern above (expensive+stateless at wide scope, cheap+stateful reset at narrow scope) is usually the better trade-off once a suite is large enough that full per-test setup is genuinely too slow.

Direct answer

Use a fake or mocked client whose fetch() method is scripted to raise TimeoutError a controlled number of times before succeeding, call the function under test, and assert both the returned result and the exact number of times the fake was called, so the test proves the retry loop actually re-invoked the client rather than merely tolerating a single failure.

Structured elaboration

The function under test has three behaviors worth testing separately, and a thorough answer covers all three rather than just the happy path:

1. Retries on TimeoutError and eventually succeeds: a fake client that fails N times then succeeds lets you assert both the returned value and calls == N + 1, which is the only way to prove it actually retried rather than, say, catching the exception and returning None on the first attempt.
2. Gives up after exhausting all retries: a fake client that always raises should cause the function to raise TimeoutError after exactly retries attempts, no more (a bug that retries indefinitely) and no fewer (a bug that gives up early).
3. Does not swallow a DIFFERENT exception type: the real function only catches TimeoutError; a fake raising ValueError should propagate immediately without being retried, since the code's except TimeoutError clause would not match it. Skipping this case is easy to miss and is exactly the kind of edge a senior candidate is expected to name unprompted.

To test backoff policy (a delay between attempts, if the real function had one), you would not sleep for real time in a test: patch the sleep/delay call (e.g. unittest.mock.patch on asyncio.sleep) and assert it was called with the expected, growing delays, or inject a fake clock/scheduler so the test runs in milliseconds regardless of what backoff schedule it is verifying.

Worked example

import pytest

async def fetch_with_retry(client, retries=3):
    for i in range(retries):
        try:
            return await client.fetch()
        except TimeoutError:
            pass
    raise TimeoutError

class FlakyClient:
    """Fails a fixed number of times, then succeeds -- lets the test assert the exact retry count."""
    def __init__(self, fail_times):
        self.fail_times = fail_times
        self.calls = 0
    async def fetch(self):
        self.calls += 1
        if self.calls <= self.fail_times:
            raise TimeoutError(f"attempt {self.calls} timed out")
        return "ok"

class AlwaysFailsClient:
    def __init__(self):
        self.calls = 0
    async def fetch(self):
        self.calls += 1
        raise TimeoutError(f"attempt {self.calls} timed out")

@pytest.mark.asyncio
async def test_retries_and_succeeds_after_two_failures():
    client = FlakyClient(fail_times=2)
    result = await fetch_with_retry(client, retries=3)
    assert result == "ok"
    assert client.calls == 3  # 2 failures + 1 success: proves it retried at least twice, not zero or one time

@pytest.mark.asyncio
async def test_gives_up_after_exhausting_retries():
    client = AlwaysFailsClient()
    with pytest.raises(TimeoutError):
        await fetch_with_retry(client, retries=3)
    assert client.calls == 3  # exactly `retries` attempts, not more and not fewer

@pytest.mark.asyncio
async def test_does_not_retry_on_non_timeout_error():
    class WrongErrorClient:
        async def fetch(self):
            raise ValueError("not a timeout")
    client = WrongErrorClient()
    with pytest.raises(ValueError):
        await fetch_with_retry(client, retries=3)

Running it:

$ pytest test_retry.py -v
collected 3 items

test_retry.py::test_retries_and_succeeds_after_two_failures PASSED
test_retry.py::test_gives_up_after_exhausting_retries PASSED
test_retry.py::test_does_not_retry_on_non_timeout_error PASSED

3 passed in 0.01s

Trade-offs and pitfalls

The single biggest pitfall is a fake client that only ever raises once and then always succeeds, which makes retries >= 1 pass trivially and does not actually confirm the loop can survive TWO consecutive failures; the fail_times parameter above exists specifically so the test can push that number up and confirm the loop keeps trying rather than bailing after the first retry. A second pitfall is asserting only the return value and never the call count: a function that caught the exception and returned a hardcoded success value without actually calling fetch() again would pass a result-only assertion while being completely broken, which is exactly why client.calls == 3 is load-bearing in both tests above, not decorative. Finally, once real backoff delays are involved, sleeping for the actual duration in a test is a real anti-pattern (it makes the suite slow and turns a unit test into a de facto timing test); patch the sleep call instead of waiting on it.

Direct answer

@pytest.mark.parametrize runs the same test body once per row of data you supply, each row becoming its own reported test case with its own pass/fail and its own readable ID in the output, instead of one test that loops internally and reports a single result for all three credential pairs.

Structured elaboration

The pattern has three parts: a parameter list (here, three (username, password, should_succeed) tuples), a test function whose parameters are named exactly to match, and a fixture or setup for anything shared across all rows (here, the object you log in against). The decorator generates one independent test invocation per tuple, so a failure on the invalid-credentials row is reported as test_login[invalid-wrong-False] failing while the two valid-credential rows still show as passed, which is the entire point: a candidate who instead writes for creds in credential_list: assert login(...) gets one pass/fail for the whole loop and loses per-case visibility the moment more than one row is involved.

Worked example

import pytest

class FakeAuthAPI:
    """Stand-in for a real login endpoint/UI so the mechanic can be shown without a server or browser."""
    _valid = {("user1", "pass1"), ("user2", "pass2")}
    def login(self, username, password):
        return (username, password) in self._valid

@pytest.fixture
def api():
    return FakeAuthAPI()

@pytest.mark.parametrize("username, password, should_succeed", [
    ("user1", "pass1", True),
    ("user2", "pass2", True),
    ("invalid", "wrong", False),
])
def test_login(api, username, password, should_succeed):
    result = api.login(username, password)
    assert result is should_succeed

Running it:

$ pytest test_login_parametrize.py -v
collected 3 items

test_login_parametrize.py::test_login[user1-pass1-True] PASSED
test_login_parametrize.py::test_login[user2-pass2-True] PASSED
test_login_parametrize.py::test_login[invalid-wrong-False] PASSED

3 passed in 0.01s

Three independent results, one per credential pair, each individually addressable in a CI report.

The same mechanic works on a trivial pure function too, which is worth knowing as an even more compact way to demonstrate the syntax when the interviewer only wants to see the API, not a login flow:

import pytest

def is_palindrome(s):
    return s == s[::-1]

@pytest.mark.parametrize("s, expected", [
    ("radar", True),
    ("hello", False),
    ("", True),
])
def test_is_palindrome(s, expected):
    assert is_palindrome(s) == expected

$ pytest -v
test_is_palindrome[radar-True] PASSED
test_is_palindrome[hello-False] PASSED
test_is_palindrome[-True] PASSED
3 passed in 0.01s

Trade-offs and pitfalls

The most common mistake is parametrizing a fixture instead of the test when the setup itself needs to vary (use @pytest.fixture(params=[...]) for that, not @pytest.mark.parametrize on the test), which produces confusing double-parametrization if both are used for the same axis. A second pitfall is mutable default state leaking across parametrized runs: if the fixture in this example returned a module-level object instead of a fresh one per test, a mutation in one row (e.g. banning user1 after a failed attempt) could silently affect a later row's result depending on execution order. Finally, once the list of tuples grows past a handful of rows, hardcoding them inline stops scaling. That is the point where the case for pulling data from an external file becomes a genuine data-management decision (how it is stored, generated, and kept in sync with the tests) that goes beyond this one script.