Test Automation Scripting Questions

Direct answer

@pytest.mark.parametrize runs the same test body once per row of data you supply, each row becoming its own reported test case with its own pass/fail and its own readable ID in the output, instead of one test that loops internally and reports a single result for all three credential pairs.

Structured elaboration

The pattern has three parts: a parameter list (here, three (username, password, should_succeed) tuples), a test function whose parameters are named exactly to match, and a fixture or setup for anything shared across all rows (here, the object you log in against). The decorator generates one independent test invocation per tuple, so a failure on the invalid-credentials row is reported as test_login[invalid-wrong-False] failing while the two valid-credential rows still show as passed, which is the entire point: a candidate who instead writes for creds in credential_list: assert login(...) gets one pass/fail for the whole loop and loses per-case visibility the moment more than one row is involved.

Worked example

import pytest

class FakeAuthAPI:
    """Stand-in for a real login endpoint/UI so the mechanic can be shown without a server or browser."""
    _valid = {("user1", "pass1"), ("user2", "pass2")}
    def login(self, username, password):
        return (username, password) in self._valid

@pytest.fixture
def api():
    return FakeAuthAPI()

@pytest.mark.parametrize("username, password, should_succeed", [
    ("user1", "pass1", True),
    ("user2", "pass2", True),
    ("invalid", "wrong", False),
])
def test_login(api, username, password, should_succeed):
    result = api.login(username, password)
    assert result is should_succeed

Running it:

$ pytest test_login_parametrize.py -v
collected 3 items

test_login_parametrize.py::test_login[user1-pass1-True] PASSED
test_login_parametrize.py::test_login[user2-pass2-True] PASSED
test_login_parametrize.py::test_login[invalid-wrong-False] PASSED

3 passed in 0.01s

Three independent results, one per credential pair, each individually addressable in a CI report.

The same mechanic works on a trivial pure function too, which is worth knowing as an even more compact way to demonstrate the syntax when the interviewer only wants to see the API, not a login flow:

import pytest

def is_palindrome(s):
    return s == s[::-1]

@pytest.mark.parametrize("s, expected", [
    ("radar", True),
    ("hello", False),
    ("", True),
])
def test_is_palindrome(s, expected):
    assert is_palindrome(s) == expected

$ pytest -v
test_is_palindrome[radar-True] PASSED
test_is_palindrome[hello-False] PASSED
test_is_palindrome[-True] PASSED
3 passed in 0.01s

Trade-offs and pitfalls

The most common mistake is parametrizing a fixture instead of the test when the setup itself needs to vary (use @pytest.fixture(params=[...]) for that, not @pytest.mark.parametrize on the test), which produces confusing double-parametrization if both are used for the same axis. A second pitfall is mutable default state leaking across parametrized runs: if the fixture in this example returned a module-level object instead of a fresh one per test, a mutation in one row (e.g. banning user1 after a failed attempt) could silently affect a later row's result depending on execution order. Finally, once the list of tuples grows past a handful of rows, hardcoding them inline stops scaling. That is the point where the case for pulling data from an external file becomes a genuine data-management decision (how it is stored, generated, and kept in sync with the tests) that goes beyond this one script.

Direct answer

A hard assertion stops the test the instant it fails, so anything checked after it never runs; a soft assertion (or "verification") records the failure and lets the test keep going, so you learn about every problem in one run instead of fixing one, re-running, hitting the next one, and repeating. Where a framework has no native soft-assertion support, collect failures in a list as you go and raise one combined error at the end of the test.

Structured elaboration

Execution flow is the entire distinction. A hard assertion (assert x == y in pytest, Assert.assertEquals in JUnit) raises immediately on failure, unwinding the test right there; anything after it, including cleanup code that was not in a finally/fixture teardown, is skipped. A soft assertion records "this check failed" without raising, continues executing the rest of the test body, and only surfaces the accumulated failures at the very end (many frameworks fail the test overall if ANY soft assertion failed, even though execution continued).

Soft assertions earn their value on tests that check several INDEPENDENT properties of one action, where knowing about all of them in a single run saves real round-trips: a submitted order confirmation page that should show the correct order number, the correct total, the correct shipping address, and the correct estimated delivery date. If you hard-assert each in sequence and the order number is wrong, you never learn the total was also wrong until you fix the first bug and re-run; soft-asserting all four tells you everything in one run.

Worked example

A collection pattern that works in any language without native soft-assert support:

class SoftAssertions:
    def __init__(self):
        self.failures = []

    def check(self, condition, message):
        if not condition:
            self.failures.append(message)

    def assert_all(self):
        if self.failures:
            raise AssertionError(
                f"{len(self.failures)} check(s) failed:\n" + "\n".join(self.failures)
            )

def test_order_confirmation():
    order = {"order_number": "A100", "total": "$41.50", "address": "12 Main St", "eta": "3 days"}
    expected = {"order_number": "A100", "total": "$45.00", "address": "12 Main St", "eta": "2 days"}

    soft = SoftAssertions()
    soft.check(order["order_number"] == expected["order_number"], f"order_number: expected {expected['order_number']!r}, got {order['order_number']!r}")
    soft.check(order["total"] == expected["total"], f"total: expected {expected['total']!r}, got {order['total']!r}")
    soft.check(order["address"] == expected["address"], f"address: expected {expected['address']!r}, got {order['address']!r}")
    soft.check(order["eta"] == expected["eta"], f"eta: expected {expected['eta']!r}, got {order['eta']!r}")
    soft.assert_all()

Running it:

$ pytest test_soft_assertions.py -v
test_order_confirmation FAILED
AssertionError: 2 check(s) failed:
total: expected '$45.00', got '$41.50'
eta: expected '2 days', got '3 days'
1 failed in 0.01s

This raises one AssertionError listing BOTH the total mismatch and the eta mismatch together, rather than stopping at whichever one happens to be checked first, exactly the outcome that makes soft assertions valuable for this kind of multi-property verification.

Trade-offs and pitfalls

Soft assertions are the wrong tool when a later check is only meaningful if an earlier one held: if the order confirmation page never loaded at all, checking four fields on a page that does not exist produces four confusing, unrelated-looking failures instead of one clear "the page never loaded" signal; that case calls for a hard assertion (or a precondition check) before switching into soft-assertion mode for the independent details. Also worth naming: the assertion-versus-verification vocabulary distinguishes these by what happens to execution flow on failure, not by how "important" the check is; a hard assertion failing does not mean the check mattered more, only that the test was written to treat it as blocking.

Direct answer

A reliable assertion checks a stable, meaningful property of the outcome (the right data appears in the right place, in a form unlikely to change for unrelated reasons) rather than an incidental detail of the current markup or wording, so the test survives cosmetic changes and only fails when the actual behavior it cares about actually breaks.

Structured elaboration

"Fragile" and "robust" are really about what SURFACE an assertion couples itself to. A fragile assertion couples to something that changes for reasons that have nothing to do with correctness: exact visual layout, incidental whitespace, a full page title string that includes a marketing tagline someone will edit next quarter, or a CSS class name a frontend engineer renames during a refactor. A robust assertion couples to the thing the test actually cares about proving: that a specific, stable identifier or piece of application state reflects the successful action, expressed in a form the team has deliberately made stable for testing (a data-test-id, a semantic success message, or the persisted record itself) rather than something that was never meant to be a contract.

Worked example

For "verify a form submission succeeded":

• Fragile example 1: asserting the exact full text of a success banner, e.g. assert banner.text == "Thank you! Your request has been received and a team member will reach out within 2 business days." A copy edit to that sentence breaks the test with zero change in actual behavior.
• Fragile example 2: asserting on a generated, unstable CSS class like .MuiButton-root-482, which a component-library upgrade or a build-hash change can alter without any user-visible difference.
• Robust example 1: asserting on a stable data-test-id="submission-success" element being present, decoupled from whatever the button's styling framework happens to generate.
• Robust example 2: asserting on the actual state change the submission was supposed to cause, for example that a new record now exists (via an API call or a database check) with the submitted values, which proves the FUNCTIONAL outcome regardless of how the confirmation UI is worded or styled.

Trade-offs and pitfalls

The most common mistake is treating "more specific" as automatically "more robust": a very precise assertion (exact copy, exact pixel position) feels thorough but is actually the most fragile kind, since it locks the test to details a product or design team changes routinely. The trade-off on the state-based robust example is that it requires the test to reach past the UI (an API check, a database query) which is more setup than reading the DOM, but that extra setup is exactly what buys freedom from UI wording/styling churn. A middle ground worth naming: asserting on a stable data-test-id gives most of the robustness of a full backend check with much less setup cost, which is why teams that can influence the frontend often standardize on test-only attributes rather than pushing every assertion down to the API layer.

Direct answer

Open the login URL, wait explicitly for the username and password fields to exist before touching them, fill and submit the form, then wait for the resulting page's title to reflect the post-login state before asserting on it, so the test never races the page's own load or redirect timing.

Structured elaboration

The key discipline this exercise tests is using an EXPLICIT wait (WebDriverWait with an expected condition) at every point where the DOM state genuinely depends on timing: once for the username field appearing after the login page loads, and once for the title changing after the redirect following a successful submit. A test that instead used time.sleep(2) in both places would usually pass locally and become flaky in CI the moment the page is slower than the guessed sleep duration; an explicit wait polls until the condition is true (or times out), so it is correct regardless of how fast or slow the page actually is.

Worked example

import pytest
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

def do_login(driver, base_url="https://example.com/login"):
    driver.get(base_url)
    wait = WebDriverWait(driver, 10)
    wait.until(EC.presence_of_element_located((By.ID, "username"))).send_keys("testuser")
    driver.find_element(By.ID, "password").send_keys("P@ssw0rd")
    driver.find_element(By.ID, "login-btn").click()
    return wait.until(EC.title_contains("Dashboard"))

@pytest.fixture
def driver():
    from selenium import webdriver
    d = webdriver.Chrome()   # real suite: chromedriver on PATH, as the question assumes
    yield d
    d.quit()

def test_login_direct(driver):
    assert do_login(driver) is True

No browser binary is available in this environment, so the control flow (correct locator usage, correct wait-then-act ordering, correct final assertion) was verified separately against a mocked driver (a realistic title string and a stub find_element) instead of webdriver.Chrome():

from unittest.mock import MagicMock, call

@pytest.fixture
def driver():
    d = MagicMock()
    d.title = "My App - Dashboard"
    d.find_element.return_value = MagicMock()
    return d

def test_login_direct_mocked(driver):
    result = do_login(driver)
    assert result is True
    driver.get.assert_called_once_with("https://example.com/login")
    assert driver.find_element.call_args_list == [
        call(By.ID, "username"),
        call(By.ID, "password"),
        call(By.ID, "login-btn"),
    ]

$ pytest test_direct_login_mocked.py -v
test_login_direct_mocked PASSED
1 passed in 0.11s

This confirms driver.get was called with the right URL exactly once, and find_element was called for username (via the WebDriverWait-guarded lookup), password, and login-btn, in the order the function performs them.

Trade-offs and pitfalls

The most common mistake on this exact exercise is waiting only ONCE, for the username field, and then asserting on the title immediately without a second wait: since the redirect to the dashboard is itself asynchronous, the title assertion can fire before the navigation completes and fail intermittently depending on network/render speed, which is exactly the kind of test that passes on a fast machine and flakes in CI. A second pitfall is hardcoding the 10-second timeout inline on every wait call rather than as one named constant, which becomes a maintenance problem the moment a team decides to tune it suite-wide.

Direct answer

Use a fake or mocked client whose fetch() method is scripted to raise TimeoutError a controlled number of times before succeeding, call the function under test, and assert both the returned result and the exact number of times the fake was called, so the test proves the retry loop actually re-invoked the client rather than merely tolerating a single failure.

Structured elaboration

The function under test has three behaviors worth testing separately, and a thorough answer covers all three rather than just the happy path:

1. Retries on TimeoutError and eventually succeeds: a fake client that fails N times then succeeds lets you assert both the returned value and calls == N + 1, which is the only way to prove it actually retried rather than, say, catching the exception and returning None on the first attempt.
2. Gives up after exhausting all retries: a fake client that always raises should cause the function to raise TimeoutError after exactly retries attempts, no more (a bug that retries indefinitely) and no fewer (a bug that gives up early).
3. Does not swallow a DIFFERENT exception type: the real function only catches TimeoutError; a fake raising ValueError should propagate immediately without being retried, since the code's except TimeoutError clause would not match it. Skipping this case is easy to miss and is exactly the kind of edge a senior candidate is expected to name unprompted.

To test backoff policy (a delay between attempts, if the real function had one), you would not sleep for real time in a test: patch the sleep/delay call (e.g. unittest.mock.patch on asyncio.sleep) and assert it was called with the expected, growing delays, or inject a fake clock/scheduler so the test runs in milliseconds regardless of what backoff schedule it is verifying.

Worked example

import pytest

async def fetch_with_retry(client, retries=3):
    for i in range(retries):
        try:
            return await client.fetch()
        except TimeoutError:
            pass
    raise TimeoutError

class FlakyClient:
    """Fails a fixed number of times, then succeeds -- lets the test assert the exact retry count."""
    def __init__(self, fail_times):
        self.fail_times = fail_times
        self.calls = 0
    async def fetch(self):
        self.calls += 1
        if self.calls <= self.fail_times:
            raise TimeoutError(f"attempt {self.calls} timed out")
        return "ok"

class AlwaysFailsClient:
    def __init__(self):
        self.calls = 0
    async def fetch(self):
        self.calls += 1
        raise TimeoutError(f"attempt {self.calls} timed out")

@pytest.mark.asyncio
async def test_retries_and_succeeds_after_two_failures():
    client = FlakyClient(fail_times=2)
    result = await fetch_with_retry(client, retries=3)
    assert result == "ok"
    assert client.calls == 3  # 2 failures + 1 success: proves it retried at least twice, not zero or one time

@pytest.mark.asyncio
async def test_gives_up_after_exhausting_retries():
    client = AlwaysFailsClient()
    with pytest.raises(TimeoutError):
        await fetch_with_retry(client, retries=3)
    assert client.calls == 3  # exactly `retries` attempts, not more and not fewer

@pytest.mark.asyncio
async def test_does_not_retry_on_non_timeout_error():
    class WrongErrorClient:
        async def fetch(self):
            raise ValueError("not a timeout")
    client = WrongErrorClient()
    with pytest.raises(ValueError):
        await fetch_with_retry(client, retries=3)

Running it:

$ pytest test_retry.py -v
collected 3 items

test_retry.py::test_retries_and_succeeds_after_two_failures PASSED
test_retry.py::test_gives_up_after_exhausting_retries PASSED
test_retry.py::test_does_not_retry_on_non_timeout_error PASSED

3 passed in 0.01s

Trade-offs and pitfalls

The single biggest pitfall is a fake client that only ever raises once and then always succeeds, which makes retries >= 1 pass trivially and does not actually confirm the loop can survive TWO consecutive failures; the fail_times parameter above exists specifically so the test can push that number up and confirm the loop keeps trying rather than bailing after the first retry. A second pitfall is asserting only the return value and never the call count: a function that caught the exception and returned a hardcoded success value without actually calling fetch() again would pass a result-only assertion while being completely broken, which is exactly why client.calls == 3 is load-bearing in both tests above, not decorative. Finally, once real backoff delays are involved, sleeping for the actual duration in a test is a real anti-pattern (it makes the suite slow and turns a unit test into a de facto timing test); patch the sleep call instead of waiting on it.