Test Automation Scripting Questions

Direct answer

The 500 with a NullReferenceException inside OrderService.SubmitOrder() means the FAILURE is in the application under test, not the test itself: the immediate triage is to check whether this is reproducible on that specific commit/environment, correlate it with anything that changed in staging-2026-02-10 or commit abc123, and file it as an application bug with the exact stack trace attached, rather than treating a 500-with-a-real-server-side-exception as a flaky test problem.

Structured elaboration

Reading the artifacts in order: the traceback shows the ASSERTION failed (assert response.status_code == 200) because the actual value was 500, and the response body itself contains a server-side NullReferenceException inside OrderService.SubmitOrder(), which is a strong signal this is a genuine backend defect (something the order service expected to be non-null was null on this request), not a test-side timing or environment issue; a flaky/environment problem would more typically show up as a timeout, a connection error, or an inconsistent intermittent result across retries, not a consistent, named exception with a stack trace pointing at a specific method.

Immediate triage steps: (1) confirm reproducibility, does this fail every time against staging-2026-02-10 and commit abc123, or only intermittently; (2) check whether commit abc123 (or a nearby one) touched OrderService.SubmitOrder() or anything it depends on, since a NullReferenceException at that specific method is a strong lead toward a recent regression; (3) check whether the specific request payload that triggered this (whatever test_submit_order actually submitted) hits an edge case the code does not null-check, since that narrows the fix to a specific missing guard rather than a vague "something broke."

Next steps to prevent recurrence: file the bug with the full stack trace, the commit, and the environment attached (not just "checkout sometimes fails"), so the owning team can reproduce it directly; add a null-check/guard at the specific point OrderService.SubmitOrder() dereferences the null value once the root field is identified; and consider whether the test itself should also assert on RESPONSE BODY shape for the success case, not just status code, so a future regression that returns 200 with a malformed body is also caught.

Trade-offs and pitfalls

The most damaging mistake here is treating this as "just retry it, tests are flaky sometimes" and moving on: a consistent, named server-side exception is exactly the kind of failure retries are most likely to mask rather than fix, since the underlying null-reference bug will still be there for the next real user who hits the same code path. A second pitfall is triaging purely from the test's own assertion (500 != 200) without reading the response body at all, since the body here is doing most of the diagnostic work (naming the exact exception type and method); a shallow triage that stops at "the API returned an error" misses information that was already captured for free.

Direct answer

A test fixture is the setup an automated test needs before it can run and the cleanup it needs afterward, whether that is a database connection, a browser instance, or a mock server; most test frameworks let you scope a fixture to run once per test, once per class of tests, or once per whole test suite/module, and choosing the right scope is a real trade-off between speed and isolation.

Structured elaboration

Common resources fixtures prepare and clean up: a database connection or transaction (opened before, rolled back or closed after), a browser/WebDriver session (launched before, quit()'d after), a mock server standing in for a third-party dependency (started before, stopped after), or simply test data seeded into a shared environment.

Scope is the key decision. Method-level (per-test) fixtures give the strongest isolation: every test gets a completely fresh resource, so nothing one test does can leak into another, at the cost of paying the setup cost (launching a browser, opening a connection) on every single test. Suite/module-level fixtures amortize that cost across many tests, which is much faster, but every test sharing that fixture must be written defensively enough not to depend on, or corrupt, state left behind by the tests that ran before it, across pytest specifically, scope='function' is the default (per-test), scope='class' shares across one test class, and scope='module'/scope='session' share across a whole file or the entire run respectively, in increasing order of both speed gained and isolation risk taken on.

Across the multiple frameworks a real team might use, the same idea appears under different names: pytest's @pytest.fixture, JUnit's @BeforeEach/@BeforeAll (paired with @AfterEach/@AfterAll), and TestNG's @BeforeMethod/@BeforeClass/@BeforeSuite, all express the same setup-scope-teardown shape.

A frequent real need is sharing a single WebDriver or database connection safely across many tests for speed: the safe pattern is a session- or module-scoped fixture for the EXPENSIVE, STATELESS part (launching the browser process itself, opening the connection pool) combined with a function-scoped fixture for anything STATEFUL that must not leak (navigating to a fresh page and clearing cookies before each test, or wrapping each test's database work in its own transaction that rolls back at the end), rather than sharing the stateful parts directly.

Worked example

import pytest

@pytest.fixture(scope="session")
def browser_process():
    # expensive, stateless: launch once for the whole run
    driver = launch_browser()
    yield driver
    driver.quit()

@pytest.fixture(scope="function")
def clean_page(browser_process):
    # cheap, stateful: reset to a known state before EVERY test
    browser_process.delete_all_cookies()
    browser_process.get("about:blank")
    yield browser_process

Here browser_process pays the expensive browser-launch cost once per test session, while clean_page guarantees every individual test starts from a clean slate (no leftover cookies or navigation state), combining the speed of session scope with the isolation function scope would otherwise provide alone.

Trade-offs and pitfalls

The most common mistake is sharing a session-scoped fixture that ALSO carries mutable state (a logged-in browser session, an open database transaction with uncommitted writes) directly across tests: the first test to run can leave the shared resource in a state the next test silently depends on, which produces order-dependent flakiness that only shows up when tests run in a different order or in parallel. Function scope avoids this entirely at the cost of speed; the combined pattern above (expensive+stateless at wide scope, cheap+stateful reset at narrow scope) is usually the better trade-off once a suite is large enough that full per-test setup is genuinely too slow.

Direct answer

Use new Actions(driver).dragAndDrop(source, target).perform() first, then verify the drop actually took effect (check some observable state change on the target, not just that no exception was thrown); if it silently did nothing, fall back to dispatching a real HTML5 drag event sequence (dragstart/dragover/drop/dragend) via executeScript, since many browsers implement HTML5 drag-and-drop through native OS-level drag events that the Actions API's synthetic mouse-move-based approach does not always trigger correctly.

Structured elaboration

Actions.dragAndDrop under the hood synthesizes a press, a sequence of mouse moves, and a release, which is exactly how NON-HTML5 drag interactions (dragging a slider, resizing a panel) actually work in the browser. Genuine HTML5 drag-and-drop (elements with draggable="true" participating in dragstart/dragover/drop events) is a SEPARATE browser mechanism that some browser/driver combinations do not reliably trigger from synthetic mouse events alone, which is why Actions.dragAndDrop can appear to succeed (no exception, the call completes) while the actual drop event handler never fires and nothing moves.

The fallback constructs and dispatches the real HTML5 drag events directly via JavaScript, using a DataTransfer object (the payload HTML5 drag events carry) and firing dragstart on the source, drop on the target, and dragend on the source, which bypasses the synthetic-mouse-event path entirely and talks to the same event handlers the application's own JavaScript listens for.

Worked example

Actions actions = new Actions(driver);
actions.dragAndDrop(source, target).perform();

// JS fallback, called via driver.executeScript(...) in Java (or execute_script in Python),
// when the native Actions-API drag produces no observable effect:
var dt = new DataTransfer();
arguments[0].dispatchEvent(new DragEvent('dragstart', {dataTransfer: dt}));
arguments[1].dispatchEvent(new DragEvent('drop', {dataTransfer: dt}));
arguments[0].dispatchEvent(new DragEvent('dragend', {dataTransfer: dt}));

No JVM or real browser is available in this environment, so the Java Actions/executeScript calls above are not claimed to have been executed; they are checked against the standard Selenium Java API shape only. What CAN be verified without a browser is the decision logic itself (check an observable count on the target before/after the Actions-API attempt; fall back to the JS event sequence only if unchanged), modeled directly and run under Python with mocked collaborators standing in for the Java calls:

from unittest.mock import Mock

def drag_with_fallback(perform_actions_drag, check_state, js_fallback):
    before = check_state()
    perform_actions_drag()
    after = check_state()
    if after != before:
        return "actions_api_worked"
    js_fallback()
    return "js_fallback_used"

# case 1: Actions API works, observable count changes 0 -> 1
states = iter([0, 1])
result = drag_with_fallback(Mock(), Mock(side_effect=lambda: next(states)), Mock())
print("case 1 (Actions API works, count changed):", result)

# case 2: Actions API silently no-ops, count stays 0 -> 0
js_fallback = Mock()
states = iter([0, 0])
result = drag_with_fallback(Mock(), Mock(side_effect=lambda: next(states)), js_fallback)
print("case 2 (Actions API silently no-ops, JS fallback used):", result)
print("JS fallback called:", js_fallback.called)

Actual output:

case 1 (Actions API works, count changed): actions_api_worked
case 2 (Actions API silently no-ops, JS fallback used): js_fallback_used
JS fallback called: True

This confirms the DECISION structure (only fall back when the observable state genuinely did not change) behaves as claimed. Translating it into Java is a direct one-to-one mapping: before/after become whatever observable count you read from the page, the comparison becomes a plain !before.equals(after), and the two branches call actions.dragAndDrop(...).perform() and the executeScript(...) fallback shown above respectively.

Trade-offs and pitfalls

The most important pitfall this exercise is testing awareness of: Actions.dragAndDrop completing without an exception is NOT evidence the drag succeeded, since the whole failure mode is silent (no error, just no effect); any test using drag-and-drop needs to assert on an actual OBSERVABLE outcome of the drop (an item count, a moved element's new position, a class change) rather than only checking that .perform() did not throw. A second pitfall with the JS fallback itself: dispatching synthetic DragEvents bypasses the browser's real user-input path entirely, so a test that only ever exercises the JS fallback risks never proving a genuine user COULD perform this drag with a real mouse; keeping the Actions-API attempt as the first try (and only falling back when it demonstrably fails) preserves that signal wherever the browser actually supports it.

Direct answer

For a real HTML <select>, use Selenium's Select helper class (select by visible text, value, or index) rather than clicking the option elements directly, since a native select's options are not always individually clickable the way ordinary DOM elements are; for a custom dropdown built from styled <div>s with ARIA roles, there is no Select helper available, so click the trigger to open it, then locate and click the matching option element directly, typically via its role="option" attribute or visible text.

Structured elaboration

The Select class exists specifically because native <select> elements render their options using the OS's own UI (not ordinary HTML the browser lets you click freely), so trying to find_element an individual <option> and .click() it directly is unreliable across browsers; Select wraps the correct underlying browser commands and offers select_by_visible_text, select_by_value, and select_by_index, choosing between them based on what is stable in the application (visible text is often most readable and most likely to survive unrelated markup changes, while value is more stable if the visible label is dynamic or localized).

Custom dropdowns (a styled trigger element that reveals a <div role="listbox"> containing <div role="option"> children on click, common in modern component libraries) are ordinary DOM elements once opened, so the pattern is: click the trigger, wait for the options container to appear, find the option matching your target (by role="option" plus its text, which is both semantically correct and accessibility-friendly), and click it directly, exactly like clicking any other visible element.

Worked example

from selenium.webdriver.support.ui import Select

def select_native_dropdown(driver, select_locator, visible_text):
    select_el = driver.find_element(*select_locator)
    Select(select_el).select_by_visible_text(visible_text)

def select_custom_dropdown(driver, trigger_locator, option_text):
    trigger = driver.find_element(*trigger_locator)
    trigger.click()
    options = driver.find_elements("css selector", "[role='option']")
    match = next(o for o in options if o.text == option_text)
    match.click()
    return match.text

Verified the custom-dropdown path against a mocked driver returning two option elements, confirming the correct one is located and clicked by text match:

custom ARIA dropdown -> Large

The native-select path uses the real selenium.webdriver.support.ui.Select import (verified it resolves against the installed Selenium 4 package); its actual selection behavior against a real <select> element cannot be exercised without a real browser, and is disclosed as traced-not-executed rather than claimed as run.

Trade-offs and pitfalls

The most common mistake is treating a custom ARIA-based dropdown the same as a native select, and trying to instantiate Select(custom_div_element), which raises an error since Select specifically requires a real <select> tag; recognizing which kind of dropdown you are looking at (check the actual rendered HTML, not just visual appearance) is the first real decision this question tests. A second pitfall on the custom-dropdown path is matching options by exact visible text alone when two options share a visual label but differ by a hidden value or data-* attribute (for example, two "Other" options meaning different things); scoping the match to a more specific attribute avoids selecting the wrong one when labels collide.

Direct answer

Use a fake or mocked client whose fetch() method is scripted to raise TimeoutError a controlled number of times before succeeding, call the function under test, and assert both the returned result and the exact number of times the fake was called, so the test proves the retry loop actually re-invoked the client rather than merely tolerating a single failure.

Structured elaboration

The function under test has three behaviors worth testing separately, and a thorough answer covers all three rather than just the happy path:

1. Retries on TimeoutError and eventually succeeds: a fake client that fails N times then succeeds lets you assert both the returned value and calls == N + 1, which is the only way to prove it actually retried rather than, say, catching the exception and returning None on the first attempt.
2. Gives up after exhausting all retries: a fake client that always raises should cause the function to raise TimeoutError after exactly retries attempts, no more (a bug that retries indefinitely) and no fewer (a bug that gives up early).
3. Does not swallow a DIFFERENT exception type: the real function only catches TimeoutError; a fake raising ValueError should propagate immediately without being retried, since the code's except TimeoutError clause would not match it. Skipping this case is easy to miss and is exactly the kind of edge a senior candidate is expected to name unprompted.

To test backoff policy (a delay between attempts, if the real function had one), you would not sleep for real time in a test: patch the sleep/delay call (e.g. unittest.mock.patch on asyncio.sleep) and assert it was called with the expected, growing delays, or inject a fake clock/scheduler so the test runs in milliseconds regardless of what backoff schedule it is verifying.

Worked example

import pytest

async def fetch_with_retry(client, retries=3):
    for i in range(retries):
        try:
            return await client.fetch()
        except TimeoutError:
            pass
    raise TimeoutError

class FlakyClient:
    """Fails a fixed number of times, then succeeds -- lets the test assert the exact retry count."""
    def __init__(self, fail_times):
        self.fail_times = fail_times
        self.calls = 0
    async def fetch(self):
        self.calls += 1
        if self.calls <= self.fail_times:
            raise TimeoutError(f"attempt {self.calls} timed out")
        return "ok"

class AlwaysFailsClient:
    def __init__(self):
        self.calls = 0
    async def fetch(self):
        self.calls += 1
        raise TimeoutError(f"attempt {self.calls} timed out")

@pytest.mark.asyncio
async def test_retries_and_succeeds_after_two_failures():
    client = FlakyClient(fail_times=2)
    result = await fetch_with_retry(client, retries=3)
    assert result == "ok"
    assert client.calls == 3  # 2 failures + 1 success: proves it retried at least twice, not zero or one time

@pytest.mark.asyncio
async def test_gives_up_after_exhausting_retries():
    client = AlwaysFailsClient()
    with pytest.raises(TimeoutError):
        await fetch_with_retry(client, retries=3)
    assert client.calls == 3  # exactly `retries` attempts, not more and not fewer

@pytest.mark.asyncio
async def test_does_not_retry_on_non_timeout_error():
    class WrongErrorClient:
        async def fetch(self):
            raise ValueError("not a timeout")
    client = WrongErrorClient()
    with pytest.raises(ValueError):
        await fetch_with_retry(client, retries=3)

Running it:

$ pytest test_retry.py -v
collected 3 items

test_retry.py::test_retries_and_succeeds_after_two_failures PASSED
test_retry.py::test_gives_up_after_exhausting_retries PASSED
test_retry.py::test_does_not_retry_on_non_timeout_error PASSED

3 passed in 0.01s

Trade-offs and pitfalls

The single biggest pitfall is a fake client that only ever raises once and then always succeeds, which makes retries >= 1 pass trivially and does not actually confirm the loop can survive TWO consecutive failures; the fail_times parameter above exists specifically so the test can push that number up and confirm the loop keeps trying rather than bailing after the first retry. A second pitfall is asserting only the return value and never the call count: a function that caught the exception and returned a hardcoded success value without actually calling fetch() again would pass a result-only assertion while being completely broken, which is exactly why client.calls == 3 is load-bearing in both tests above, not decorative. Finally, once real backoff delays are involved, sleeping for the actual duration in a test is a real anti-pattern (it makes the suite slow and turns a unit test into a de facto timing test); patch the sleep call instead of waiting on it.