Microsoft SDET (Software Development Engineer in Test) Entry Level Interview Preparation Guide

Direct answer

Below is a reusable HTTP GET function with retry and exponential backoff for transient server errors and network errors, with configurable attempt count and base delay.

Structured elaboration

The function distinguishes what's worth retrying (a timeout, a connection error, a 5xx that's likely transient) from what isn't (a 4xx, which means the request itself is wrong and retrying an unmodified request will just fail the same way again), and backs off exponentially between attempts so a struggling server isn't hit with an immediate retry storm.

Worked example

import time
import requests


def resilient_get(url, max_attempts=5, initial_backoff=0.5, backoff_factor=2,
                   retry_statuses=(500, 502, 503, 504)):
    last_exception = None
    delay = initial_backoff

    for attempt in range(1, max_attempts + 1):
        try:
            resp = requests.get(url, timeout=5)
            if resp.status_code not in retry_statuses:
                return resp  # success, or a non-retryable error (e.g. 4xx): return as-is
            last_exception = None
        except (requests.ConnectionError, requests.Timeout) as e:
            last_exception = e
            resp = None

        if attempt == max_attempts:
            if last_exception:
                raise last_exception
            return resp  # exhausted retries on a retryable status; return the last response

        time.sleep(delay)
        delay *= backoff_factor

    raise RuntimeError("unreachable")  # defensive; loop always returns or raises above

Thread-safety. As written, resilient_get has no shared mutable state at all: delay, attempt, and last_exception are all local to each call, so many test threads calling it concurrently don't interact with each other in any way. The one thing worth being deliberate about in concurrent use is the underlying requests session: this version uses the module-level requests.get, which creates a new connection per call and is safe under concurrency but doesn't reuse connections. If you switch to a shared requests.Session() for connection pooling (a reasonable optimization under high concurrency), the Session object itself needs to be either one per thread or explicitly documented as thread-safe for your use case, since requests.Session is not guaranteed thread-safe for concurrent use by requests' own documentation.

Verified with a fixture that fails twice with a 503 and then succeeds:

call_count = [0]

def flaky_get(url, timeout):
    call_count[0] += 1
    class FakeResp:
        status_code = 503 if call_count[0] <= 2 else 200
    return FakeResp()

requests.get = flaky_get  # monkeypatched into requests.get for this test

resp = resilient_get("http://fake/x", initial_backoff=0.01)
print(f"attempts made: {call_count[0]}")
print(f"final status_code: {resp.status_code}")

Running resilient_get against this fixture (with initial_backoff=0.01 to keep the test fast) returns a 200 after exactly 3 attempts, confirming the retry loop and the eventual-success path both work:

attempts made: 3
final status_code: 200

Trade-offs and pitfalls

A 4xx status code deliberately does NOT trigger a retry in this implementation: retrying an unmodified request that the server has already rejected as invalid wastes time and, in the worst case, can look like an attempted abuse pattern to the server (repeated requests to an endpoint that keeps rejecting them). If a caller genuinely wants to retry a 429 (rate-limited) specifically, that status needs to be added to retry_statuses deliberately, and ideally the delay should respect a Retry-After header if the server provides one, rather than blindly following the function's own generic backoff schedule.

Direct answer
The situation I'd describe is a mid-sized project where my manager wanted me to present a set of results to a client as more conclusive than the underlying data actually supported, because the client relationship was under strain and a confident-sounding update would help. My personal value was straightforward accuracy in what I present, even when the more cautious version is less comfortable to deliver; my manager's approach prioritized relationship repair over precision in that specific moment. I did not treat it as a fight to win outright; I looked for a version of the update that was honest and still served the relationship.

Structured elaboration

1. Name the actual tension precisely, not just "we disagreed." In this case it was not that my manager wanted me to lie; it was a difference in where to draw the line between appropriately confident communication and overstating certainty, which is a much more common and more defensible kind of workplace values conflict than an outright integrity violation.
2. Raise the concern directly and early, privately, before the moment it would matter (the client meeting), rather than either silently complying or making it a public confrontation. I asked my manager one on one what specifically in the data supported the stronger framing, which turned the conversation from a disagreement about values into a conversation about evidence.
3. Offer an alternative that serves the underlying goal your manager actually cares about. My manager's real goal was preserving the client relationship, not the specific wording; I proposed a version that led with the two results we were genuinely confident in, was transparent about the one metric still trending in the wrong direction, and paired it with a concrete next step and timeline. This served the relationship-repair goal without requiring me to overstate anything.
4. Be honest about what you would do if the answer had been no. If my manager had insisted on the original framing after that conversation, my actual next step would have been to ask to attach a short written appendix with the caveated numbers, so the honest version existed in the record even if it wasn't the headline; if that had also been refused, I would have escalated to my manager's manager rather than either comply silently or refuse outright, because the stakes (client trust, and my own credibility if the caveated number surfaced later) were high enough to warrant it.
5. Reflect honestly on what you learned, including about your own judgment, not only about the other person. I learned that raising the concern as a specific evidentiary question ("what supports this framing") got further, faster, than raising it as a values statement ("I'm not comfortable with this") would have, because it gave my manager something concrete to respond to.

Worked example
The client update, as originally proposed, said: "engagement is up and the rollout is on track." What the underlying data actually showed: two of three key metrics had improved meaningfully, but the third (a retention metric the client cared about specifically) had been flat to slightly down for three weeks running, with a plausible but unconfirmed hypothesis for why. The version I proposed and we ultimately sent said: "engagement and adoption are both up meaningfully this period; retention is currently flat, and we have identified a likely cause we're testing a fix for over the next two weeks, with a follow-up update once we have results." The client's actual reaction was more positive than my manager expected, specifically because the concrete next step read as more credible than an unqualified "on track" would have.

Trade-offs & pitfalls
The common failure in answering this question is picking an example that is really just "I disagreed with a decision," with no genuine values dimension, or the opposite extreme, an example so severe (fraud, safety, legal risk) that it reads as a one-time crisis story rather than the kind of ordinary, recurring tension this question is actually probing for. Another pitfall is describing the resolution as pure capitulation ("I raised it once, they said no, I dropped it") or pure martyrdom ("I refused and it cost me"), neither of which shows the judgment interviewers are actually testing for: the ability to find a version of the truth that serves both your own integrity and the legitimate underlying goal the other person had.

Direct answer
Statement coverage only asks whether every line of code executed at least once across the test suite; branch coverage asks whether every possible outcome (true AND false) of every decision point was exercised; path coverage goes further and asks whether every distinct route through the function's control flow was exercised. A test suite can reach 100% statement coverage while a real bug ships, because a decision point can have a branch that is never TAKEN even though the line containing the decision itself still counts as "executed."

Structured elaboration

Worked example (executed, the bug is real)

def safe_divide(a, b):
    if b != 0:
        result = a / b
    return result

A test suite containing only safe_divide(10, 2) achieves 100% STATEMENT coverage: the if line executes, the assignment line executes, and the return line executes, three statements, three executions, 100%. Running it: safe_divide(10, 2) = 5.0, test passes.

Running the SAME function with safe_divide(10, 0), the case that only branch coverage would have forced into the suite (the FALSE branch of if b != 0), produces an actual crash:

BUG CONFIRMED: UnboundLocalError on b=0 -> cannot access local variable 'result' where it is not associated with a value

This is a genuine, executed failure, not a hypothetical: result is only ever assigned inside the if block, so when b == 0 the function reaches return result with result never defined. Statement coverage was satisfied by the single passing test because the if line itself counts as "covered" the moment it runs, regardless of which way the condition resolves; only branch coverage's requirement to exercise the FALSE outcome would have forced a test that discovers this crash.

Trade-offs & pitfalls
A common misreading of this example is to conclude "always aim for the strongest criterion (path coverage) everywhere"; in practice path coverage is combinatorially infeasible for any function with more than a few decision points (loops in particular create unbounded path counts), so most teams target branch coverage as the practical middle ground and reserve path-level rigor (or MC/DC, one level stronger than branch coverage for compound conditions) for safety- or correctness-critical code paths specifically, rather than the whole codebase uniformly. The other pitfall is treating a coverage PERCENTAGE as a proxy for confidence at all: this example shows 100% statement coverage coexisting with a crash-on-the-most-basic-edge-case bug, so a coverage number answers 'what ran', never 'was the assertion correct'.

Direct answer

The signals worth tracking to measure test automation's impact on release velocity and product quality fall into two groups: velocity signals (how fast the team can ship with confidence) and quality signals (how well defects are being caught before they reach users), and each metric needs a stated limitation alongside it, since no single number tells the whole story.

Structured elaboration

Velocity signals:

• Release cycle time: time from code complete to production release. Indicates whether automation is actually speeding up delivery, not just adding process. Compute as the average or median time across recent releases. Limitation: influenced by many factors besides testing (approval processes, deployment mechanics), so a change here cannot be attributed to automation alone without controlling for those.
• CI feedback time: how long a developer waits for test results after pushing a change. Indicates day-to-day developer experience with the automated suite. Compute as median pipeline duration. Limitation: a fast pipeline that skips important checks looks good on this metric while quietly reducing quality.

Quality signals:

• Defect escape rate: the percentage of defects found in production versus those found pre-release. Indicates whether the test suite is actually catching what matters. Compute as production defects divided by total defects found (pre-release plus production) over a period. Limitation: depends on production issues actually being reported and correctly classified as defects, which can undercount silent failures.
• Automation pass rate (excluding known flaky tests): the percentage of automated test runs that pass. Indicates suite health and trustworthiness. Compute per run, tracked as a trend. Limitation: a suite with weak assertions can show a high, comforting pass rate while not actually testing much of value.
• Mean time to detect (MTTD): how long between a defect being introduced and it being caught, whether by automation or in production. Indicates how quickly the feedback loop closes. Compute as the average, across a recent set of defects, of (time caught minus time introduced), where the introduction point is estimated from the commit or deploy that first shipped the defect (via git bisect or blame) and the catch point is the timestamp it was flagged by a test, code review, or production report. Limitation: hard to measure precisely for defects only caught much later, since pinpointing the introduction moment is itself an estimate.

A weekly or per-release reporting cadence to a product manager or non-technical stakeholder should distill this down to a small, digestible subset, typically test pass rate, defect escape rate, and mean time to detection, presented with a simple threshold or trend (for example, "escape rate above 10% this release, up from 6% last release, worth investigating") rather than the full underlying metric set, since a non-technical audience needs the signal, not the full instrumentation detail.

Worked example

A team reviewing a recent quarter finds: release cycle time dropped from 5 days to 3 days after investing in automation (a velocity win), CI feedback time dropped from 45 minutes to 12 minutes (a strong developer-experience signal), defect escape rate held steady at 8% (automation is maintaining quality while speed improved, not trading one for the other), and automation pass rate sits at 97% excluding a known, tracked set of 6 flaky tests under active remediation (a healthy, trustworthy number specifically because the flaky exceptions are explicitly called out rather than hidden inside the aggregate). Presented to a product manager weekly: "cycle time down 40%, defect escape rate stable at 8%, meaning we are shipping faster without shipping more bugs."

Trade-offs and pitfalls

The most common mistake is tracking only velocity metrics and missing that speed improved because quality checks were quietly reduced, not because automation got genuinely better; defect escape rate specifically exists to catch that trade-off. The second mistake is reporting automation pass rate without excluding or separately tracking known flaky tests, which either hides a real reliability problem inside a falsely comforting aggregate, or makes a genuinely healthy suite look worse than it is because of a small number of known, already-tracked issues.

Approach (brief)

• Use TestNG DataProvider to supply browser names and capabilities.
• Use ThreadLocal<WebDriver> for thread-safe driver per test.
• Annotate tests and DataProvider to run in parallel.

Code example

import org.testng.annotations.*;
import org.openqa.selenium.*;
import org.openqa.selenium.chrome.*;
import org.openqa.selenium.firefox.*;
import org.testng.Assert;

public class CrossBrowserTest {
  private static ThreadLocal<WebDriver> tlDriver = new ThreadLocal<>();

  private WebDriver getDriver() { return tlDriver.get(); }

  @DataProvider(name = "browsers", parallel = true)
  public Object[][] browsers() {
    return new Object[][] {
      {"chrome"}, {"firefox"}
    };
  }

  @BeforeMethod
  @Parameters("browser")
  public void setUp(Object[] params) {
    String browser = (String) params[0];
    WebDriver driver;
    if ("chrome".equalsIgnoreCase(browser)) {
      ChromeOptions opts = new ChromeOptions();
      opts.addArguments("--headless=new"); // example capability
      driver = new ChromeDriver(opts);
    } else {
      FirefoxOptions opts = new FirefoxOptions();
      opts.setHeadless(true);
      driver = new FirefoxDriver(opts);
    }
    tlDriver.set(driver);
    getDriver().manage().window().maximize();
  }

  @Test(dataProvider = "browsers")
  public void sampleE2ETest(String browser) {
    WebDriver d = getDriver();
    d.get("https://example.com");
    Assert.assertTrue(d.getTitle().contains("Example"));
  }

  @AfterMethod
  public void tearDown() {
    if (getDriver() != null) { getDriver().quit(); tlDriver.remove(); }
  }
}

Notes

• Run TestNG with parallel="methods" or "tests" in testng.xml or configure suites in CI.
• Use remote WebDriver/DesiredCapabilities for Selenium Grid/Cloud providers.

Direct answer

Name your actual top two or three non-compensation priorities, in a real priority order, and explain how you'd weigh them against each other when they point in different directions, since with pay and title held equal, that's exactly what the question is testing.

The framework

1. Pick priorities you can rank, not a flat list: product or mission impact, team and manager quality, learning and mentorship, technical or process maturity, and autonomy are the common axes; naming three and ranking them is stronger than naming six with equal weight.
2. Explain how you'd verify each one during the process, not just what you'd ask for in the offer letter: concrete sources like current employees, public engineering or product writing, or specific interview questions.
3. Show you understand the trade-off structure: many real choices are exactly two-offer comparisons where the axes conflict, mission-driven but slower-moving versus fast-growing but less defined, deep mentorship versus direct product impact, nonprofit versus commercial. Naming a real conflict you'd have to resolve is stronger than implying one company would win on everything.
4. Tie the ranking to where you actually are in your career right now, since the right answer changes over time and saying so is a sign of self-awareness, not indecision.

Worked example

Right now my top priority is product impact and ownership of a defined problem, ahead of brand or stability, because I want to build a track record of shipping things that mattered, not just being present. If [Company A] were mission-driven but slower-moving, with a clearer sense of purpose but less individual ownership, and [Company B] were fast-growing with a less defined mission but more scope handed to individual contributors, I'd weigh scope and ownership higher right now and lean toward [Company B], while checking during the process whether its speed comes at the cost of the kind of technical or process maturity I'd need to actually execute well.

Trade-offs and pitfalls

The weak version of this answer treats every factor as equally important, which reads as indecisive rather than thoughtful; the strong version picks a real ranking, names a genuine trade-off between two plausible offers, and explains why that ranking fits where you are right now, not a universal truth.

An 80%-UI-test regression suite is an inverted pyramid: the CI cost and flakiness live disproportionately at the most expensive, least precise level. The goal of a migration plan here is not "delete the UI tests," it is "prove the same business coverage more cheaply, then retire the UI test only once its replacement is proven equivalent."

1. Inventory

Catalog every UI test by what it actually verifies, not by its name: for each test, identify the underlying business assertion (for example, "a discount code reduces the order total correctly") separately from the UI mechanics used to exercise it (clicking through a cart page). Many UI tests will turn out to duplicate the same handful of business assertions through slightly different click paths, which is valuable information for step 2.

2. Selection criteria for migration candidates

Prioritize migrating a UI test to the API level when: (a) its business assertion does not depend on rendering, layout, or client-side interaction behavior itself, meaning the same assertion can be verified by calling the API directly; (b) it is one of several UI tests covering the same underlying business rule, since only one of them needs to stay at the UI level to prove the flow renders correctly, while the rest can move down; (c) it is currently a source of flakiness (timing-dependent, brittle selectors), since those are exactly the tests whose UI framing is adding risk without adding proportional confidence. Leave at the UI level anything whose actual subject IS the rendering or interaction behavior itself (does the button visibly disable during submission, does a validation message appear in the right place).

3. Incremental rollout strategy

Migrate in small batches grouped by business area (checkout, account management), running the new API-level test and the old UI test IN PARALLEL for one full release cycle before retiring the UI test, so you have a real comparison window rather than trusting the migration on faith. Start with the batch identified as most duplicative and most flaky in the inventory, since that batch gives the fastest CI-time win with the least coverage risk.

4. Metrics to track parity

Track, per migrated batch: the number of distinct production defects each UI test has caught historically (from incident postmortems or bug trackers) against whether the new API-level test would have caught the same defects if replayed against the historical bug; overall CI wall-clock time before and after; and flakiness rate (failures that resolve on rerun with no code change) before and after. A drop in caught-defect equivalence for a batch is the signal to keep more of that batch's UI coverage rather than fully retiring it.

5. Risk mitigation during the transition

Never retire a UI test until its replacement has run in parallel for a full cycle with no coverage gap identified; keep a small, deliberately curated UI layer for the handful of assertions that are genuinely about rendering and interaction, since no amount of API-level testing can verify those; and treat the migration as reversible, keeping the retired UI tests in version control (not deleted) for one additional cycle in case a gap surfaces late.

Trade-offs and pitfalls

The main pitfall is treating "80% UI tests" as inherently wrong without checking what those tests actually verify: if a genuinely large share of your business coverage requires rendering and interaction assertions (a highly visual, interaction-heavy product), a smaller UI share than 80% might still be too aggressive a cut. The inventory step exists precisely to avoid migrating tests whose real subject the API level cannot see.

Direct answer

A useful set of test-reliability metrics includes a flakiness score (how often a test's result changes without a real code change), mean time to detect (MTTD) a genuinely failing test, mean time to repair (MTTR) once detected, and the pass-rate trend over time; each needs a precise, computable definition, not just a name, or it can't reliably feed a dashboard or gate a build.

Structured elaboration

Definitions:

• Flakiness score: a common, simple definition is the flip rate, the fraction of consecutive same-commit reruns of a test where its result changed (pass to fail or fail to pass) without any code change in between:
  $\text{flip rate} = \frac{\text{number of result flips observed}}{\text{total reruns observed}}$
  A test with a flip rate near 0 is stable; a test flipping on a meaningful fraction of reruns is flaky enough to warrant quarantine review.

• Mean time to detect (MTTD): the average time between when a test would first genuinely fail due to a real regression and when that failure is actually surfaced and actioned (not merely re-run and ignored):
  $\text{MTTD} = \frac{1}{n}\sum_{i=1}^{n} \left(t_{\text{detected},i} - t_{\text{introduced},i}\right)$
  This depends on being able to identify $t_{\text{introduced}}$ retrospectively (often via bisection once a regression is found), so it's typically computed after the fact from a sample of known regressions rather than in real time.

• Mean time to repair (MTTR) for test failures: the average time from a test failure being flagged to the underlying test (or the code it covers) being fixed:
  $\text{MTTR} = \frac{1}{n}\sum_{i=1}^{n} \left(t_{\text{fixed},i} - t_{\text{flagged},i}\right)$

• Pass-rate trend: the rolling pass rate over a moving window (e.g. trailing 7 days), tracked over time to spot a slow degradation before it becomes a crisis, rather than looking only at a single day's snapshot.

Presentation and alerting: flakiness score feeds a per-test quarantine threshold (above a certain flip rate, flag for quarantine review); MTTD and MTTR feed team-level or suite-level health dashboards, with an alert if either trends upward meaningfully over a rolling window, since a rising MTTD in particular means regressions are sitting undetected longer, a leading indicator of risk rather than a lagging one.

Worked example

A test with 20 observed reruns across recent commits, 3 of which showed a result flip with no underlying code change, has a flip rate of 3/20 = 0.15, likely above a reasonable quarantine threshold (commonly set somewhere in the 0.1-0.2 range depending on the team's risk tolerance) and worth flagging for investigation.

Trade-offs & pitfalls

MTTD specifically is hard to measure precisely in real time (you often only know $t_{\text{introduced}}$ in retrospect, once you've found and bisected a regression), so it's usually a periodically-computed, retrospective metric rather than a live dashboard number; presenting it as if it were live and precise overstates the confidence you actually have in it.

Direct answer

I'd answer with a staged range and named milestones rather than a single date, and I'd be explicit that doing the normal case and handling it when it goes wrong are two different bars, with the second one usually taking longer and being the real definition of unsupervised.

Structured elaboration

• Break readiness into distinct levels with visible evidence for each, not one line. Something like: getting oriented, practicing in a safe or low-stakes setting, doing real work with someone checking my output, working independently on the common path, and finally handling it independently including when things break. Each level should have something concrete that shows I've reached it, not just a self-assessment.
• Give a range with a confidence qualifier, not a false-precise date. Something like "probably four to six weeks before I can handle the common path on my own, and I'd want a few more weeks with someone reachable before I'd call myself fully unsupervised on the failure cases, since that's usually where the real ramp time goes."
• Separate doing the task from handling it when it breaks. These are genuinely different skills: the first is often learnable quickly by following a pattern, the second requires having actually seen or understood the failure modes, which usually takes longer and is what "unsupervised" really has to mean.
• Name what actually shortens the ramp, versus what doesn't. Access to someone who can unblock the first few hard problems quickly, a safe environment to practice in, and exposure to past incidents or failure history genuinely help. Just reading more documentation on my own past a certain point mostly doesn't.
• Set checkpoints, not just an end date. Agreeing on visible milestones along the way means both of us can tell early if the estimate is drifting, instead of only finding out at the original deadline.

Worked example

When I took over an unfamiliar production system with no formal handoff, my manager asked how long before they could stop checking in on it. I laid it out in stages rather than a date: two weeks to understand the system's normal operation and get comfortable reading its monitoring, then two to three weeks of handling routine changes with someone reviewing before they went out, and then a final stretch, harder to predict exactly, before I'd be confident handling an actual incident without help, since I hadn't seen one yet. I gave a range of six to nine weeks total, with the caveat that the second half depended on whether anything actually broke during that window for me to learn from, since reading about failure modes and living through one aren't the same thing. We agreed on a checkpoint at three weeks to see whether the first stage was tracking, which it was, and by week seven an incident actually happened, I handled it with someone reachable but not directly involved, and that became the real evidence that closed out the estimate rather than the calendar date alone.

Trade-offs and pitfalls

Giving a single confident date to sound decisive is a common trap, and it backfires badly when it slips, since it reads as either poor judgment or unmet expectations. Overhedging is the opposite failure: an answer so qualified it gives the manager nothing usable to plan around. The most consequential mistake is declaring readiness once the routine case is handled while quietly ignoring the failure-handling gap, since that's exactly the part that shows up as a real incident later, at the worst possible time to discover you weren't actually ready.

Direct answer

Capture the browser's console log and network activity during the test run using Selenium 4's Chrome DevTools Protocol (CDP) integration, or the simpler driver.get_log('browser') API for console-only capture, and attach both to the CI failure report so a teammate can see the client-side JavaScript error or failed network call without ever needing to reproduce the flaky run locally.

Structured elaboration

Two complementary capture mechanisms exist, at different levels of depth. driver.get_log('browser') is the lighter-weight option: it returns the browser's own console log entries (JavaScript errors, warnings, console.log output) as a simple list, with no setup beyond enabling browser logging in the driver's capabilities. Selenium 4's native CDP integration (driver.execute_cdp_cmd(...), or the higher-level driver.get_log/Network domain listeners in newer bindings) goes further, giving access to the full Chrome DevTools Network domain, request/response headers, timing, and failed requests, which is what you need when the suspected cause is a failed or slow NETWORK call rather than a pure JavaScript error.

The most useful log types for this scenario specifically: console logs (to catch a JavaScript error that prevented an event listener from attaching, matching this question's premise), and network/CDP Network.responseReceived/Network.loadingFailed events (to catch a background request that failed or never completed, which can silently break functionality that depended on its result without ever showing up as a visible error). Attaching both to the CI report, alongside the existing screenshot-on-failure artifact, means a teammate reading the failure the next morning has the same diagnostic picture you would have had watching it fail live.

Worked example

def capture_browser_diagnostics(driver):
    console_logs = driver.get_log('browser')
    driver.execute_cdp_cmd('Network.enable', {})
    # In a real run, CDP network events are captured via a listener registered before
    # navigation; a fuller CDP-based capture pipeline records request/response pairs as
    # they occur rather than fetching them after the fact.
    return {
        "console": console_logs,
        # network events would be appended here by the listener during the test
    }

Attaching this to the CI report (as a JSON artifact alongside the existing screenshot) means a JavaScript error or a failed background request is visible without needing to reproduce the failure interactively.

Trade-offs and pitfalls

The most common mistake is capturing ONLY the console log and assuming that covers "client-side issues": a failed background network call that the application handles silently (retries without logging, or simply drops the result) produces NO console error at all, so relying on console logs alone would miss exactly the class of bug this question's premise describes. A second pitfall is enabling verbose CDP network capture on every single test run regardless of pass/fail, which adds real overhead at scale; a common middle ground is capturing lightweight console logs always, and only spinning up the heavier CDP network capture path when a test actually fails, similar to how screenshot capture is usually failure-triggered rather than always-on.