Cross-Browser and Cross-Platform Testing Questions

Direct answer

Do not compare absolute timings, and do not assert an order the specifications leave open. Instead, record every callback into a log of labelled events, wait until all expected labels have arrived (a settle barrier driven by a counter, not a sleep), and assert relative order by index in that log. Split the rules into two groups: guarantees every engine must meet (synchronous code first, microtasks before any timer, equal-delay timers in scheduling order, a microtask queued inside a timer callback running before the next timer) and per-engine expectations for orderings the standards leave open (kept as data in a config object). Run the same page in Playwright's Chromium, Firefox and WebKit projects, repeat each test ten times, and keep the check in CI knowing that Playwright's WebKit build is not real Safari.

Why the orders differ

JavaScript runs on one thread with a microtask queue (promise callbacks, queueMicrotask) that drains completely after each running piece of code, and several task queues (timers, message events, rendering callbacks such as requestAnimationFrame) from which the browser's event loop picks the next task. The HTML specification fixes the microtask rules and the order of timers with equal delays, but it does not require any particular order between different task sources, such as a MessageChannel message versus a setTimeout(fn, 0) timer, or where a requestAnimationFrame callback lands relative to timers. Engines choose their own scheduling there. A flow that accidentally depends on such an order works in one engine and breaks in another, which is the reported bug.

Terms the probe uses. A task source is a category of scheduled work (timers, messages, rendering) that the browser keeps in its own queue. MessageChannel creates two linked ports; posting on one port fires a message event on the other as a separate task. requestAnimationFrame asks the browser to call a function just before the next screen repaint. process.nextTick and setImmediate exist only in Node: nextTick runs a callback as soon as the current operation finishes, before the event loop moves on, and setImmediate runs it in the loop's check phase, after the poll phase that handles I/O. A settle barrier is a counter that starts at the number of callbacks scheduled and resolves a promise when the last one has fired.

A five-line example shows the basic order:

// tiny.mjs
console.log('A: synchronous');
setTimeout(() => console.log('D: timer, delay 0'), 0);
Promise.resolve().then(() => console.log('C: promise callback'));
console.log('B: synchronous');

Run in Node 20.18, it prints A, B, C, D in that order. The two synchronous lines finish first (A, B). When the running code ends, the microtask queue drains, so the promise callback runs (C). Only then does the event loop take the next task, the timer (D), although its delay is 0. The HTML specification gives browsers the same rule: microtasks run when the call stack empties, before the next task. The probe below repeats this with more kinds of callback and checks which parts of the order are guaranteed.

What to log and what to assert

The probe schedules the same work in every environment and logs a label when each callback fires:

// probe.mjs
// Runs in a browser page (via page.evaluate) and in Node. Self-contained on purpose.
export function runProbe() {
  return new Promise((resolve) => {
    const log = [];
    let pending = 0;
    const track = (label, schedule) => {
      pending++;
      schedule(() => {
        log.push(label);
        if (--pending === 0) resolve(log); // settle barrier: every scheduled callback has fired
      });
    };

    log.push('sync-start');
    track('micro-promise', (cb) => Promise.resolve().then(cb));
    track('micro-queue', (cb) => queueMicrotask(cb));
    track('timeout-a0', (cb) => setTimeout(cb, 0));
    track('timeout-b0', (cb) => setTimeout(cb, 0));
    track('timeout-c10', (cb) => setTimeout(cb, 10));
    // Microtask queued from inside a timer callback: must run before the next timer.
    track('timeout-d5', (cb) => setTimeout(() => { Promise.resolve().then(() => log.push('micro-in-d5')); cb(); }, 5));
    track('timeout-e5', (cb) => setTimeout(cb, 5));
    if (typeof MessageChannel !== 'undefined') {
      track('message', (cb) => { const ch = new MessageChannel(); ch.port1.onmessage = () => { ch.port1.close(); cb(); }; ch.port2.postMessage(0); });
    }
    if (typeof requestAnimationFrame === 'function') track('raf', (cb) => requestAnimationFrame(() => cb()));
    if (typeof setImmediate === 'function') track('immediate', (cb) => setImmediate(cb));
    if (typeof process !== 'undefined' && process.nextTick) track('nexttick', (cb) => process.nextTick(cb));
    log.push('sync-end');
  });
}

How track(label, schedule) works: schedule is a function that arranges for its argument to be called later ((cb) => setTimeout(cb, 0)). track adds 1 to pending, then passes schedule a wrapper that logs the label, subtracts 1, and resolves the whole promise when pending reaches 0. The test therefore waits exactly as long as the slowest callback and no longer. Labels that a primitive cannot produce (no MessageChannel in some hosts, no setImmediate in browsers) are simply not scheduled, so one probe file runs everywhere.

The rules live in a second file. invariantFailures returns a list of broken rules, so a failure message names the exact rule; ENGINE_RULES holds the allowed message-versus-timer relation per engine:

// invariants.mjs
// Order rules that every engine must satisfy, plus per-engine rules kept as data.
const at = (log, label) => log.indexOf(label);

// Engines may legitimately disagree about message-channel vs zero-delay timer order.
export const ENGINE_RULES = {
  chromium: { messageVsTimeout: ['after'] },
  firefox: { messageVsTimeout: ['after'] },
  webkit: { messageVsTimeout: ['before', 'after'] },
};

export function invariantFailures(log) {
  const fails = [];
  const need = (ok, msg) => { if (!ok) fails.push(msg); };
  const macro = ['timeout-a0', 'timeout-b0', 'timeout-d5', 'timeout-e5', 'timeout-c10'];
  need(at(log, 'sync-end') === at(log, 'sync-start') + 1, 'synchronous code must finish before any callback');
  for (const m of ['micro-promise', 'micro-queue']) {
    need(at(log, m) > at(log, 'sync-end'), `${m} must run after the synchronous code`);
    for (const t of macro) need(at(log, m) < at(log, t), `${m} must run before ${t}`);
  }
  need(at(log, 'micro-promise') < at(log, 'micro-queue'), 'microtasks run in the order they were queued');
  need(at(log, 'timeout-a0') < at(log, 'timeout-b0'), 'equal-delay timers run in scheduling order');
  need(at(log, 'timeout-d5') < at(log, 'micro-in-d5') && at(log, 'micro-in-d5') < at(log, 'timeout-e5'),
    'a microtask queued inside a timer callback runs before the next equal-delay timer');
  return fails;
}

export function messageRelation(log) {
  return at(log, 'message') < at(log, 'timeout-a0') ? 'before' : 'after';
}

Why these assertions and not others: each guaranteed rule compares two labels by their position in the log, so machine speed cannot change the verdict. The two 5 ms timers (timeout-d5, timeout-e5) share a delay, so their relative order is fixed by scheduling order; the 10 ms timer is deliberately not compared with them, because two timers with different delays can swap when the page stalls between scheduling them. requestAnimationFrame is only required to come after the synchronous code: its slot among timers varies between runs even inside one engine.

Making it deterministic and non-flaky

1. Settle barrier, not a sleep. The probe resolves only when its counter reaches zero, so the test never reads the log early and never waits a fixed time. A hung callback fails by the test timeout rather than silently passing.
2. Relative order by index. Compare positions in the log, never durations or timestamps.
3. Open orders become data. Where engines legitimately differ, the allowed relations sit in ENGINE_RULES. If WebKit is seen producing both orders across repeated runs, as it is here, its entry allows both; if an engine is stable, pin its single value so a change in either direction is noticed.
4. Repeat. Ten repeats per engine surface an order that is only usually true. A rule that fails in one repeat out of ten is a real defect or a rule that was never guaranteed, and either way it should not stay in the suite as written.
5. Isolate the page. The page is created fresh with setContent, with no network and no app code, so the check fails only because of scheduling.

The Playwright project runs the same spec in three engines with ten repeats each:

// playwright.config.mjs
import { defineConfig, devices } from '@playwright/test';

export default defineConfig({
  testDir: '.',
  testMatch: 'order.spec.mjs',
  repeatEach: 10,           // repeat each test 10 times in every project
  workers: 1,               // one page at a time keeps timer pressure comparable
  timeout: 15_000,
  reporter: 'line',
  projects: [
    { name: 'chromium', use: { ...devices['Desktop Chrome'] } },
    { name: 'firefox', use: { ...devices['Desktop Firefox'] } },
    { name: 'webkit', use: { ...devices['Desktop Safari'] } },
  ],
});

// order.spec.mjs
import { test, expect } from '@playwright/test';
import { runProbe } from './probe.mjs';
import { ENGINE_RULES, invariantFailures, messageRelation } from './invariants.mjs';

test('scheduling order keeps the guaranteed rules', async ({ page, browserName }) => {
  await page.setContent('<!doctype html><title>order</title>');
  const log = await page.evaluate(runProbe); // resolves only when every callback has fired
  expect(invariantFailures(log), log.join(' > ')).toEqual([]);

  // rAF only has to come after the synchronous code; its slot among timers is not asserted.
  expect(log.indexOf('raf')).toBeGreaterThan(log.indexOf('sync-end'));

  // Engine-specific expectation, read from data rather than hard-coded in the test.
  expect(ENGINE_RULES[browserName].messageVsTimeout).toContain(messageRelation(log));
});

The same probe runs in Node, where it also proves the rule set can fail, by feeding it a doctored log in which a timer jumped ahead of a microtask:

// node-run.mjs
import { runProbe } from './probe.mjs';
import { invariantFailures } from './invariants.mjs';

const log = await runProbe();
console.log(log.join(' > '));
console.log('failures:', invariantFailures(log));

// Teeth check: a doctored log where a timer jumped ahead of a microtask must be reported.
const bad = log.filter((l) => l !== 'timeout-a0');
bad.splice(bad.indexOf('micro-promise'), 0, 'timeout-a0');
console.log('doctored log failures:', invariantFailures(bad));

The last four lines of node-run.mjs exist because a rule set that has never failed has not been shown to work. They edit the real log so a timer jumps ahead of a microtask and require invariantFailures to report it.

Worked result

Run in the Playwright 1.48.2 image (Chromium 130.0, Firefox 131.0, WebKit 18.0, Node 20.18, probe loaded as an ES module), node node-run.mjs printed:

sync-start > sync-end > micro-promise > micro-queue > nexttick > immediate > timeout-a0 > timeout-b0 > message > timeout-d5 > micro-in-d5 > timeout-e5 > timeout-c10
failures: []
doctored log failures: [
  'micro-promise must run before timeout-a0',
  'micro-queue must run before timeout-a0'
]

npx playwright test reported 30 passed (3 engines x 10 repeats) on each of three consecutive invocations. In separate exploratory loops on the same page, the observed logs showed what the config encodes: Chromium and Firefox put message after timeout-a0 in every observed run, WebKit sometimes put message first, and raf appeared at different slots from run to run in all three engines, which is why it has no position rule. Node-only labels (nexttick, immediate) are logged but not asserted; they are absent from the browser rules and their placement relative to promise callbacks is an environment detail the guard does not depend on.

Reading the Node log label by label:

1. sync-start, sync-end: the probe function runs to its end first; everything else was only queued.
2. micro-promise, micro-queue: the microtask queue drains in the order the callbacks were queued, before any timer.
3. nexttick comes after the promise callbacks here, which can look backwards because nextTick is described as running first. Node's documentation explains it: in a CommonJS file the next-tick queue runs before promise callbacks, but an ES module is processed as part of the microtask queue, so Node is already draining microtasks and the promise callbacks go first. probe.mjs is an ES module. The same three lines show both orders:

// ticks.cjs (the same three lines are saved as ticks.mjs)
Promise.resolve().then(() => console.log('promise'));
process.nextTick(() => console.log('nextTick'));
console.log('sync');

Run in Node 20.18, ticks.cjs prints sync, nextTick, promise, and ticks.mjs prints sync, promise, nextTick.
4. immediate precedes timeout-a0. Node's event-loop guide says that when setTimeout(fn, 0) and setImmediate are both called from the main module, the order is non-deterministic and depends on the performance of the process. That is why the probe records immediate and asserts nothing about it.
5. timeout-a0, timeout-b0: equal delays run in scheduling order, a guaranteed rule.
6. message lands between timeout-b0 and timeout-d5. Node delivers message ports through its own machinery, so this slot is a Node detail. The specifications do not order a message task against timers, which is why the browser projects check it against ENGINE_RULES and not against a fixed position.
7. timeout-d5, micro-in-d5, timeout-e5: the d5 callback queued a promise callback, and the microtask queue drains straight after that callback, before the next timer is taken.
8. timeout-c10 is last only because nothing stalled; it is not compared with the 5 ms timers.

If you keep only two rules from this, keep these: synchronous code finishes before any callback, and every queued microtask runs before the next timer or message task. The other assertions follow from the same two ideas.

What the guard does not prove

Playwright's WebKit is a build of the WebKit engine running on Linux, not Safari on iOS or macOS, so it can miss behaviour that comes from Safari's own scheduling or from iOS itself. Keep the production bug as a regression case here (it catches engine-level scheduling), and add a small real-Safari check on a cloud device or a macOS runner for the specific flow that broke, before relying on this alone. Also fix the app: the durable remedy is that the flow stops depending on an order the specifications leave open, for example by chaining promises or awaiting an explicit completion signal instead of racing a timer against a message.

Running the code

# put the files above in an empty folder (ticks.cjs and ticks.mjs hold the same three lines)
echo '{"type":"module"}' > package.json
docker run --rm --ulimit core=0 -v "$PWD":/w -w /w mcr.microsoft.com/playwright:v1.48.2-jammy \
  sh -c 'timeout 200 sh -c "npm i --no-audit --no-fund @playwright/test@1.48.2 && node tiny.mjs && node ticks.cjs && node ticks.mjs && node node-run.mjs && npx playwright test"'

Direct answer

For a small startup, run browser tests in containers on every PR and rent a cloud device farm for the small set of checks that need real Safari or real phones. If the suite is already written in Selenium, a Selenium Grid in Docker is the container half of that answer; if it is not, use Playwright's own browsers in the same containers and skip the Grid. The reason is cost shape: containers use CI capacity you already pay for and give the fast, reproducible feedback a PR needs, while the one thing containers cannot do is run real Safari or a real device, and a startup should not buy or maintain hardware to cover that small slice. Choose the farm alone only if nobody on the team has time to maintain any test infrastructure at all.

Terms. A container is a lightweight, isolated package of an application and everything it needs, started from an image. Selenium Grid is a server that receives WebDriver commands (the standard remote-control protocol for browsers) and routes each test session to a browser on one of several machines (nodes). In its dynamic mode the Grid starts a fresh container for each session and removes it afterwards. A session is one browser opened by one test, so concurrent sessions are tests running at the same moment. Capabilities are the key-value settings a test sends to say which browser, version and operating system it wants. Docker Compose files are YAML files that describe several containers and how they connect, and Kubernetes is a system that schedules containers across many machines. A cloud device farm is a hosted service that rents real or virtual browsers and phones by the minute or by parallel-session count. Real devices are physical phones.

Comparison

Recommendation and costs

1. Every PR: containerised Chromium and Firefox, plus Playwright's WebKit build if you use Playwright (it is not a Selenium Grid option). That is the fast, deterministic gate, and it costs CI minutes only.
2. Nightly and pre-release: a small cloud plan (a few concurrent sessions) running the smoke subset on real Safari on macOS and one real iPhone and one real Android phone. The point is the hardware-only defects, not volume.
3. Reassess when: the monthly farm bill exceeds the cost of owning and maintaining a few devices (include the engineer-hours; a lab usually loses for a startup), or the farm's queue time starts to slow releases. Move more of the farm's work into containers if the defects it finds turn out to be reproducible there.

Cost model, showing the break-even shape: let P be the farm plan price per month, c the CI price per hour, h the hours of Grid-backed CI per month, and m the monthly cost of the engineer-hours spent maintaining the Grid. Containers are cheaper than the farm while c x h + m < P.

Worked example with assumed inputs (not vendor quotes): a farm plan with enough concurrent sessions to run the PR suite costs P = 1,200 per month; the CI machines cost c = 1 per hour and run h = 200 hours; maintaining the Grid takes 5 engineer-hours at 60 per hour, so m = 300. Then c x h + m = 200 + 300 = 500, below 1,200, so containers win. If upkeep grows to 20 hours, m = 1,200 and the total is 200 + 1,200 = 1,400, above 1,200, so the farm wins. The break-even upkeep is (1,200 - 200) / 60, about 16.7 hours a month. For a small team m (an engineer's time) tends to dominate, so the usual pattern is containers for volume and a small paid farm for fidelity, not a larger self-hosted Grid.

Pitfalls

• Running every test on the farm: the latency and per-minute cost make PR feedback slow and expensive.
• Expecting Docker to cover Safari: it covers Chromium-based and Gecko-based browsers; Playwright WebKit in Docker is an engine proxy and not iOS or macOS Safari.
• A Grid without video and logs: a failure in a remote container is hard to diagnose; budget storage for artifacts. Note that the Selenium Docker project says video recording is not supported for headless browsers.
• Not pinning image versions, which gives up the reproducibility that was the reason to use containers.

Direct answer

Verify the component in layers, cheapest first. (1) Resize it in the browser and in DevTools device mode, and run an automated check across the three browser engines (Chromium, Firefox and WebKit, the engine behind Safari) at a handful of widths. (2) Inspect what emulation cannot show on a few real phones or cloud-hosted real devices. (3) Choose coverage by risk and real traffic, not by a device list. Emulation is good enough for layout logic: breakpoints (the widths where CSS changes the layout), wrapping, overflow and keyboard order. You need a real device for anything that depends on the actual mobile browser, the touch hardware or the operating system: iOS Safari's collapsing address bar and viewport units, the on-screen keyboard resizing the page, real font rendering, momentum scrolling (scrolling that keeps gliding after the finger lifts) and genuine touch gestures.

Step 1: Verify layout in emulation and in three engines

1. Resize by hand first. Drag the window from 320 CSS px (CSS pixels, the browser's logical pixels, not device hardware pixels) up to 1440 and watch for horizontal scrollbars, clipped text, overlapping items and awkward gaps between breakpoints. Most responsive bugs live between the named breakpoints, not at them.
2. Use DevTools device mode for fast iteration. Chrome documents it as a "first-order approximation" of a mobile device: it simulates viewport size, pixel ratio, touch input and CPU and network throttling, but it does not run the page on a mobile device, and the docs say there are aspects of mobile devices DevTools will never simulate (Chrome DevTools: device mode). That one sentence is the dividing line for the whole answer.
3. Automate the layout rules so the check runs on every pull request in all three engines. The test below renders the component at 320, 375, 768 and 1280 px and asserts the number of grid columns, that there is no horizontal overflow, that every button is at least 44 by 44 CSS px, and that the Tab key visits each card in order. The projects include Playwright's Pixel 5 and iPhone 13 device descriptors, which set viewport, user agent and touch support. The iPhone descriptor runs Playwright's WebKit build, which is not iOS Safari: it is the same engine family on Linux with an iPhone-sized viewport, so it catches engine-level layout differences but not iOS-specific browser behaviour.

The component is a card grid whose columns come from repeat(auto-fit, minmax(240px, 1fr)), so it needs no media query. In plain words: make as many columns as fit, each at least 240 px wide, and let the columns share leftover space equally (1fr is one equal share). A narrow screen therefore gets one column and a wide one gets more, with no breakpoint in the CSS:

// card-grid.mjs
export const html = `<!doctype html><meta name="viewport" content="width=device-width, initial-scale=1"><title>cards</title>
<style>
  body { margin: 0; padding: 16px; font: 16px system-ui, sans-serif; }
  .grid { display: grid; gap: 16px; grid-template-columns: repeat(auto-fit, minmax(240px, 1fr)); }
  .card { border: 1px solid #888; padding: 12px; min-width: 0; overflow-wrap: anywhere; }
  .card button { min-height: 44px; min-width: 44px; padding: 0 16px; font: inherit; }
  .card button:focus-visible { outline: 3px solid #05f; }
</style>
<main class="grid">
  ${[1, 2, 3, 4, 5].map((i) => `<article class="card"><h2>Card ${i}</h2><p>averyveryverylongunbrokenorderidentifier-${i}-0123456789abcdef0123456789abcdef</p><button>Open</button></article>`).join('')}
</main>`;

// playwright.config.mjs
import { defineConfig, devices } from '@playwright/test';

export default defineConfig({
  testDir: '.',
  testMatch: 'card-grid.spec.mjs',
  workers: 1,
  timeout: 20_000,
  reporter: 'list',
  projects: [
    { name: 'desktop-chromium', use: { ...devices['Desktop Chrome'] } },
    { name: 'desktop-firefox', use: { ...devices['Desktop Firefox'] } },
    { name: 'desktop-webkit', use: { ...devices['Desktop Safari'] } },
    { name: 'pixel-5', use: { ...devices['Pixel 5'] } },          // Chromium with mobile emulation
    { name: 'iphone-13', use: { ...devices['iPhone 13'] } },      // WebKit with iPhone viewport, UA and touch
  ],
});

// card-grid.spec.mjs
import { test, expect } from '@playwright/test';
import { html } from './card-grid.mjs';

const WIDTHS = [320, 375, 768, 1280];
// Expected columns for repeat(auto-fit, minmax(240px, 1fr)): body padding 16 + 16, gap 16.
const expectedColumns = (w) => Math.floor((w - 32 + 16) / (240 + 16));

for (const width of WIDTHS) {
  test(`grid holds at ${width}px`, async ({ page }, info) => {
    await page.setViewportSize({ width, height: 800 });
    await page.setContent(html);
    const m = await page.evaluate(() => {
      const grid = document.querySelector('.grid');
      const cols = getComputedStyle(grid).gridTemplateColumns.split(' ').length;
      const btn = document.querySelector('button').getBoundingClientRect();
      return {
        cols,
        overflowX: document.documentElement.scrollWidth - document.documentElement.clientWidth,
        btnW: Math.round(btn.width), btnH: Math.round(btn.height),
        coarse: matchMedia('(pointer: coarse)').matches,
        hoverNone: matchMedia('(hover: none)').matches,
      };
    });
    console.log(`${info.project.name} ${width}px cols=${m.cols} overflowX=${m.overflowX} button=${m.btnW}x${m.btnH} coarse=${m.coarse} hoverNone=${m.hoverNone}`);
    expect(m.cols).toBe(expectedColumns(width));
    expect(m.overflowX).toBeLessThanOrEqual(0);       // no horizontal scrollbar
    expect(m.btnH).toBeGreaterThanOrEqual(44);        // house target; WCAG 2.2 AA minimum is 24
    expect(m.btnW).toBeGreaterThanOrEqual(44);
  });
}

test('keyboard reaches every button in order', async ({ page, browserName }) => {
  await page.setContent(html);
  const seen = [];
  for (let i = 0; i < 5; i++) {
    await page.keyboard.press('Tab');
    seen.push(await page.evaluate(() => document.activeElement.closest('.card')?.querySelector('h2').textContent));
  }
  console.log(`${browserName} tab order: ${seen.join(', ')}`);
  expect(seen).toEqual(['Card 1', 'Card 2', 'Card 3', 'Card 4', 'Card 5']);
});

The test prints three values per width that need a key. overflowX is the page's scrollable width minus its visible width; 0 means no horizontal scrollbar. coarse is whether the media query (pointer: coarse) matches, meaning the main input is imprecise, such as a finger. hoverNone is whether (hover: none) matches, meaning the main input cannot hover, as with touch. WCAG is the Web Content Accessibility Guidelines, and AA is its middle conformance level. A :focus-visible style is applied when the browser decides a focus ring helps the user, typically after keyboard focus.

The expected column count is computed, not guessed. N columns need N x 240 px plus (N - 1) gaps of 16 px, and the available width is W - 32, so N fits when N x 240 + (N - 1) x 16 <= W - 32, which rearranges to N <= (W - 32 + 16) / (240 + 16). With 16 px body padding on each side and a 16 px gap, the usable width is W - 32 and the number of 240 px columns that fit is floor((W - 32 + 16) / (240 + 16)): 320 px gives floor(304 / 256) = 1, 375 gives floor(359 / 256) = 1, 768 gives floor(752 / 256) = 2, and 1280 gives floor(1264 / 256) = 4 (four columns need 4 x 240 + 3 x 16 = 1,008 px of the 1,248 available, and five would need 1,264). Run in the Playwright 1.48.2 container (Chromium 130, Firefox 131, WebKit 18.0), all 25 tests pass (5 projects x 4 widths = 20 layout tests, plus one Tab-order test per project) and the log prints one line for each of them; 8 lines are excerpted here, and every omitted test is held to the same assertions:

desktop-chromium 320px cols=1 overflowX=0 button=72x44 coarse=false hoverNone=false
desktop-chromium 768px cols=2 overflowX=0 button=72x44 coarse=false hoverNone=false
desktop-chromium 1280px cols=4 overflowX=0 button=72x44 coarse=false hoverNone=false
desktop-firefox 320px cols=1 overflowX=0 button=72x44 coarse=false hoverNone=true
desktop-webkit 320px cols=1 overflowX=0 button=74x44 coarse=false hoverNone=false
pixel-5 320px cols=1 overflowX=0 button=72x44 coarse=true hoverNone=true
iphone-13 1280px cols=4 overflowX=0 button=74x44 coarse=true hoverNone=true
chromium tab order: Card 1, Card 2, Card 3, Card 4, Card 5
25 passed (4.1s)

Two details in that output are exactly why real checks beat assumptions. The button is 72 px wide in Chromium and Firefox but 74 px in WebKit: the same CSS produces a slightly different size because each engine lays out text differently. And headless Firefox in this image reports (hover: none) as true on a desktop viewport, while headless Chromium and WebKit report false: any hover-only affordance (a menu that opens only on :hover) has to be guarded with @media (hover: hover) and tested in more than one engine rather than assumed.

Step 2: Where emulation is enough and where a real device is needed

Run the real-device pass as a short, scripted spot check, not a full regression: the three or four interactions that differ on phones (open, scroll, type in a field, rotate) on one recent iPhone and one mid-range Android phone. A cloud device farm (a service that rents you real phones over the network) is the right tool if the team owns no devices, because the spot check is a handful of minutes per release.

Step 3: Decide which viewports and platforms come first

Rank by traffic times risk, using the product's own analytics rather than a generic list:

1. Pull the last 30 to 90 days of sessions by browser, operating system and viewport width from the analytics tool. Cover the combinations that make up the top share of sessions first.
2. Add risk: any engine where the component uses a feature with known gaps (container queries, dvh units, :has()), and always include WebKit because WebKit is the engine behind Safari on iPhone.
3. Pick three widths that sit just inside breakpoint edges (for example 320 as the narrowest realistic phone, one width just under the tablet breakpoint, one just over) plus a wide desktop, not every device size.
4. Run the fast set (the three engines at those widths) on every pull request and the real-device spot check before release.

Commit to this default: three engines at four widths on every pull request, real devices at release, and re-rank the list each quarter from fresh traffic data. It flips toward heavier real-device coverage when the component is touch-first (a swipe carousel, a drag handle), handles text entry near the screen bottom, or the traffic is mostly mobile.

Touch targets and keyboard use

The test above already asserts 44 by 44 CSS px for the button. That is the team's own target, deliberately larger than the WCAG 2.2 AA minimum (Success Criterion 2.5.8, Target Size Minimum), which requires pointer targets of at least 24 by 24 CSS px unless a spacing, equivalent-control, inline, user-agent or essential exception applies (W3C: Target Size (Minimum)). For the keyboard, the Tab order test proves every control is reachable in visual order, and :focus-visible gives a ring only for keyboard users. Do the rest of the accessibility audit (screen readers, contrast) as its own pass.

Pitfalls

• Treating "it looks right in the Chrome device toolbar" as cross-browser evidence: it is one engine.
• Using Playwright's iPhone 13 run as proof of iOS Safari behaviour. It is WebKit with an iPhone-shaped viewport.
• Testing only at named breakpoints, so the bug at 600 to 700 px is never seen.
• Choosing devices by what the team owns instead of what visitors use.

Running the code

# in an empty folder holding the three files above
echo '{"type":"module"}' > package.json
docker run --rm --ulimit core=0 -v "$PWD":/w -w /w mcr.microsoft.com/playwright:v1.48.2-jammy \
  sh -c 'timeout 200 sh -c "npm i --no-audit --no-fund @playwright/test@1.48.2 && npx playwright test"'

Direct answer

Rely on feature detection in both application code and tests. Feature detection asks the browser "can you do this thing?" and acts on the answer. User-agent (UA) sniffing reads the navigator.userAgent string, which names a browser and version, and guesses what that browser can do. The guess goes wrong because UA strings are inherited (browsers copy tokens from older browsers), spoofed, and frozen in time (browsers have deliberately stopped updating parts of the string, such as the minor version and the OS version), whereas a capability check tests the actual thing you are about to use. Keep UA parsing for analytics and for narrowly scoped, time-limited workarounds of a known engine bug that has no detectable symptom.

An example of each

Feature detection (application code). Run non-urgent work when the browser is idle, using requestIdleCallback where it exists and a timer where it does not:

const whenIdle = 'requestIdleCallback' in window
  ? (fn) => requestIdleCallback(fn)
  : (fn) => setTimeout(fn, 1);

In CSS the equivalent is a feature query, which applies rules only if the browser understands them:

@supports (display: grid) { .layout { display: grid; gap: 2rem; } }

UA sniffing (the thing to avoid).

const isSafari = /Safari/.test(navigator.userAgent);
if (isSafari) { useFallbackScheduler(); }   // a stand-in that uses timers instead of idle callbacks

The two feature checks above test the capability itself, so they stay correct in every engine. The sniff only appears to work on one engine: Chrome's UA string also contains the word Safari, so Chrome takes the Safari branch too, and it is wrong there. The next section shows the evidence.

What goes wrong with sniffing at scale

Run in the Playwright 1.48.2 container against its Chromium, Firefox and WebKit builds, this probe prints (identical on 3 runs):

chromium  {"uaSaysSafari":true,"uaSaysChrome":true,"uaSaysGecko":true,"requestIdleCallback":true,"hasSelector":true}
firefox   {"uaSaysSafari":false,"uaSaysChrome":false,"uaSaysGecko":true,"requestIdleCallback":true,"hasSelector":true}
webkit    {"uaSaysSafari":true,"uaSaysChrome":false,"uaSaysGecko":true,"requestIdleCallback":false,"hasSelector":true}

Reading the output:

• /Safari/ matches Chromium's UA as well as WebKit's, so isSafari above is true for Chrome. MDN says the same: Chrome reports both Chrome and Safari, and Safari detection has to check for the absence of Chrome/ and Chromium/.
• /Gecko/ is true in all three engines, because Chromium and WebKit UA strings carry a legacy "like Gecko" token (Gecko is Firefox's engine; the others added the phrase long ago so that sites written for Gecko would serve them the same pages). MDN names this as a source of false positives for Gecko detection.
• The one real capability gap in the output, requestIdleCallback, follows the engine (absent in this Playwright WebKit build) and says nothing about the UA token. A feature check handles it with one line, correctly, and if Safari adds the API later the check starts taking the fast path without a code change. A sniff stays wrong until someone edits it.

The wider problems, per MDN's guidance on browser detection:

1. Spoofing and mixed signals. Browsers routinely pretend to be other browsers, so the string is not a reliable statement of identity.
2. Stale assumptions. A browser can gain or lose a feature in any release; a name or version table written last year is already out of date, and large rule tables nobody owns decay silently.
3. Missed capability. Users whose browser is not on your list are sent to a slow fallback even though they could run the fast path.
4. Bad device inference. MDN says never to use the OS token to decide mobile, tablet or desktop: Android runs on tablets as well as phones. If a UA check is unavoidable, MDN's rule is to look for Mobi (/Mobi/.test(navigator.userAgent), the token phones add to the string) and to serve the desktop site to any device without it, which should support touch input anyway because desktop devices may have touchscreens. Even that is a fallback: prefer navigator.maxTouchPoints (the number of simultaneous touch points the device supports, 0 when there is no touch input), media queries and flexible layout.
5. Client hints are not a way out. User-agent client hints are a newer mechanism (the Sec-CH-UA-* request headers and the navigator.userAgentData object) that report the browser brand, platform and mobile flag in separate structured fields. They are harder to spoof in Blink browsers, but MDN still warns against changing site functionality based on them.

In tests

• Assert capabilities, not names. To decide whether a test applies, check the feature the test needs (await page.evaluate(() => 'requestIdleCallback' in window)) and skip with a reason when it is absent. Tying a skip to a UA regex quietly shrinks coverage when a UA changes.
• Skip by the engine you launched, not by sniffing. Playwright tells a test which project it is running under (browserName), so test.skip(browserName === 'webkit', 'reason and ticket link') records an engine-specific skip honestly. It is still debt, so give each one an owner and a review date.
• Log the environment (UA, viewport, browserName) in the failure artifacts. Reading the UA to report where something ran is legitimate; branching product behaviour on it is the problem.
• Test the fallback too. A feature check has two branches; run the suite with the feature removed (for example by deleting window.requestIdleCallback in an init script, a snippet Playwright runs in the page before any of the app's own scripts, added with page.addInitScript) so the unused branch cannot rot.

Trade-offs and when sniffing is acceptable

Feature detection fails when a feature exists but is buggy, which a presence check cannot see. Then test the behaviour (run a tiny probe, such as measuring a layout result) rather than the name, and only as a last resort accept a UA-based workaround: scope it to one named engine and a version range, put a ticket and removal date on it, and cover it with a test that fails when the bug is fixed upstream so the workaround is deleted.

Running the code

Files: package.json with { "private": true, "type": "module" }, then the two files below. Inside mcr.microsoft.com/playwright:v1.48.2-jammy:

npm i @playwright/test@1.48.2
npx playwright test

// playwright.config.js
import { defineConfig } from '@playwright/test';

export default defineConfig({
  testDir: '.',
  workers: 1,
  reporter: 'list',
  projects: [
    { name: 'chromium', use: { browserName: 'chromium' } },
    { name: 'firefox', use: { browserName: 'firefox' } },
    { name: 'webkit', use: { browserName: 'webkit' } },
  ],
});

// detect.spec.js
import { test } from '@playwright/test';

test('sniffing versus detecting', async ({ page }, info) => {
  await page.goto('data:text/html,<p>x</p>');
  const r = await page.evaluate(() => {
    const ua = navigator.userAgent;
    return {
      uaSaysSafari: /Safari/.test(ua),
      uaSaysChrome: /Chrome\//.test(ua),
      uaSaysGecko: /Gecko/.test(ua),
      requestIdleCallback: 'requestIdleCallback' in window,
      hasSelector: CSS.supports('selector(:has(a))'),
    };
  });
  console.log(info.project.name.padEnd(9), JSON.stringify(r));
});

Playwright's WebKit is built from WebKit's main branch and is not Safari on iOS or macOS, so the requestIdleCallback result describes that build only.

Direct answer

Work from evidence outward in four stages. First pin down exactly where the bug does and does not reproduce (iOS version, device model, Safari tab versus Home Screen app versus in-app web view, fresh versus aged profile). Second, attach the Mac's Safari Web Inspector to the physical device over USB and add a small page-side logger that ships errors and lifecycle events to a server, so you see what the device saw. Third, rank platform-specific suspects by cheap tests, changing one variable at a time. Fourth, fix it and keep a real-device check, because the fast browser tests cannot prove the fix.

Why a simulator and a macOS WebKit build both pass: they run on a Mac's CPU, memory, network, input hardware and settings. A physical phone adds its own memory limits, touch input, on-screen keyboard, retracting browser toolbars, power modes, content blockers and storage history. Playwright's WebKit is built from upstream WebKit sources and is not the branded Safari (Playwright's browsers page says so), and its iPhone profiles such as devices['iPhone 13'] are emulation of viewport, pixel ratio and user-agent string.

Stage 1: Characterise the failure

Fill in this table before touching code; each row removes a family of causes.

Stage 2: Collect from the device

Safari Web Inspector on the physical device. Apple's Safari Developer Guide describes the setup: on the device turn on Settings > Safari > Advanced > Web Inspector, connect by USB, then pick the device and page from Safari's Develop menu on the Mac (Web Inspector opens in its own window; the guide is archived, so menu wording may differ in current iOS). Use its panels in this order:

1. Console for exceptions and warnings.
2. Network for failed, blocked or slow requests, cookies sent, and cache behaviour.
3. Timelines to record a session and see long tasks (stretches where the page's main thread is busy for 50 ms or more, per MDN, which make taps feel late), layout and memory growth.
4. Storage to inspect cookies, localStorage and IndexedDB as the device actually holds them.

A page-side logger for what you cannot attach to. Real users and test devices cannot always be tethered. Ship errors, unhandled promise rejections, page lifecycle events and viewport changes to a log endpoint (use navigator.sendBeacon, which queues a small request that the browser transmits asynchronously without delaying unload or the next navigation, and call it from a visibilitychange listener when document.visibilityState === 'hidden', the moment MDN calls the most reliable; pagehide and unload are not reliably fired, especially on mobile, so keep pagehide only as a fallback, and expect the very last event to be lost occasionally). How to read the probe. addInitScript registers the collector to run before any page script on every navigation, so it is in place before the bug can happen. page.route plus route.fulfill answers the request for https://app.test/ from memory instead of the network; an https address makes the page a secure context (a page served over HTTPS or from localhost, which many browser APIs require), and no real server is needed. Firefox rejects Playwright's isMobile option, so the script removes it for Firefox only. setTimeout(() => null.foo, 0) and Promise.reject(...) are deliberate faults: they fire the collector's error and unhandledrejection listeners so there is something to log, and the same fault is then worded differently by each engine. visualViewport is the part of the page currently visible, shrinking when the on-screen keyboard opens, which is why its resize event is logged. The features function asks each engine whether an API exists, the same check a page would use instead of reading the user-agent.

The collector below runs in all three Playwright engines, and the run shows why you must keep the raw message and the user-agent: the same bug is worded differently per engine, and logging tools that group by message will split it.

// probe.mjs
import { chromium, firefox, webkit, devices } from 'playwright';

// Page-side collector: the events you would ship to a log endpoint from a real device.
const collector = () => {
  window.__log = [];
  const add = (type, detail) => window.__log.push({ type, detail });
  window.addEventListener('error', (e) => add('error', e.message));
  window.addEventListener('unhandledrejection', (e) => add('rejection', String(e.reason)));
  window.addEventListener('pageshow', (e) => add('pageshow', 'persisted=' + e.persisted));
  window.addEventListener('pagehide', (e) => add('pagehide', 'persisted=' + e.persisted));
  window.visualViewport?.addEventListener('resize', () =>
    add('vv-resize', Math.round(window.visualViewport.height)));
};

const features = () => ({
  dvh: CSS.supports('height', '100dvh'),
  visualViewport: 'visualViewport' in window,
  deviceMemory: 'deviceMemory' in navigator,
  performanceMemory: 'memory' in performance,
  screenOrientationLock: !!(screen.orientation && screen.orientation.lock),
  vibrate: 'vibrate' in navigator,
});

for (const [name, type] of [['chromium', chromium], ['firefox', firefox], ['webkit', webkit]]) {
  const browser = await type.launch();
  const { isMobile, ...phone } = devices['iPhone 13'];   // Firefox rejects isMobile
  const context = await browser.newContext(name === 'firefox' ? phone : { ...phone, isMobile });
  const page = await context.newPage();
  await page.addInitScript(collector);
  // Serve a page from a fake https origin so the page is a secure context.
  await page.route('https://app.test/', (route) =>
    route.fulfill({ contentType: 'text/html', body: '<title>t</title><p>hi</p>' }));
  await page.goto('https://app.test/');
  await page.evaluate(() => { setTimeout(() => null.foo, 0); Promise.reject(new Error('boom')); });
  await page.setViewportSize({ width: 390, height: 700 });
  await page.waitForFunction(() => window.__log.some((l) => l.type === 'error'));
  await page.waitForFunction(() => window.__log.some((l) => l.type === 'rejection'));
  const log = await page.evaluate(() => window.__log.filter((l) => l.type === 'error' || l.type === 'rejection'));
  const feat = await page.evaluate(features);
  console.log(`--- ${name} ${browser.version()}`);
  console.log('error  :', log.find((l) => l.type === 'error').detail);
  console.log('features:', JSON.stringify(feat));
  await browser.close();
}

Run in the Playwright 1.48.2 container (Chromium 130, Firefox 131, WebKit 18.0), it prints:

--- chromium 130.0.6723.31
error  : Uncaught TypeError: Cannot read properties of null (reading 'foo')
features: {"dvh":true,"visualViewport":true,"deviceMemory":true,"performanceMemory":true,"screenOrientationLock":true,"vibrate":true}
--- firefox 131.0
error  : TypeError: null has no properties
features: {"dvh":true,"visualViewport":true,"deviceMemory":false,"performanceMemory":false,"screenOrientationLock":true,"vibrate":false}
--- webkit 18.0
error  : TypeError: null is not an object (evaluating 'null.foo')
features: {"dvh":true,"visualViewport":true,"deviceMemory":false,"performanceMemory":false,"screenOrientationLock":false,"vibrate":false}

Read the output two ways. The error lines show three wordings for one exception. The features lines show engine differences (for example screen.orientation.lock is absent in this WebKit build while present in the other two), which is why behaviour is gated by feature detection rather than by user-agent. None of it says what iOS Safari does; the build is a Linux WebKit and only the real phone can answer that.

Stage 3: Suspects, each with a one-variable test

Order of attack. The rows are ordered by how cheap the test is, which is also the order to try them: the two-minute checks first (viewport units and keyboard, one logged number each; Back/forward restore, one navigation), then the ones that need setup (an aged profile, throttled CPU). Let the symptom move a row up: content cut off at the bottom or hidden behind the keyboard points at the first two rows; stale data or dead timers after tapping Back points at the third; a failure only after a week away points at the storage row; a page that restarts by itself points at the last row. Change one row at a time and keep the result in a log so the bisection is traceable.

Stage 4: Fix and keep it fixed

1. Bisect by build: deploy each candidate commit to a preview URL and open it on the device (a device cloud with real iPhones is the automated option).
2. Fix the root cause behind a feature check (CSS.supports('height', '100dvh')), not a user-agent test.
3. Add the narrowest reproducible test you can: a Playwright test that sets the matching viewport size and asserts the layout rule, labelled as emulation.
4. Keep one real-device smoke test of the exact flow in the nightly lane, and watch the production error rate for that iOS version after release: the device check proves the fix, the production metric proves it held.

Pitfalls

Do not "fix" it by sniffing the user-agent, and do not close the ticket on a green simulator run: the simulator already passed while the bug existed.

Running the code

# put probe.mjs and a package.json containing {"type":"module"} in one folder, then:
docker run --rm --ulimit core=0 -v "$PWD":/w -w /w mcr.microsoft.com/playwright:v1.48.2-jammy sh -c \
  'npm i --silent playwright@1.48.2 && timeout 120 node probe.mjs'