Cross-Browser and Cross-Platform Testing Questions

Direct answer

Work from evidence outward in four stages. First pin down exactly where the bug does and does not reproduce (iOS version, device model, Safari tab versus Home Screen app versus in-app web view, fresh versus aged profile). Second, attach the Mac's Safari Web Inspector to the physical device over USB and add a small page-side logger that ships errors and lifecycle events to a server, so you see what the device saw. Third, rank platform-specific suspects by cheap tests, changing one variable at a time. Fourth, fix it and keep a real-device check, because the fast browser tests cannot prove the fix.

Why a simulator and a macOS WebKit build both pass: they run on a Mac's CPU, memory, network, input hardware and settings. A physical phone adds its own memory limits, touch input, on-screen keyboard, retracting browser toolbars, power modes, content blockers and storage history. Playwright's WebKit is built from upstream WebKit sources and is not the branded Safari (Playwright's browsers page says so), and its iPhone profiles such as devices['iPhone 13'] are emulation of viewport, pixel ratio and user-agent string.

Stage 1: Characterise the failure

Fill in this table before touching code; each row removes a family of causes.

Stage 2: Collect from the device

Safari Web Inspector on the physical device. Apple's Safari Developer Guide describes the setup: on the device turn on Settings > Safari > Advanced > Web Inspector, connect by USB, then pick the device and page from Safari's Develop menu on the Mac (Web Inspector opens in its own window; the guide is archived, so menu wording may differ in current iOS). Use its panels in this order:

1. Console for exceptions and warnings.
2. Network for failed, blocked or slow requests, cookies sent, and cache behaviour.
3. Timelines to record a session and see long tasks (stretches where the page's main thread is busy for 50 ms or more, per MDN, which make taps feel late), layout and memory growth.
4. Storage to inspect cookies, localStorage and IndexedDB as the device actually holds them.

A page-side logger for what you cannot attach to. Real users and test devices cannot always be tethered. Ship errors, unhandled promise rejections, page lifecycle events and viewport changes to a log endpoint (use navigator.sendBeacon, which queues a small request that the browser transmits asynchronously without delaying unload or the next navigation, and call it from a visibilitychange listener when document.visibilityState === 'hidden', the moment MDN calls the most reliable; pagehide and unload are not reliably fired, especially on mobile, so keep pagehide only as a fallback, and expect the very last event to be lost occasionally). How to read the probe. addInitScript registers the collector to run before any page script on every navigation, so it is in place before the bug can happen. page.route plus route.fulfill answers the request for https://app.test/ from memory instead of the network; an https address makes the page a secure context (a page served over HTTPS or from localhost, which many browser APIs require), and no real server is needed. Firefox rejects Playwright's isMobile option, so the script removes it for Firefox only. setTimeout(() => null.foo, 0) and Promise.reject(...) are deliberate faults: they fire the collector's error and unhandledrejection listeners so there is something to log, and the same fault is then worded differently by each engine. visualViewport is the part of the page currently visible, shrinking when the on-screen keyboard opens, which is why its resize event is logged. The features function asks each engine whether an API exists, the same check a page would use instead of reading the user-agent.

The collector below runs in all three Playwright engines, and the run shows why you must keep the raw message and the user-agent: the same bug is worded differently per engine, and logging tools that group by message will split it.

// probe.mjs
import { chromium, firefox, webkit, devices } from 'playwright';

// Page-side collector: the events you would ship to a log endpoint from a real device.
const collector = () => {
  window.__log = [];
  const add = (type, detail) => window.__log.push({ type, detail });
  window.addEventListener('error', (e) => add('error', e.message));
  window.addEventListener('unhandledrejection', (e) => add('rejection', String(e.reason)));
  window.addEventListener('pageshow', (e) => add('pageshow', 'persisted=' + e.persisted));
  window.addEventListener('pagehide', (e) => add('pagehide', 'persisted=' + e.persisted));
  window.visualViewport?.addEventListener('resize', () =>
    add('vv-resize', Math.round(window.visualViewport.height)));
};

const features = () => ({
  dvh: CSS.supports('height', '100dvh'),
  visualViewport: 'visualViewport' in window,
  deviceMemory: 'deviceMemory' in navigator,
  performanceMemory: 'memory' in performance,
  screenOrientationLock: !!(screen.orientation && screen.orientation.lock),
  vibrate: 'vibrate' in navigator,
});

for (const [name, type] of [['chromium', chromium], ['firefox', firefox], ['webkit', webkit]]) {
  const browser = await type.launch();
  const { isMobile, ...phone } = devices['iPhone 13'];   // Firefox rejects isMobile
  const context = await browser.newContext(name === 'firefox' ? phone : { ...phone, isMobile });
  const page = await context.newPage();
  await page.addInitScript(collector);
  // Serve a page from a fake https origin so the page is a secure context.
  await page.route('https://app.test/', (route) =>
    route.fulfill({ contentType: 'text/html', body: '<title>t</title><p>hi</p>' }));
  await page.goto('https://app.test/');
  await page.evaluate(() => { setTimeout(() => null.foo, 0); Promise.reject(new Error('boom')); });
  await page.setViewportSize({ width: 390, height: 700 });
  await page.waitForFunction(() => window.__log.some((l) => l.type === 'error'));
  await page.waitForFunction(() => window.__log.some((l) => l.type === 'rejection'));
  const log = await page.evaluate(() => window.__log.filter((l) => l.type === 'error' || l.type === 'rejection'));
  const feat = await page.evaluate(features);
  console.log(`--- ${name} ${browser.version()}`);
  console.log('error  :', log.find((l) => l.type === 'error').detail);
  console.log('features:', JSON.stringify(feat));
  await browser.close();
}

Run in the Playwright 1.48.2 container (Chromium 130, Firefox 131, WebKit 18.0), it prints:

--- chromium 130.0.6723.31
error  : Uncaught TypeError: Cannot read properties of null (reading 'foo')
features: {"dvh":true,"visualViewport":true,"deviceMemory":true,"performanceMemory":true,"screenOrientationLock":true,"vibrate":true}
--- firefox 131.0
error  : TypeError: null has no properties
features: {"dvh":true,"visualViewport":true,"deviceMemory":false,"performanceMemory":false,"screenOrientationLock":true,"vibrate":false}
--- webkit 18.0
error  : TypeError: null is not an object (evaluating 'null.foo')
features: {"dvh":true,"visualViewport":true,"deviceMemory":false,"performanceMemory":false,"screenOrientationLock":false,"vibrate":false}

Read the output two ways. The error lines show three wordings for one exception. The features lines show engine differences (for example screen.orientation.lock is absent in this WebKit build while present in the other two), which is why behaviour is gated by feature detection rather than by user-agent. None of it says what iOS Safari does; the build is a Linux WebKit and only the real phone can answer that.

Stage 3: Suspects, each with a one-variable test

Order of attack. The rows are ordered by how cheap the test is, which is also the order to try them: the two-minute checks first (viewport units and keyboard, one logged number each; Back/forward restore, one navigation), then the ones that need setup (an aged profile, throttled CPU). Let the symptom move a row up: content cut off at the bottom or hidden behind the keyboard points at the first two rows; stale data or dead timers after tapping Back points at the third; a failure only after a week away points at the storage row; a page that restarts by itself points at the last row. Change one row at a time and keep the result in a log so the bisection is traceable.

Stage 4: Fix and keep it fixed

1. Bisect by build: deploy each candidate commit to a preview URL and open it on the device (a device cloud with real iPhones is the automated option).
2. Fix the root cause behind a feature check (CSS.supports('height', '100dvh')), not a user-agent test.
3. Add the narrowest reproducible test you can: a Playwright test that sets the matching viewport size and asserts the layout rule, labelled as emulation.
4. Keep one real-device smoke test of the exact flow in the nightly lane, and watch the production error rate for that iOS version after release: the device check proves the fix, the production metric proves it held.

Pitfalls

Do not "fix" it by sniffing the user-agent, and do not close the ticket on a green simulator run: the simulator already passed while the bug existed.

Running the code

# put probe.mjs and a package.json containing {"type":"module"} in one folder, then:
docker run --rm --ulimit core=0 -v "$PWD":/w -w /w mcr.microsoft.com/playwright:v1.48.2-jammy sh -c \
  'npm i --silent playwright@1.48.2 && timeout 120 node probe.mjs'

Direct answer

Use a three-layer hybrid. Run the bulk of checks on desktop-browser containers and on emulators or simulators, because they are cheap, fast and scriptable. Rent a cloud device farm for a short, fixed list of real phones and Safari builds on a nightly and pre-release cadence. Build a self-hosted device lab only for a need the farm cannot meet (special hardware, a locked-down network, a data-residency rule) or when sustained utilisation makes owning cheaper than renting. Trust emulation for logic and layout; trust only real hardware for touch feel, hardware APIs, performance and platform-owned UI.

For a team that ships only a website, the first two rows of the hybrid table below (desktop and mobile browsers) are the layers to build; the native-app row applies when the product also has iOS or Android apps.

Terms. An emulator (Android) or simulator (iOS) is software on a computer that imitates a phone's operating system, so it runs your app or browser without a physical phone. A cloud device farm is a hosted service renting real phones and browsers. A self-hosted device lab is a rack of physical phones you own, connected to your CI.

Comparison

The Android emulator's documentation says it simulates incoming calls and texts, location, network speeds, rotation and other sensors, and also says that on machines with weaker specs the emulator may not run smoothly and a physical device is the better choice. Both halves matter: emulation is rich enough for most logic, and slow or underspecified hosts are a reason to shift work to real hardware.

Recommended hybrid

Checks move up a layer on a trigger: from emulator to farm when the check depends on something in the real-hardware list below, or when a defect reached users on a device the emulator run had passed; from farm to a self-hosted lab when the monthly farm bill rises above the lab's monthly cost (the break-even above) or a hardware, network or data-residency need cannot be met on rented devices.

Choose the farm device list from your telemetry (the models and OS versions that carry most sessions) and keep it to roughly the number your plan runs concurrently.

Lab versus farm, with illustrative inputs (assumed, not quotes): 10 phones at 400 currency units each, written off over 24 months, is $10 \times 400 / 24 \approx 166.67$ per month; 8 hours of upkeep a month at 60 per hour is $8 \times 60 = 480$; the lab costs about $166.67 + 480 = 646.67$ per month before space, replacement and network. Upkeep time, not the phones, is the largest term, and it is the cost that people forget. For the comparison, 646.67 is the break-even farm price. With an assumed farm plan of 500 per month, the farm is cheaper by 646.67 - 500 = 146.67 per month; with an assumed plan of 1,000, the lab looks cheaper by 353.33 per month, but only while all 10 phones stay busy and upkeep stays at 8 hours, and before space and replacement costs.

Checks to trust only on real hardware

1. Touch and scrolling feel: momentum, rubber-banding (a list bouncing back when scrolled past its end), gesture conflicts with the browser's back-swipe.
2. On-screen keyboard behaviour: how it resizes the viewport and covers inputs, input zoom (the browser enlarging the page when a text field gets focus), autofill.
3. Hardware APIs: camera, microphone, biometrics, NFC (near-field communication), Bluetooth, vibration.
4. Performance: start-up time, scroll smoothness, memory pressure and low-end Android; an emulator uses the host's CPU, so its numbers describe the host.
5. Platform-owned UI and delivery: push notifications through the real vendor service, payment sheets, system share and file pickers, permission dialogs.
6. Network realism: real cellular radio, handoffs between Wi-Fi and cellular, captive portals (the sign-in pages that public Wi-Fi shows before it allows internet access); an emulator can throttle the network but not reproduce a radio.
7. Real Safari on iPhone and macOS: Playwright's WebKit is not iOS Safari (and Playwright's docs say it does not drive branded Safari), so Safari-specific rendering, cookie and storage behaviour must be confirmed on the real browser at least before each release.

Pitfalls

• Treating a passing emulator run as proof of device performance.
• A farm list copied from "popular devices" rather than your own session data.
• Lab devices that auto-update their OS mid-run: pin the version, or it becomes a flake source.
• Using a self-hosted lab to avoid a farm bill, then spending more engineer hours than the bill.

Direct answer

On a subscription product with a React web app and a React Native mobile app, I led the move from two separate implementations to one shared TypeScript package for the logic that has no platform in it (pricing, validation, date handling, API types), while keeping everything that touches the screen, storage or navigation separate per platform. I tested the shared package once in Node, the web app across Chromium, Firefox and WebKit, and the mobile app on an iOS simulator and an Android emulator, and each part ran in CI only when its inputs changed. The compatibility bug that made the case for the whole approach was a date that shifted by the user's time zone offset.

Situation

The web and mobile teams each had their own copy of the pricing rules and the form validation. They drifted: the invoice preview on the phone and the one on the web occasionally disagreed by a rounding rule, and each fix landed twice, weeks apart. My task was to remove the duplication without slowing either team, and to make cross-platform testing a routine step instead of something found by customers.

What I decided to share and what I did not

The rule I used: share what is a pure function of its inputs; do not share what calls a platform API. I tried sharing a component library first and dropped it within the first iteration, because the web and native primitives differ enough that the abstraction cost more than it saved. For the few places where behaviour had to differ I used a file-name convention rather than if statements: React Native documents that it picks .ios. and .android. files automatically, and .native. files for code shared between React Native and the web, so session-store.native.ts and session-store.ts sat side by side behind one interface, and I told the web bundler (the build tool that packs the web app's code into files for the browser) to ignore .native files so the web bundle did not carry unused code.

How I tested across environments

1. Shared package: unit tests run once in Node, plus a file of golden pricing cases (fixed input and expected-price pairs that both apps' tests read) that the web tests and the mobile tests both read, so a disagreement between the apps fails a build.
2. Web: Playwright with three browser projects (Chromium, Firefox, WebKit) at a desktop viewport and a phone viewport, so six runs per change. I picked which browsers got the full suite from our own traffic data and gave the others a smoke set. Because Playwright's WebKit is derived from upstream WebKit and is not the Safari release, I also kept a small real-Safari check on a real iPhone for the release candidate.
3. Mobile: component tests on every change, and end-to-end flows on one iOS simulator and one Android emulator per merge, with a short list of real devices before release.

Build and CI

A single monorepo (one repository holding several packages) with workspaces (the package manager feature that links those packages together locally): packages/shared, apps/web, apps/mobile. A change under packages/shared ran every job; a change under one app ran only that app's jobs, with dependency caching on both. The shared package was versioned with the repository, so there was never a question of which version an app was built against.

A compatibility problem I solved

The API returned timestamps like 2019-01-01T00:00:00 with no time zone marker, and both apps called Date.parse on them. Date.parse returns the number of milliseconds since the Unix epoch, the moment 00:00 UTC on 1 January 1970, and the script below divides by 3,600,000 to print hours. To check one value by hand: 2019-01-01 is 49 years after the epoch, including 12 leap years (1972 to 2016), so 49 x 365 + 12 = 17,897 days, and 17,897 x 24 = 429,528 hours. Midnight in Los Angeles in January is 08:00 UTC, so a time read as Los Angeles local time is 8 hours later: 429,536. The shared package got a single parsing helper and a test that I proved with this script in the three Playwright engines, with the browser's time zone set to Los Angeles (UTC-8 in January):

// dates.mjs
import { chromium, firefox, webkit } from 'playwright';

const inputs = ['2019-01-01', '2019-01-01T00:00:00', '2019-01-01 00:00', '1970/01/01', '2019-01-01T00:00:00Z'];

for (const [name, type] of [['chromium', chromium], ['firefox', firefox], ['webkit', webkit]]) {
  const browser = await type.launch();
  const context = await browser.newContext({ timezoneId: 'America/Los_Angeles' });   // UTC-8 in January
  const page = await context.newPage();
  const result = await page.evaluate((list) => list.map((s) => [s, Date.parse(s)]), inputs);
  console.log(`${name.padEnd(9)}` + result.map(([s, v]) => `${JSON.stringify(s)}=${Number.isNaN(v) ? 'NaN' : v / 3600000 + 'h'}`).join('  '));
  await browser.close();
}

Run in the Playwright 1.48.2 container (Chromium 130, Firefox 131, WebKit 18.0), it printed the same lines on each of 10 runs:

chromium "2019-01-01"=429528h  "2019-01-01T00:00:00"=429536h  "2019-01-01 00:00"=429536h  "1970/01/01"=8h  "2019-01-01T00:00:00Z"=429528h
firefox  "2019-01-01"=429528h  "2019-01-01T00:00:00"=429536h  "2019-01-01 00:00"=429536h  "1970/01/01"=8h  "2019-01-01T00:00:00Z"=429528h
webkit   "2019-01-01"=429528h  "2019-01-01T00:00:00"=429536h  "2019-01-01 00:00"=429536h  "1970/01/01"=8h  "2019-01-01T00:00:00Z"=429528h

Values are hours since the Unix epoch, so 429536 minus 429528 is the 8-hour Los Angeles offset. A date-only string (2019-01-01) and an explicit Z string are read as UTC (429528 h), but a date-time string with no offset is read as local time and lands 8 hours later (429536 h). That is the ECMAScript (the language standard JavaScript follows) rule MDN documents: date-only forms are UTC, date-time forms without an offset are local. So the same API value meant a different instant for every user depending on their time zone. The fix had two parts: the API now sends an explicit Z or offset, and the shared helper rejects strings without one and is the only place dates are parsed. MDN also documents that formats outside the standard ones, such as 2019-01-01 00:00 and 1970/01/01, are implementation-defined and "may not work across all browsers". All three engines here parsed both of them the same way, which says nothing about Safari: Playwright's WebKit is derived from upstream WebKit but is not the Safari release, so a Safari-only parsing difference would not show in this lane, and that is why the real-device check stayed.

The second problem was session storage. The browser keeps the web session in a cookie or web storage and the phone keeps it in the platform's credential storage, and the shared API client had been calling one of them directly. I put a small SessionStore interface in the shared package with two implementations picked by file name, tested the API client against an in-memory fake, and tested each real implementation only in its own environment (a Playwright test on web, an end-to-end flow on the simulator and emulator).

Result

• Pricing and validation went from two implementations to one, so a fix lands once.
• Both apps' test suites read the same golden pricing cases, and the invoice-preview disagreements stopped appearing in testing.
• The time-zone bug has a regression test that runs in three browser engines at once.
• Each pull request ran only the jobs affected by its changed paths, so the extra matrix did not make every change wait for every environment.

What I would do differently

I would add the real-Safari check on day one instead of after the first Safari-specific report, and write the golden-case contract tests before extracting the shared package so the extraction was proven by them, not followed by them. I would also stop trying to share UI components earlier, since I spent a sprint on an abstraction I then removed.

Pitfalls when telling this story

Name the rule you used to decide what to share, not just what you shared. Say which browsers got the full suite and why (traffic), and what you could not cover (real devices) and how you compensated. Keep every claim about results to what you actually measured or counted.

Running the code

# save dates.mjs and a package.json containing {"type":"module"} in one folder, then:
docker run --rm --ulimit core=0 -v "$PWD":/w -w /w mcr.microsoft.com/playwright:v1.48.2-jammy sh -c \
  'npm i --silent playwright@1.48.2 && timeout 100 node dates.mjs'