Visual Regression Testing Questions

Clarify goals & constraints

• Prevent unintended visual regressions with fast, actionable CI feedback across supported viewports/devices; minimize noise and reviewer burden; store baselines + diffs efficiently.

High-level architecture

• CI job runs after build -> render pages/components in headless browser(s) -> capture screenshots -> compare against baselines -> store diffs + metadata -> notify PR with summary + links to review UI diffs (designers/engineers).

Baseline management

• Baselines versioned alongside build artifacts/artifacts bucket (S3/Blob) and referenced by commit SHA + branch + release tag.
• Promote baselines: unapproved diffs create "candidate baseline"; merge/QA approval promotes candidate to canonical baseline.
• Provide CLI/web UI to accept/reject baselines; keep retention policy (e.g., last N + tagged releases).

Pixel diff thresholds & strategy

• Use multiple thresholds: strict pixel-perfect for critical UX elements; perceptual SSIM or delta-E with tolerance for soft elements.
• Default: reject if > 0.5% of pixels changed or max-diff > 20 (adjust per component type).
• Use region masking and ignore-by-css to avoid known-unstable areas.

Diff review workflow

• PR comment/badge with summary (pass/fail, % changed, top-3 diffs).
• Link to visual review tool (Percy/Chromatic or custom with thumbnails, side-by-side, flicker).
• Assign diffs to designer for visual approval; engineer can annotate CSS/DOM causes. Approval triggers baseline promotion.

Handling dynamic content

• Deterministic test data & mocked APIs for primary flows.
• For unavoidable dynamic areas: sanitize timestamps/IDs, use CSS masks, or snapshot DOM state instead of full-page render.
• Use visual regression hooks to wait for stable paint (network idle + no animation).

Device/viewport matrix

• Prioritize breakpoints: mobile/375, tablet/768, desktop/1366 + high-DPI variants for critical pages/components.
• Use risk-based selection: core flows on full matrix; low-risk components on single viewport.
• Leverage browser-emulation (Chromium/Firefox/WebKit) and real-device pools for critical releases.

Storage & artifacts

• Store baselines/diffs in object storage with metadata (commit, test id, viewport, browser).
• Use deduplication + compression; keep thumbnails separately for quick UI.
• Expose CDN-backed URLs for reviewers; garbage collect old artifacts per retention policy.

Performance & scaling

• Parallelize screenshot tasks by route/component and viewport; use containerized headless browsers.
• Use incremental testing: only run visual checks for changed UI files or affected routes via dependency mapping.
• Cache static assets and pre-render pages to reduce load time; avoid full site renders where component-level snapshots suffice.
• Monitor CI runtime; abort long-running jobs with progressive feedback (first N failures shown fast).

Trade-offs

• Strict thresholds reduce false negatives but increase noise; risk-based thresholds balance signal.
• Full matrix + real devices increases confidence at cost of CI time; mitigate via incremental and prioritized runs.

Result: deterministic, scalable visual regression pipeline producing fast, designer-friendly diffs with clear promotion paths for baselines and practical performance trade-offs.