Accessibility Testing Questions

Approach (high level)
I’d add layered checks: fast deterministic checks (contrast, axe-core) run on every PR; smoke accessibility checks (keyboard nav, lightweight screen-reader runs) run nightly or on release branches; full manual/assistive-tech validation as gating for major releases.

Architecture / CI flow

• Pre-merge: contrast + automated axe-core (CI job) across Chromium & Firefox headless.
• Post-merge/nightly: keyboard-navigation smoke tests + lightweight TalkBack/VoiceOver checks on device farms or emulators.
• Alerts: failing job opens GH/Jira ticket with artifact links (screenshots, HTML snippets, axe JSON).

Implementation details

• Contrast: use a fast DOM color-extraction utility (puppeteer/playwright), compare WCAG formulas.
• Axe-core: integrate axe-core + jest-playwright or mocha; save axe results as SARIF for dashboards.
• Keyboard smoke tests: scripted sequences with Playwright/Selenium using tabindex/querySelector focus order assertions; validate focus-visible and skip-links.
• Screen-reader smoke: run VoiceOver via macOS automation (osascript) and TalkBack via Android emulator + adb/TTS logging to capture spoken output; assert presence of expected accessible name/role snippets rather than full transcript to reduce brittleness.

Example: run axe in Playwright

// node, Playwright + axe-core
const { injectAxe, getAxeResults } = require('axe-playwright');
await injectAxe(page);
const results = await getAxeResults(page);

Platform limitations & mitigations

• VoiceOver/TalkBack behaviors differ; emulator TTS timing varies. Mitigate by:
  • Running screen-reader checks nightly not on every PR.
  • Using tolerant assertions (contains expected label) and waiting for stable focus/state.
  • Running only high-value checks (existence of accessible-name, role, and important landmarks).

Handling false positives / flakiness

• Classify failures: deterministic (contrast, axe rule with clear fix) vs flaky (screen-reader timing).
• Flaky pipeline policy: require X consecutive failures across N runs before opening a blocking ticket.
• Use rule exclusions with annotations and document technical debt in backlog; create repro artifacts (HAR, DOM snapshot, axe JSON, video) to triage.
• Maintain baseline of known AT differences and map to a “noise” whitelist.

Metrics & feedback

• Track mean time to fix a11y defects, PR failure rate caused by a11y checks, and flaky-rate. Use these to tune thresholds and cadence.

This design balances fast feedback with realistic AT variability, surfacing actionable issues while minimizing noise for developers.

Approach (high level)

I treat cross-browser a11y as part of CI: run automated checks per browser/viewport, triage results, then do targeted manual and assistive-technology (AT) validation for gaps.

Automated tools & techniques

• axe-core (browser extension + npm/lib) integrated into Selenium/WebDriver or Playwright tests
• pa11y / pa11y-ci for CI reports
• Lighthouse CI for Chrome and synthetic metrics
• WAVE API for quick checks
• Integrate visual regression (Percy) to spot focus/contrast regressions
• Run across Chrome, Firefox, Edge, Safari on multiple viewports via BrowserStack / Sauce

Limitations of automated checks

• They catch ~30–50% of issues (semantics, contrast, ARIA misuse, missing labels) but not keyboard flow, cognitive issues, or context-specific semantics
• False positives/negatives require triage

When to do manual / AT validation

• Keyboard navigation, focus order, skip links, modal traps
• Screen reader testing (NVDA, VoiceOver, JAWS) for dynamic content/ARIA
• Mobile screen readers (TalkBack / VoiceOver) and gesture checks
• Complex interactions (drag/drop, canvas, custom widgets) and usability for cognitive/learning disabilities

I document reproducible steps and add failing cases to automation where possible (e.g., assertions for role/label presence, focus visibility, and contrast thresholds).

Approach summary
Automate accessibility checks for dynamic UI by combining programmatic axe-core scans with deterministic focus/navigation assertions. Use automated scans to catch semantic/attribute issues and targeted functional tests to validate ARIA live regions and focus management after re-renders.

Automated + manual strategy

• Use axe-core (integrated into Playwright/Puppeteer or Selenium) to run post-action scans whenever the UI mutates (after network responses, state updates, modal opens, etc.).
• Add focused functional tests that simulate keyboard users: Tab/Shift+Tab, Enter/Escape, arrow keys, and ensure focus lands where expected.
• Validate ARIA live regions by injecting a mutation observer or polling for live region text updates and asserting announced content and polite/assertive roles.

Sample Playwright + axe-core snippet

// javascript
const { injectAxe, checkA11y } = require('axe-playwright');
await page.goto(url);
await injectAxe(page);
// perform action that causes dynamic content/focus change
await page.click('#open-dialog');
// run axe after UI stabilizes
await checkA11y(page, null, { detailedReport: true });

Focus/navigation example

• After creating an item, assert focus moves to the new item's primary control:

const focused = await page.evaluate(() => document.activeElement.id);
expect(focused).toBe('item-123-edit-btn');

When to fail vs warn

• Fail: violations that block keyboard/screen-reader interaction (missing focusable controls, focus traps, incorrect ARIA roles, critical contrast, missing live region updates when content is only visually conveyed).
• Warn: low-severity issues or best-practice improvements (suggested ARIA tweaks, non-critical contrast, redundant roles) that should be reviewed but don’t prevent access.
• Implement CI policy: fail pipeline on any axe “serious” or “critical” violations and on deterministic focus assertion failures; log “moderate/minor” as warnings with links to DOM snapshots and repro steps.

Why this works
Combining semantic scans with deterministic keyboard/focus assertions captures both static attribute problems and runtime interaction regressions that automated scanners alone can miss. Continuous CI enforcement ensures regressions are caught early while triaging warns keeps noise manageable.

Approach overview
I would treat automated a11y checks as first-class tests in the UI pipeline: fast checks on every PR for rapid feedback, deeper scans nightly, and full-site audits on a cadence (weekly or before releases).

Where to run checks

• PRs: run lightweight axe-core or Lighthouse CI on affected pages/components; surface failures as GitHub/GitLab annotations so devs fix during code review.
• Nightly: run broader site-wide checks, multiple viewports, and generate historical reports.
• Pre-release: run full Lighthouse audits and export artifacts.

Handling noisy results & false positives

• Classify rules by severity and confidence; fail PRs only on high-severity deterministic failures.
• Use baseline/approval mechanism: allow known issues to be recorded with IDs and expiration; store suppression lists in repo.
• Improve selectors/tests to avoid transient DOM states; retry or wait for stable state before scanning.
• Add comment with failing HTML snippets and links to failing element for quick triage.

Combining automated + manual

• Automated for regression, coverage, and blocking issues.
• Manual review for keyboard flows, screen reader experience, cognitive and content clarity.
• Establish an a11y triage workflow: automation flags → accessibility champion reviews → assign dev/UX fixes → update baseline if false positive.
• Track metrics (open a11y defects, time-to-fix) and run periodic paired reviews (tester + developer) for complex flows.

Example tools: axe-core in Selenium/Cypress, axe-cli for node, Lighthouse CI in GitHub Actions, and reports visualized in Allure/HTML for reviewers.