Web Accessibility Testing Toolchain

Most teams test a11y (accessibility) like this: just before release, they find a Mac, turn on VoiceOver, close their eyes, poke around twice, and conclude, "It reads something aloud, so it should be fine." More commonly, they do not test at all. They wait for user complaints, a demand letter from a law firm, or a failed EU compliance review before they panic.

That approach treats a11y testing as a choice between "do not test" and "rely entirely on human listening." Both extremes are wrong. a11y has long had a mature, layered automation toolchain, ranging from immediate feedback as you write code to CI gates that block merges. But tools are not omnipotent: automation can catch at most 30%-40% of a11y issues. The remaining majority, involving semantics, order, and real interaction quality, must be tested by a person using a screen reader. These approaches do not conflict; they are two essential parts of the same system.

This article explains the tools used at each of the four a11y testing layers, the problems each layer catches, how to configure them, and, most importantly, which problems automation cannot detect at all and must be left to people.

1. Four Testing Layers: Why One Tool Is Not Enough

Establish this framework first, or you will fall into the illusion that installing the axe extension means a11y is complete.

a11y issues cover an enormous range. They include obvious basic errors such as "an img has no alt," but also issues exposed only through real interaction, such as "CSS removed the focus ring," "a dynamically rendered dialog does not trap focus," or "updates in an aria-live region are not announced." No single tool can cover everything. Mature practice uses four layers, each answering a different question:

1. Lint layer (while coding): eslint-plugin-jsx-a11y. It reports issues the moment you type JSX, including missing alt text, a positive tabindex, or a misspelled aria-* attribute. It catches "the code was written incorrectly." It costs almost nothing and should be the first line of defense.
2. Automated scanning layer (during development): axe-core, its browser extensions, and Lighthouse's a11y audit. These run dozens of rules against the rendered DOM and catch DOM-level violations such as insufficient contrast, duplicate IDs, and form controls without labels. It catches "the rendered DOM is noncompliant."
3. E2E integration layer (in CI): @axe-core/playwright. This embeds axe in your E2E tests and automatically asserts that "this page has no violations" after completing a workflow in a real browser. It catches "the real page, including dynamic rendering, authenticated states, and asynchronous data, has no violations."
4. Manual listening layer (before release): VoiceOver (Mac) / NVDA (Windows). Actually complete a workflow with a screen reader. It catches semantic and experiential problems that automation can never detect, including reading order, whether the heading hierarchy makes sense, and whether dynamic updates are announced.

Why must all four layers be combined? Because every layer has blind spots. Lint sees only source code and cannot execute rendering logic. An axe scan sees only the current DOM snapshot and cannot catch state changes after interaction. E2E axe tests can inspect post-interaction states, but they can check only problems that rules can determine. Semantic questions such as "is this button's wording clear to a blind user?" and "is the focus order reasonable?" require human judgment and are beyond axe's rule engine.

Deque, the company behind axe-core, reports in its own research that automated testing can cover only about 30%-40% of a11y issues. The remaining 60%+, including semantics, order, wording, and real interaction quality, requires manual testing. Remember this rule: automation is a safety net, not a waiver of responsibility; manual listening is an irreplaceable layer, not an optional extra.

The next four sections examine each layer in turn.

2. Layer One: eslint-plugin-jsx-a11y for Immediate Feedback While Coding

This layer has the lowest cost and the fastest return. Its premise is that many a11y errors can be identified directly in JSX source code, without waiting for rendering. For example, <img> must have alt, tabIndex cannot be positive, and aria-* values must be valid. These problems should be highlighted the instant they are written, not discovered after the browser starts.

Installation and Configuration

ESLint 9 flat config setup (eslint.config.js):

// eslint.config.js
import jsxA11y from 'eslint-plugin-jsx-a11y';

export default [
  ...其它配置,
  jsxA11y.flatConfigs.recommended,  // 推荐规则集，开箱即用
];

Legacy .eslintrc configuration:

{
  "extends": ["plugin:jsx-a11y/recommended"],
  "plugins": ["jsx-a11y"]
}

After installation, typing <img src="/hero.png" /> without alt text immediately produces an editor error; you do not have to wait for a build. That is the value of this layer: eliminate a class of bugs at the instant it appears.

What It Actually Catches

These are the most useful rules in the recommended ruleset, ordered by how often I have seen them prevent problems:

• alt-text: <img> must have alt, even if it is the empty string alt="", which means "this is a decorative image; screen readers should skip it." This is the most commonly omitted attribute.
• aria-role: The role attribute must contain a valid ARIA role. A typo such as role="buton" produces an immediate error. It also validates aria-* values; for example, aria-expanded can only be true/false.
• tabindex-no-positive: Prohibits positive values of tabIndex={1} or greater. A positive tabIndex disrupts the document's natural focus order, causing keyboard users to jump unpredictably around the page. This is a classic a11y antipattern. Always use 0, which allows focus in document order, or -1, which allows programmatic focus without adding the element to the Tab sequence.
• no-autofocus: Prohibits autoFocus. Forcibly stealing focus can make screen-reader users "lose their place." The page loads and focus jumps directly to an input before they have even heard the page title.
• anchor-is-valid: An <a> must either have a valid href or be replaced with a <button>. A pseudo-link such as <a onClick> is a double failure: a screen reader announces it as a "link" even though it does not behave like one, and it cannot receive keyboard focus.
• label-has-associated-control: A <label> must be associated with a form control through htmlFor or by wrapping it. Otherwise, the label is orphaned and the input becomes an "unnamed edit field" to a screen reader.
• no-static-element-interactions: Flags a <div onClick> without role/tabIndex/onKeyDown. It works with a mouse but is unusable from a keyboard or screen reader. This rule forces you to use a semantic element or implement the complete set of interaction attributes.

What It Cannot Catch

Lint sees only static source code and cannot inspect the runtime DOM. If JavaScript dynamically renders an <img> without alt text, or a component library generates a problematic internal DOM structure, lint cannot see it. It also cannot detect contrast, focus management, or aria-live behavior for dynamic content. Those belong to the following layers.

In short: lint is the first gate, but it is only a gate, not the entire flood barrier. It helps your code "make fewer mistakes from the outset," but it cannot guarantee that the rendered output is compliant.

3. Layer Two: axe-core + Lighthouse for Browser Extensions and Automated Scanning

After lint, the next layer runs rules against the actual rendered DOM. Its core engine is axe-core, Deque's open-source a11y rule engine and the de facto industry standard. Lighthouse's a11y audit, Chrome DevTools' Accessibility panel, and numerous browser extensions all use axe-core underneath. Learn it once and use that knowledge everywhere.

What axe-core Tests

axe-core includes more than 50 rules grouped by tags: wcag2a, wcag2aa, wcag21a, wcag21aa, corresponding to WCAG 2.x Levels A/AA, plus best-practice. Each rule answers whether a category of implementation complies. Examples include:

• color-contrast: Whether text and background contrast meets WCAG AA, with 4.5:1 for normal text and 3:1 for large text.
• duplicate-id: Duplicate id values, which can make aria-labelledby/htmlFor point to the wrong element.
• region: Whether page content is enclosed by semantic landmarks such as <main>/<nav>/<header>. Without them, content is orphaned and difficult for screen-reader users to navigate.
• heading-order: Whether heading levels are skipped, such as jumping directly from h1 to h4.
• button-name/link-name: Buttons and links must have an accessible name from text, aria-label, or aria-labelledby. Otherwise, the screen reader announces only "button," leaving the user with no idea what it does.

One important axe design principle is worth remembering: it reports only violations it can determine with confidence and deliberately minimizes false positives. Rules with uncertain results are categorized as incomplete, requiring manual review, rather than being reported directly as violations. This is what makes axe suitable as a CI gate. Nobody will use a tool to block merges if it constantly produces false alarms.

How to Use It: Browser Extensions + Lighthouse

During development, the fastest approach is to install the axe DevTools browser extension, available for Chrome, Firefox, and Edge. Open any page and run a scan. Within seconds, it produces a report showing the violating elements, failed rules, severity levels (critical/serious/moderate/minor), and remediation instructions. It is one of the most effective tools for diagnosing a11y issues on an existing page.

The Lighthouse a11y audit runs a subset of axe. You can run it through the Lighthouse panel in Chrome DevTools or use Lighthouse CI against build output, as this project already does:

// .lighthouserc.json
{
  "ci": {
    "collect": {
      "staticDistDir": "./dist",
      "numberOfRuns": 3,
      "settings": { "preset": "desktop" }
    },
    "assert": {
      "assertions": {
        "categories:accessibility": ["error", { "minScore": 0.95 }]
      }
    }
  }
}

This sets the passing threshold for the a11y score to 0.95, with 1.0 being a perfect score. One detail is worth noting: each Lighthouse a11y audit is scored as either 0 or 1; a rule either passes completely or receives zero, so a score of 0.95 means a small number of audits did not pass. Do not let a high score create complacency; inspect the specific failures.

Limitations of This Layer

An axe scan performs static evaluation against a DOM snapshot at a particular moment. This means:

• It tests only the state present at the instant of the scan. Your page may have five tabs, three dialogs, and two dropdown menus, but the scanner sees only the default expanded state. It does not know that DOM for the other interaction states exists.
• It cannot test the interaction process. After a dialog opens, does focus move into it? Can the Tab key cycle within the dialog? When the dialog closes, does focus return to the triggering button? These are process problems that snapshot scanning cannot detect.
• It cannot determine whether something is reasonable. axe can tell you that a button has an aria-label, but it cannot judge that the label "click here" is meaningless to a blind user. That requires human judgment.

The axe extension is therefore an excellent development-time "mine detector," but a clean scan does not mean there are no problems. To cover interaction and dynamic states, proceed to the next layer.

4. Layer Three: @axe-core/playwright for Automatic Violation Assertions in E2E Tests

This layer embeds axe in E2E tests, addressing two blind spots in the previous layer: it can inspect the DOM after interactions and the real page under authenticated states or asynchronous data. The principle is simple. After Playwright completes a page load or interaction, run an axe analysis against the current DOM and assert that there are no violations. This allows axe to inspect every state exercised by the test.

Installation and Basic Usage

pnpm add -D @axe-core/playwright

Minimal example:

// e2e/a11y.spec.ts
import { test, expect } from "@playwright/test";
import AxeBuilder from "@axe-core/playwright";

test("首页无 a11y 违规", async ({ page }) => {
  await page.goto("/");
  const results = await new AxeBuilder({ page }).analyze();
  expect(results.violations).toEqual([]);
});

This completes the assertion that "after this page is rendered, it passes every axe rule." It requires no separate testing infrastructure and uses the same browser and assertion framework as your E2E suite.

Advanced Usage: Testing Post-Interaction States and Selecting Rules

The real value comes from testing the DOM after an interaction. Examples include checking whether an open dialog is compliant or whether a validation error can be read by a screen reader after form submission fails:

test("登录失败后错误提示可访问", async ({ page }) => {
  await page.goto("/login");
  // 故意触发校验失败
  await page.getByRole("button", { name: "登录" }).click();
  // 对失败态的 DOM 跑 axe
  const results = await new AxeBuilder({ page })
    .include("#login-form")          // 只扫登录表单区域
    .withTags(["wcag2a", "wcag2aa"]) // 只跑 WCAG A/AA 级规则
    .analyze();
  expect(results.violations).toEqual([]);
});

Two techniques are particularly important:

• .include(selector): Scans only the specified region. A whole-page scan can introduce noise from third-party components or advertising iframes. Restricting the scan to your component produces cleaner results.
• .withTags([...]): Runs only rules for specified levels. The default includes wcag2a/aa + wcag21a/aa + best-practice, but many teams want to enforce only the WCAG A/AA compliance baseline while treating best-practice as non-blocking guidance. This keeps CI more stable.

Handling "Known Noise": Disable Rules Sparingly

Scanning a legacy project often reveals dozens of violations and turns CI red. The team's first reaction may be to "disable everything until it turns green." That is the worst approach. Classify and manage the issues instead:

test("历史页面（含已知问题，逐步修复）", async ({ page }) => {
  await page.goto("/legacy");
  const results = await new AxeBuilder({ page })
    .disableRules(["color-contrast", "heading-order"])  // 只禁有跟踪 issue 的
    .analyze();
  expect(results.violations).toEqual([]);
});

The discipline is simple: every entry in disableRules must have a corresponding ticket in the issue tracker. Remove each disabled rule when its issue is fixed. Disabling rules indiscriminately destroys this layer while creating a false sense that "testing is already in place." My hard metric for teams is: the number of disabled rules may only decrease, never increase, and new code must introduce zero new violations. This lets the gate block regressions in new code while legacy debt is gradually repaid, ensuring a11y improves instead of deteriorating.

5. Layer Four: VoiceOver / NVDA for Manual Screen-Reader Testing

The first three layers are automated and together catch 30%-40% of issues. For the remaining 60%, there is no shortcut; someone must actually listen with a screen reader. Automation can never replace this layer, yet most teams skip it entirely. The result is "axe is completely green, but blind users still cannot use the product."

What Automation Cannot Test

Here are several common situations where axe reports no violations but screen-reader users have a terrible experience:

• Incorrect reading order. You use CSS flex-direction: row-reverse or order to change the visual sequence without changing the DOM order. What blind users hear is completely different from what sighted users see. axe inspects the DOM and cannot detect CSS reordering.
• Missing semantic heading structure. Whether heading levels from <h1> to <h4> form a sensible hierarchy without skips, whether there is only one h1, and whether h2 represents major sections are questions axe cannot judge. Screen-reader users navigate by jumping between headings, so poorly structured headings make navigation impossible.
• aria-live is missing or misconfigured. Dynamically loaded lists, cart quantity changes after "Add to cart," and form validation messages must use aria-live to be announced. Omit it and blind users do not know anything happened; set it to assertive and it may interrupt their current task. Only listening reveals whether the balance is right.
• Focus management is broken. Focus does not move into an opened dialog and continues moving through the page behind it; the Tab key can leave the dialog; closing it does not return focus to the triggering button. The interaction flow completely breaks down. An E2E axe scan checks whether the dialog DOM is compliant, not whether focus moves as expected.
• Wording is useless to blind users. A button says "click here." Sighted users may infer its purpose from visual context, but a blind user hearing a "click here" link has no idea where it goes. An icon button's aria-label might simply say "button." axe considers these compliant because a label exists, even though they convey no meaning.

These problems share one trait: they concern "experience," not "structure." A rule engine can evaluate structure, but it cannot judge experience. A person must perform this layer.

How to Test: VoiceOver on Mac and NVDA on Windows

VoiceOver is built into macOS/iOS. It is free and requires no installation, so Mac developers have no excuse to skip this layer. Shortcut reference (VO = Control + Option):

• VO + F5: Turn VoiceOver on/off.
• VO + →/←: Read elements one by one in DOM order.
• VO + Command + H: Navigate by heading. Use this to test the heading hierarchy and hear whether you can quickly reach the target section.
• VO + U: Open the Rotor and browse by headings, links, forms, or landmarks. This is advanced navigation used by blind users, and listening immediately reveals how well your page supports it.

NVDA is the most widely used free screen reader on Windows and is open source. Its default shortcuts use Insert as the modifier key, known as the NVDA key, with Numpad and arrow keys for navigation. Mac developers who need to test NVDA without a Windows machine can use a cloud-hosted Windows environment driven by Playwright or install it in a virtual machine.

What Manual Testing Should Cover

Do not try to traverse every page; that is unrealistic. Focus on core workflows by ROI:

1. Critical conversion paths: Registration/login, search, checkout, and form submission. Can a blind user complete them independently?
2. Dynamic interactions: Dialogs, dropdowns, tab switching, and toast notifications. Does focus move correctly, and are updates announced?
3. Keyboard accessibility: Disconnect the mouse. Can every operation be completed using only Tab/Shift+Tab/Esc/Enter? This often catches many problems where menus appear only on mouse hover and cannot be opened from the keyboard.
4. Heading and landmark navigation: Use the Rotor to navigate. Can you reach the main content within three seconds?

A pragmatic goal is to select one or two core workflows for a screen-reader walkthrough in every iteration, rather than pursuing complete coverage. The point is to establish that "someone actually used it," not turn the exercise into a pre-release ritual.

6. CI Gates: Make a11y Violations Block Merges

After adopting all four layers, the final step is integrating them into CI. Otherwise, the tools are merely "installed but never run," and three months later every rule is failing with nobody paying attention. The core purpose of the gate is to automate the requirement that "new code introduces no new violations."

GitHub Actions Configuration

A minimal usable a11y CI job:

# .github/workflows/a11y.yml
name: Accessibility
on:
  pull_request:
    paths: ['src/**', 'e2e/**']

jobs:
  a11y:
    runs-on: ubuntu-latest
    timeout-minutes: 15
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: '20', cache: 'pnpm' }
      - run: pnpm install --frozen-lockfile
      - run: pnpm exec playwright install --with-deps chromium

      # 1. Lint 层：jsx-a11y 规则跟着主 lint job 跑,这里不重复

      # 2. E2E + axe 断言层
      - name: a11y E2E 断言
        run: pnpm exec playwright test --grep @a11y

      # 3. Lighthouse a11y 审计(可选,和构建产物一起跑)
      - name: Lighthouse a11y
        run: pnpm build && pnpm seo:lighthouse

Two practical details deserve emphasis:

• Use an @a11y tag to isolate a11y tests. Mixing a11y tests with functional E2E tests creates confusion. Tag a11y tests with @a11y, run them separately, and inspect a separate report. Functional failures and a11y failures follow different remediation processes.
• Make this job a required check in branch protection. Otherwise, CI can fail while the change is still merged, making the gate meaningless. This is the most important process step. A technically complete setup without organizational enforcement is effectively no setup at all.

Tighten Gradually: Do Not Block Everything at Once

The biggest problem when adding a11y gates to a legacy project is that "CI immediately turns red and blocks every PR." The correct strategy is to treat new and existing code separately:

• Zero tolerance for new code: Apply @axe-core/playwright assertions only to the region changed by the current PR, using .include() to target the modified component. New code that does not comply fails immediately.
• Baseline management for legacy code: For whole-site scans, record the current violation count as a baseline and make CI assert that "the violation count must not exceed the baseline." Each time an issue is fixed, manually reduce the baseline by one. This ensures legacy debt only decreases while new code adds no violations.

This strategy of "freezing the baseline + blocking regressions" is the only practical way to introduce a11y into a legacy project from scratch. Blocking everything at once merely pressures the team to manipulate disableRules.

Summary

• All four testing layers are essential: lint (blocks source errors while coding) → automated axe scans (checks the rendered DOM during development) → @axe-core/playwright (asserts post-interaction states in CI) → manual screen-reader testing (validates semantic experience before release). Every layer has blind spots; combining them provides the closest thing to complete coverage.
• eslint-plugin-jsx-a11y is the zero-cost first gate: Errors involving alt/aria/tabindex/label that can be identified from source are eliminated as code is written, and the recommended ruleset works out of the box.
• axe-core is the reusable automation engine: Browser extensions, Lighthouse, and Playwright all use it underneath. Its 50+ rules report only "confirmed violations" and deliberately minimize false positives, making it suitable as a CI gate.
• CI gates should tighten gradually: Apply zero tolerance to new code, freeze the legacy baseline so it can only decrease, and require a corresponding ticket for every disableRules entry. Blocking everything at once only encourages self-deception.
• Automation covers only 30%-40%; manual listening is irreplaceable: Reading order, heading semantics, aria-live announcements, focus management, and effective wording are "experience" problems that rule engines cannot judge. Someone must actually listen using VoiceOver/NVDA.

In one sentence: the maturity of a11y testing is not measured by how many automation tools you install, but by whether you acknowledge that automation has a ceiling and actually make time to close your eyes and listen to your own product.