Scanner Tool Selection and Benchmarking

Choosing an accessibility engine for enterprise batch scanning is a procurement decision disguised as a technical one: the tool you pick sets your false-positive rate, your throughput ceiling at thousands of routes, and the shape of every result your pipeline has to normalize for the next five years. The wrong instinct is to compare marketing pages or trust a vendor’s rule count; the right one is to run each candidate against your own route set under a shared harness and measure determinism, coverage against the WCAG success criteria you actually gate on, and cost per thousand scans. This guide is the selection layer of the broader Automated Scanning & Dynamic Content Ingestion strategy: it gives you the axes that separate a scanner that blocks real regressions from one that floods a backlog, and a reproducible method for benchmarking candidates before you commit an organization to one.

The failure this page prevents is expensive and slow to surface. A team adopts a scanner because it reported the most violations in a demo, wires it into CI, and discovers six months later that half those findings are false positives nobody trusts, the engine version floats so counts drift between runners, and the tool cannot hold an authenticated session — so the most important routes go unscanned. By then the output format is baked into dashboards and ticket routing, and switching means rewriting the normalization layer. Selecting deliberately, against measured criteria, is what keeps that decision reversible and correct.

Prerequisites & Environment Parity #

A benchmark is only meaningful if every candidate runs under identical conditions. Before scoring anything, pin the surface so the only variable is the scanner:

• A frozen route sample. Draw 50–200 real routes from your own property — not a vendor’s fixture site — spanning your hardest surfaces: authenticated dashboards, data tables, custom widgets, infinite-scroll feeds, and at least one page you know is broken. Commit the list. Every tool scans exactly this set.
• A pinned browser. Each candidate must drive the same Chromium build, installed via playwright install --with-deps chromium, so rendering differences do not masquerade as coverage differences. A tool that ships its own bundled browser (Pa11y drives Puppeteer’s Chromium) needs its browser version recorded alongside its results.
• Pinned engine versions. Record the exact version of every engine under test — axe-core@4.10.2, the Pa11y release, the axe DevTools Pro build. A floating version invalidates a benchmark the moment a minor bump adds or removes a rule.
• A defined conformance target. Decide whether you are gating WCAG 2.2 Level AA before you score coverage, because “more rules” is only better if the extra rules map to criteria you enforce. The criterion-by-criterion breakdown of what is machine-decidable lives in the WCAG 2.2 vs 3.0 success criteria taxonomy, and the levels that actually block a deploy are set in the A/AA/AAA compliance level mapping.
• A settled DOM at scan time. Single-page applications must reach a hydrated, stable DOM before any engine evaluates them, or every tool under-reports equally and your benchmark measures nothing. The stabilization signals are covered in the Playwright headless scanning workflows.

Treat the benchmark harness as a committed artifact, not a one-off script. You will re-run it every time a candidate ships a major version, and the value of the exercise is that the numbers are comparable across those runs.

The Selection Criteria That Actually Decide It #

Vendors compete on a single headline number — rules, or violations found in a demo. That number is nearly useless in isolation. Seven axes actually determine whether a scanner survives contact with an enterprise route set, and they trade off against each other.

Rule coverage against your gated SCs. What matters is not the raw rule count but how many rules map to the success criteria you enforce, and how many are automatable at all — roughly a third of WCAG criteria can be machine-decided; the rest need human review regardless of tool. A scanner with 90 rules that all target AA criteria you gate is worth more than one with 140 rules where the surplus are best-practice checks carrying no conformance obligation. Count coverage as “rules that fire on a criterion in your target set”, not rules in the box.

False-positive rate. Every false positive is a tax on human triage and a reason for the team to stop trusting the gate. Measure it directly: sample findings, have an auditor label each true or false, and compute the rate per tool. A scanner that reports twice as many violations at a 30% false-positive rate is often worse than a quieter one at 5%, because the noise trains reviewers to dismiss real findings.

Determinism and version-pinning. The same commit must produce the same counts on two runners and on two days. Tools that inject a lockfile-pinned engine (axe-core) are deterministic by construction; tools that auto-update or resolve a floating engine turn environment drift into phantom regressions. This is a hard gate, not a preference — a non-deterministic scanner cannot back a CI block.

Throughput at scale. A tool that scans ten routes in a demo may OOM-kill a runner at three thousand. Measure wall-clock seconds per route, peak memory per worker, and whether the tool parallelizes cleanly across shards. A CLI that spins a fresh browser per route (the simple Pa11y invocation) is far slower than a persistent context that scans many routes per browser.

Authenticated-state support. Most of an enterprise’s real surface sits behind login. A scanner you can hand a storage_state or an injected session scans the app a user actually sees; one that only takes a URL scans the marketing shell and reports a reassuring, meaningless zero. This axis alone eliminates tools for many real deployments.

Output shape for normalization. Your pipeline has to reduce every finding to a common record — rule id, impact, WCAG SC, selector, route. A tool that emits structured JSON with stable rule ids and impact labels (axe-core’s shape) normalizes cleanly; one that emits a flat CLI report or renumbers issues between versions forces a fragile parser. The output contract you standardize on is enforced by JSON Schema validation for accessibility data; pick a tool whose native shape maps onto it without heroics.

Cost and support. Open-source engines (axe-core, Pa11y) cost nothing in license but everything in the harness you build and maintain. Commercial suites (Deque’s axe DevTools Pro / WorldSpace) add guided intelligent testing, a larger rule set, saved results via API, and a support contract — real value for a large program, at per-seat or per-scan cost. The question is whether the commercial delta buys coverage and support you would otherwise build yourself.

The diagram below routes a candidate through these axes as a hard-gate-first decision, so that a tool failing a non-negotiable criterion is eliminated before you spend time measuring the softer ones.

The two commonest comparison decisions each get their own guide: the three-way engine choice in axe-core vs Pa11y vs Deque WorldSpace for enterprise batch scanning, and the narrower CI question of why a Lighthouse score cannot back a gate in axe-core vs Lighthouse for CI accessibility.

How the Tools Differ Mechanically #

Understanding why counts differ between tools requires knowing that these are not four interchangeable scanners — they occupy different layers.

axe-core is a JavaScript library. You inject it into a page and call axe.run(context, options); it evaluates the live DOM and returns structured JSON. It has no runner, no browser, and no opinion about how you drive it — that is your harness’s job, which is exactly what makes it deterministic and embeddable.

Pa11y is a Node CLI runner. It drives its own headless browser (Puppeteer’s Chromium), loads a URL, and runs a wrapped engine against it — either axe-core or HTML_CodeSniffer — then prints or emits the results. It is a convenience layer over an engine, not an engine itself, so its coverage depends on which engine you point it at.

Deque axe DevTools Pro / WorldSpace is the commercial suite built on axe-core. It adds a larger proprietary rule set, guided intelligent testing (IGT) that walks a human through the checks a machine cannot decide, saved results and history via an API, and a support contract. You are paying for coverage beyond the open-source rules and for the human-in-the-loop workflow.

Lighthouse is a Chrome and CI audit that bundles a subset of axe-core’s rules and rolls them into a single 0–100 score alongside performance and SEO. The score is a communication device, not a conformance signal, and the subset is why its accessibility number diverges from a direct axe run.

This is why two tools report different counts on the same page: they may run different engines, different rule subsets, different versions of the same engine, or evaluate the DOM at a different moment. A benchmark controls for all four.

Step-by-Step: Benchmark Candidates on Your Route Set #

The method is to run every candidate through one harness, normalize each tool’s output to a common record, and compare on the axes above. Each step is independently runnable.

1. Define the benchmark contract #

Fix the record every tool must be reduced to, so comparisons are apples-to-apples regardless of native output shape.

from dataclasses import dataclass, field

@dataclass
class Finding:
    """One normalized violation, tool-agnostic."""
    tool: str          # "axe-core", "pa11y", "axe-devtools-pro"
    route: str
    rule_id: str       # e.g. "color-contrast", "image-alt"
    impact: str        # critical | serious | moderate | minor
    wcag_sc: str       # e.g. "1.4.3"
    selector: str      # CSS path to the node

@dataclass
class ToolRun:
    tool: str
    engine_version: str
    routes: int
    wall_seconds: float
    peak_rss_mb: float
    findings: list[Finding] = field(default_factory=list)

2. Adapt each tool to the contract #

Write one adapter per candidate that maps its native output onto Finding. The adapter’s size is itself a signal — a tool that needs a large, brittle adapter scores worse on the output-shape axis.

def from_axe(route: str, payload: dict) -> list[Finding]:
    """axe-core JSON: violations[].nodes[] with stable rule ids and tags."""
    out = []
    for v in payload["violations"]:
        # axe tags carry the WCAG SC, e.g. "wcag143" -> "1.4.3".
        sc = next((t[4] + "." + t[5] + "." + t[6:]
                   for t in v["tags"] if t.startswith("wcag") and t[4:].isdigit()), "")
        for node in v["nodes"]:
            out.append(Finding(
                tool="axe-core", route=route, rule_id=v["id"],
                impact=v.get("impact") or "minor", wcag_sc=sc,
                selector=node["target"][0] if node["target"] else "",
            ))
    return out

3. Run every tool against the frozen route set #

Drive all candidates through the same routes on the same pinned browser, capturing timing and memory so throughput is measured, not estimated.

import time, resource

def benchmark(tool_name, scan_fn, adapter, routes, engine_version) -> ToolRun:
    run = ToolRun(tool=tool_name, engine_version=engine_version,
                  routes=len(routes), wall_seconds=0.0, peak_rss_mb=0.0)
    start = time.perf_counter()
    for route in routes:
        payload = scan_fn(route)               # tool-specific invocation
        run.findings.extend(adapter(route, payload))
    run.wall_seconds = time.perf_counter() - start
    # ru_maxrss is KB on Linux; convert to MB for a comparable throughput number.
    run.peak_rss_mb = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1024
    return run

4. Score coverage and the false-positive rate #

Coverage is findings that map to a gated SC; the false-positive rate needs a human-labeled sample, because no tool can score its own accuracy.

def coverage_by_sc(run: ToolRun, gated_scs: set[str]) -> dict[str, int]:
    """Count distinct rules that fired on each success criterion you gate."""
    hits: dict[str, set] = {}
    for f in run.findings:
        if f.wcag_sc in gated_scs:
            hits.setdefault(f.wcag_sc, set()).add(f.rule_id)
    return {sc: len(rules) for sc, rules in hits.items()}

def false_positive_rate(sample: list[tuple[Finding, bool]]) -> float:
    """sample is (finding, is_true_positive) pairs labeled by an auditor."""
    if not sample:
        return 0.0
    false_pos = sum(1 for _f, ok in sample if not ok)
    return round(false_pos / len(sample), 3)

5. Produce the comparison and decide #

Reduce each ToolRun to the seven axes, print a table, and let the hard gates from the decision tree eliminate before the soft axes rank. Feed the winning tool’s output straight into the JSON Schema validation for accessibility data contract so the shape you benchmarked is the shape production enforces.

Configuration Reference #

The knobs that govern a fair benchmark, independent of which tool wins.

Verification & Testing #

Prove the harness itself is fair before trusting its numbers. Two invariants catch most rigging: every tool must scan the same route count, and the known-broken fixture must be caught by every candidate — a tool that misses it has a coverage gap the benchmark must surface, not hide.

def test_harness_is_fair(runs, route_sample):
    # Every tool saw the identical route set — no tool got a shorter list.
    for run in runs:
        assert run.routes == len(route_sample), f"{run.tool} scanned a different set"

def test_known_broken_route_is_caught(runs):
    # The deliberately broken fixture must fail image-alt (SC 1.1.1) everywhere.
    for run in runs:
        rule_ids = {f.rule_id for f in run.findings if "known-broken" in f.route}
        assert "image-alt" in rule_ids, f"{run.tool} missed the seeded violation"

In CI, re-run the benchmark on a schedule and on every candidate’s major version bump. The comparison is only decision-grade if it is reproducible; a one-off run taken during a sales cycle is a demo, not a benchmark.

Failure Modes & Troubleshooting #

Counting rules instead of coverage. Symptom: a tool “wins” on rule count but finds nothing new on your routes. Root cause: the surplus rules are best-practice or target criteria you do not gate. Fix: score coverage as rules that fire on your gated SCs against your route set, and discard the box number entirely.

Benchmarking on the vendor’s fixtures. Symptom: every tool looks great and they all agree. Root cause: vendor demo sites are simple, static, and pre-hydrated — nothing like your authenticated single-page app. Fix: benchmark exclusively on your own frozen route sample, including the surfaces that actually break.

Unpinned engines across candidates. Symptom: re-running the benchmark next week produces different rankings. Root cause: one or more tools resolved a floating engine version. Fix: pin and record the exact engine version for every candidate, and treat a version bump as a reason to re-run, not to trust the old numbers.

Ignoring the false-positive rate. Symptom: the highest-scoring tool gets disabled by the team within a sprint. Root cause: raw violation count rewarded a noisy engine, and reviewers stopped trusting it. Fix: measure the false-positive rate on a human-labeled sample and weight it as heavily as coverage.

Scanning the unauthenticated shell. Symptom: a tool reports a clean, low count on routes you know are complex. Root cause: it could not hold the session and landed on a marketing or login shell. Fix: inject authenticated storage state and assert on a post-login landmark before trusting any count. Findings from the winning tool then flow into the error categorization and triage pipelines and the sharded batch validation architecture.

Frequently Asked Questions #

Related #

• Axe-Core Enterprise Configuration
• JSON Schema Validation for Accessibility Data
• axe-core vs Pa11y vs Deque WorldSpace for Enterprise Batch Scanning
• axe-core vs Lighthouse for CI Accessibility
• Batch Validation Architecture