WCAG Audit Automation

axe-core vs Lighthouse for CI Accessibility

A Lighthouse accessibility score cannot back a CI gate because it is a weighted 0–100 rollup of a subset of axe-core’s rules — a page can score 100 and still fail success criteria the subset never checks, and a single new critical violation may barely move the number. Run axe-core directly against your route when you need a pass/fail gate; keep the Lighthouse score only as a rough, human-facing trend indicator. This guide answers the narrow question of why the two diverge and how to gate correctly, and sits within the Scanner Tool Selection and Benchmarking topic area under Automated Scanning & Dynamic Content Ingestion. It is deliberately distinct from the three-way engine comparison: this is about score-versus-gate, not tool-versus-tool.

When This Applies

Reach for this when someone proposes gating merges on a Lighthouse accessibility score, or when a green Lighthouse number sits next to a page you know has real barriers. The specific conditions:

  • Your CI already runs Lighthouse for performance, and there is pressure to reuse its accessibility category as the a11y gate “for free”.
  • A page reports a high Lighthouse accessibility score yet fails a manual keyboard or screen-reader pass.
  • You need a defensible pass/fail signal for WCAG 2.2 AA, not a composite score that blends many checks into one number.

If you only want a directional dashboard metric for stakeholders and are not gating on it, the Lighthouse score is fine as-is — the problem is exclusively about using it as a merge block.

Minimal Reproducible Example

The trap is a CI step that fails the build when the Lighthouse accessibility score drops below a threshold. It looks quantitative and objective, and it is neither for conformance purposes:

// lh_gate_naive.js — DO NOT SHIP as an a11y conformance gate.
const lighthouse = require('lighthouse');
const chromeLauncher = require('chrome-launcher');

async function gateOnLighthouse(url) {
  const chrome = await chromeLauncher.launch({ chromeFlags: ['--headless'] });
  const runnerResult = await lighthouse(url, { onlyCategories: ['accessibility'], port: chrome.port });
  const score = runnerResult.lhr.categories.accessibility.score * 100; // 0–100
  await chrome.kill();

  // WRONG: a score of 92 can hide a keyboard trap the subset never tests,
  // and adding one serious violation might drop the score by only 3 points.
  if (score < 90) process.exit(1);
  process.exit(0);
}

Two flaws make this unfit as a gate. The score is a subset of axe rules — Lighthouse bundles a curated selection, not the full engine — so criteria outside that subset are invisible to it. And the score is weighted and averaged, so violation severity is smeared: a critical image-alt failure (SC 1.1.1) and a minor nit can move the number by similar amounts, which means no score threshold cleanly separates “ship” from “block”.

Correct Implementation

Gate on axe-core’s structured findings bucketed by impact, and let Lighthouse be a non-gating trend line if you want it at all. The gate reads discrete violations, not a composite score:

# axe_gate.py — gate on discrete axe-core violations, not a rolled-up score.
from playwright.sync_api import sync_playwright

# The full engine, pinned in your lockfile — not Lighthouse's curated subset.
AXE_OPTS = "{runOnly:{type:'tag',values:['wcag2a','wcag2aa','wcag22aa']}}"

def scan(url: str) -> list[dict]:
    with sync_playwright() as p:
        browser = p.chromium.launch()
        page = browser.new_context().new_page()
        page.goto(url, wait_until="networkidle")
        page.add_script_tag(path="node_modules/axe-core/axe.min.js")
        result = page.evaluate(f"() => axe.run(document, {AXE_OPTS})")
        browser.close()
        return result["violations"]   # each carries a discrete `impact` label

def gate(url: str) -> int:
    violations = scan(url)
    # Block on severity, not on an average. A single critical/serious fails.
    hard = [v for v in violations if v.get("impact") in {"critical", "serious"}]
    for v in hard:
        # e.g. rule_id="image-alt" -> SC 1.1.1, or "color-contrast" -> SC 1.4.3
        print(f"[BLOCK] {v['id']} ({v.get('impact')}) — {len(v['nodes'])} nodes")
    return 1 if hard else 0

The difference is categorical, not cosmetic. axe-core hands you every violation as a discrete record with an impact you can threshold on and a rule id that maps to a WCAG success criterion; Lighthouse hands you one number that has already collapsed that structure. Gating needs the structure back. If you still want the Lighthouse score, compute it in a separate, non-blocking step that reports the trend without holding a green build hostage to a metric it was never designed to enforce.

Pipeline Integration Note

The axe-core gate here is the same one that scales across a real property: its discrete output normalizes into the record enforced by JSON Schema validation for accessibility data, shards through the batch validation architecture, and the rules it runs are tuned once in the axe-core enterprise configuration. Lighthouse has no place in that gating path, but its accessibility trend can live alongside the performance metrics your team already tracks — a stakeholder-facing number, explicitly labeled as indicative rather than conformance-grade. When you choose between running one engine or several for the gate, the broader trade-offs are laid out in the parent Scanner Tool Selection and Benchmarking guide.

Gotchas

  • A perfect Lighthouse score is routinely non-conformant. Because the subset omits checks and cannot evaluate anything requiring human judgement, a page can score 100 and still fail keyboard operability (SC 2.1.1) or focus order (SC 2.4.3). Never communicate a 100 as “accessible” — communicate it as “passed the automated subset Lighthouse runs”.
  • Lighthouse’s single-run scoring is noisy in CI. The score can vary run to run on the same commit because of timing and throttling, so even as a trend metric it needs multiple runs or a tolerance band; a hard threshold on a jittery number produces phantom failures. The discrete axe gate, driven from a stabilized DOM, does not have this problem.
  • Authenticated routes defeat a default Lighthouse run. Pointed at a bare URL it scores the login or marketing shell, inflating the number for an app it never reached. The axe-on-Playwright gate injects a captured session so it scores the real authenticated surface — the routes where the gate actually matters.

Frequently Asked Questions

Why can a page score 100 on Lighthouse accessibility and still fail WCAG?

Because the score reflects only the curated subset of axe rules Lighthouse bundles, and only the machine-decidable part of those. Criteria that require human judgement — meaningful focus order, sensible reading sequence, appropriate alt text — sit entirely outside what any automated subset can verify, so a perfect score means “passed the automated checks Lighthouse runs”, not “conformant”.

Is the Lighthouse accessibility score useful for anything?

Yes, as a low-effort directional trend for stakeholders who want a single number moving in the right direction over time. It is genuinely useful for spotting a large regression at a glance and for non-technical reporting. It is only unsuitable as a merge gate, where you need discrete, severity-labeled violations rather than a weighted average.

Does Lighthouse run axe-core under the hood?

It runs a subset of axe-core’s rules, not the whole engine, and then weights and averages the outcomes into its category score. That is why a direct axe run reports violations Lighthouse never mentions: the rules outside the bundled subset simply are not evaluated, and the ones inside it lose their individual severity in the rollup.