WCAG Audit Automation

Building a WCAG Conformance Scorecard

A WCAG conformance scorecard reports, per success criterion and per level, whether a surface is passing, failing, or unevaluated — computed by joining findings against a registry of applicable criteria rather than counting violations alone. This page shows how to build that scorecard so an empty result reads as “not tested” instead of “compliant”, and shapes the output so it can later feed a graded WCAG 3.0 score instead of a flat pass/fail.

The scorecard is the criterion-coverage view within the Compliance Reporting Dashboards layer of the Accessibility Compliance Baseline for Enterprise Web Ops. Where the MTTR and drift tracking guide measures remediation flow over time, this page measures conformance state across the criteria themselves — a structural snapshot, not a time series.

When This Applies

Build a scorecard when the question is “which WCAG success criteria do we actually conform to?” rather than “how many violations are open?”. It applies when:

  • A conformance claim or VPAT needs a per-criterion basis — Level A and AA broken out criterion by criterion, each marked supports, does-not-support, or not-evaluated.
  • A raw violation count hides that whole criteria are untested; a surface with zero 2.5.8 findings might have full target-size conformance or might simply never have been checked for it.
  • You want a score that degrades gracefully toward the outcome-based, graded model in the WCAG 2.2 vs 3.0 success criteria taxonomy, rather than a binary that collapses on a single failure.

It does not apply to remediation-speed questions, which are time-based and belong to the MTTR and drift guide. The two are complementary: this scorecard tells you what conforms, that guide tells you how fast gaps close.

Minimal Reproducible Example

The defining mistake is inferring conformance from the absence of findings. It reports a perfect score for a criterion nobody ever tested.

# scorecard_naive.py — WRONG. Absence of a violation is treated as a pass.
def scorecard(findings):
    all_criteria = {"1.1.1", "1.4.3", "2.1.1", "2.4.3", "2.4.7", "2.5.8"}
    failing = {f["wcag_sc"] for f in findings}
    # Any criterion without a finding is declared "pass" — even if untested.
    return {sc: ("fail" if sc in failing else "pass") for sc in all_criteria}

The flaw is that pass and not-evaluated are folded together. A criterion that no rule maps to — most of WCAG is not machine-testable — comes back pass, inflating the score and turning the scorecard into a source of false assurance in exactly the audits it is meant to support.

Correct Implementation

Drive the scorecard from a registry of applicable criteria, and mark each as fail if any finding maps to it, pass only if it was in scope of the scan and clean, and not-evaluated otherwise. The registry records which criteria automation can decide, so machine silence never masquerades as conformance.

from dataclasses import dataclass

# Registry: which criteria are in scope, their level, and whether automation
# can decide them. Sourced from the criterion taxonomy, not from findings.
@dataclass(frozen=True)
class Criterion:
    sc: str
    level: str          # "A" or "AA"
    machine_testable: bool

REGISTRY = [
    Criterion("1.1.1", "A",  True),
    Criterion("1.4.3", "AA", True),
    Criterion("2.1.1", "A",  False),   # keyboard operability: partial automation
    Criterion("2.4.3", "A",  False),   # focus order: needs human judgment
    Criterion("2.4.7", "AA", True),
    Criterion("2.5.8", "AA", True),    # target size, new in 2.2
]

def build_scorecard(findings, evaluated_scs: set[str]) -> dict:
    """Per-criterion conformance state from findings plus the registry.

    `findings`      : violation objects carrying a wcag_sc field.
    `evaluated_scs` : criteria the scan actually exercised this run.
    """
    failing = {f["wcag_sc"] for f in findings}
    rows = []
    for c in REGISTRY:
        if c.sc in failing:
            state = "fail"
        elif c.machine_testable and c.sc in evaluated_scs:
            state = "pass"           # in scope, exercised, and clean
        else:
            state = "not-evaluated"  # no rule, or never exercised this run
        rows.append({"sc": c.sc, "level": c.level, "state": state})
    return {"rows": rows, "summary": _summarize(rows)}

def _summarize(rows) -> dict:
    """Coverage % = passing / (passing + failing), by level. Excludes
    not-evaluated from the denominator so coverage never counts silence."""
    out = {}
    for level in ("A", "AA"):
        lv = [r for r in rows if r["level"] == level]
        passed = sum(r["state"] == "pass" for r in lv)
        failed = sum(r["state"] == "fail" for r in lv)
        decided = passed + failed
        out[level] = {
            "passed": passed,
            "failed": failed,
            "not_evaluated": sum(r["state"] == "not-evaluated" for r in lv),
            "coverage_pct": round(100 * passed / decided, 1) if decided else None,
        }
    return out

The coverage denominator is passed + failed, never the full criterion list — dividing by everything would let untested criteria drag the score toward zero just as folding them into pass drags it toward 100. Both are wrong; only the decided set is honest.

To make the scorecard forward-compatible with WCAG 3.0’s graded scoring, emit a normalized 0–1 score per criterion instead of only a boolean, so the same structure can carry a partial rating later without a schema change:

def graded_score(row, weights=None) -> float:
    """0.0 fail, 1.0 pass, and None for not-evaluated so it is excluded from
    any weighted roll-up. Ready to carry WCAG 3.0 partial ratings (0.0-1.0)."""
    weights = weights or {}
    if row["state"] == "not-evaluated":
        return None
    base = 1.0 if row["state"] == "pass" else 0.0
    return base * weights.get(row["sc"], 1.0)

Storing a float rather than a flag is the single decision that lets a 2.2 pass/fail scorecard evolve into a 3.0 graded score without rebuilding the warehouse.

Pipeline Integration Note

The scorecard reads the same finding_fact rows the parent compliance reporting dashboards guide populates, joined against the criterion registry maintained alongside the WCAG 2.2 vs 3.0 success criteria taxonomy. The evaluated_scs set is not guessable from findings — it must be recorded by the scanner as the criteria its active rule set exercised, which is fixed in the axe-core enterprise configuration. Without that provenance the scorecard cannot tell a clean pass from an untested criterion, which is the entire distinction it exists to preserve.

Gotchas

  • One rule can map to several criteria, and one criterion to several rules. color-contrast decides 1.4.3, but 2.5.8 target-size may be probed by more than one check. Resolve the rule-to-criterion mapping in the registry, and mark a criterion fail if any mapped rule fails and pass only if all mapped rules were exercised and clean.
  • Partial automation must not read as a full pass. Criteria like 2.1.1 keyboard and 2.4.3 focus order are only partially machine-decidable. Mark them not-evaluated for the automated scorecard and route them to manual review rather than letting an automated clean scan claim conformance the tooling cannot actually verify.
  • Multi-route apps need criterion state aggregated, not averaged. A criterion is conformant for an app only if it passes on every route where it was evaluated; a single failing route makes the criterion fail for the app. Roll up with a worst-case fold across routes, not a percentage, so one broken page cannot be averaged away.

Frequently Asked Questions

Why not treat the absence of a violation as a pass?

Because absence has two meanings — either the criterion was tested and clean, or it was never tested at all — and most WCAG criteria are not fully machine-testable. Folding untested criteria into pass inflates the score and produces false assurance in the conformance claims the scorecard is meant to support, so untested criteria must be marked not-evaluated instead.

How does a scorecard differ from the compliance drift metric?

The scorecard is a structural snapshot of which success criteria conform right now, criterion by criterion; drift is a time-based net flow of findings opened versus resolved. One answers what passes, the other answers how the backlog is moving, and they read the same findings for different questions.

How do I make the scorecard ready for WCAG 3.0 graded scoring?

Store a normalized 0-to-1 score per criterion rather than only a boolean pass or fail, and keep not-evaluated as a distinct null that is excluded from any weighted roll-up. That structure can carry a partial rating later without a schema migration, so a 2.2 pass/fail scorecard evolves into a 3.0 graded score in place.

What denominator should conformance coverage percentage use?

Use the decided set — passing plus failing criteria — and exclude not-evaluated criteria entirely. Dividing by every criterion drags coverage toward zero on untested criteria, while counting untested criteria as passing drags it toward 100; only the criteria automation actually decided give an honest percentage.