Baseline Scanning Setup for Enterprise Web Ops

Standing up the first reliable accessibility-scanning baseline in a repository is less about picking a scanner and more about making its output deterministic: a pinned engine, a version-controlled route manifest, environment parity that eliminates “works on my machine” disputes, and a committed snapshot of accepted known-issues so the gate enforces no new regressions rather than an unreachable zero-violation ideal. This guide is the on-ramp to the broader accessibility compliance baseline for enterprise web ops section: it shows how to lay out the project, pin the runtime, describe the surface once, and freeze a baseline so every subsequent scan measures drift against a known-good state instead of relitigating debt the team has already triaged.

The trap most teams fall into is starting the gate at zero violations. A real enterprise application carries dozens of pre-existing, already-ticketed findings on day one. A gate that blocks on all of them is disabled within a sprint; a gate that blocks on none of them catches nothing. The baseline snapshot is what resolves that tension — it records the exact set of findings present when the gate went live, and the gate fails only when a scan produces a finding that is not already in that snapshot.

Prerequisites & Environment Parity #

A baseline is only meaningful if the environment that produces it is reproducible. Two scans of the same commit must yield byte-identical findings, or the snapshot drifts and the gate produces phantom regressions. Pin the following before you record a single result:

• axe-core, pinned to an exact version (axe-core@4.10.2) in package.json with a committed lockfile. A minor engine bump can add a rule or relabel an impact, which changes counts under a fixed configuration and invalidates the snapshot.
• Playwright for Python, pinned to an exact release (playwright==1.44.0) with its browser provisioned through playwright install --with-deps chromium rather than a system Chrome that updates on its own. The engine configuration itself is governed by the axe-core enterprise configuration guide, and the traversal and DOM-stabilization mechanics live in Playwright headless scanning workflows.
• pytest, pinned (pytest==8.2.0) and used purely as the runner that drives the scan and asserts against the snapshot. It is the harness, not the scanner.
• A parity-defined environment. The baseline records findings for one specific rendering context. Fix the viewport (1280x800), the locale (en-US), the timezone, and the reduced-motion preference explicitly, because target-size (SC 2.5.8), color-contrast (SC 1.4.3), and reflow findings all shift when those dimensions move. A baseline captured on staging must be captured against a prod-equivalent build — same bundle, same feature flags — or the snapshot describes a page that never ships.

Treat the whole scanning setup as infrastructure-as-code. The configuration, the manifest, and the snapshot are committed files versioned alongside the application, not runner-local state or a dashboard setting someone can quietly edit.

The Baseline Snapshot Model #

The mechanism that makes a first baseline usable is a two-file split: a route manifest that declares what gets scanned, and a baseline snapshot that declares which findings are already accepted. The scanner reads the manifest, produces a fresh result set, and diffs that set against the snapshot. Anything in the fresh set but not in the snapshot is a new regression and fails the gate. Anything in the snapshot but not in the fresh set is a fixed issue the snapshot can shed.

The route manifest is a flat, reviewable description of the surface. Each entry names a route, the journey or auth state required to reach it, and the parity context it must render in. Keeping it as data — not scattered across test files — means adding a route to the gate is a one-line diff a reviewer can reason about, and the same manifest drives local runs and CI identically.

The snapshot is a set of stable fingerprints, not raw axe payloads. A fingerprint is a hash of the dimensions that identify a finding independently of run-to-run noise: the rule id, the impact, the route, and a normalized target selector — never volatile fields like screenshot bytes or a node’s live handle. Fingerprinting on stable dimensions is what lets the same underlying violation match across two runs so it is recognized as known rather than flagged as new.

The gate’s rule is deliberately asymmetric: a fingerprint present in the scan but absent from the snapshot is a regression and blocks; a fingerprint present in the snapshot but absent from the scan is progress and can be pruned. That asymmetry is what turns an intimidating pile of legacy debt into a hard line the team can actually hold — nobody is asked to fix everything at once, but nobody can add anything new.

Step-by-Step Implementation #

The sequence below produces a working baseline you can commit and gate on. Each step is independently testable.

1. Lay out the repository #

Keep scanning concerns in one place so configuration, manifest, and snapshot are reviewed together and cannot drift apart.

repo/
├── a11y/
│   ├── axe-config.json          # engine config: context + options (see axe-core guide)
│   ├── routes.manifest.json     # what to scan, and in which parity context
│   ├── baseline.snapshot.json   # accepted known-issue fingerprints
│   ├── scan.py                  # drives Playwright + axe against one route
│   └── test_baseline.py         # pytest: scan the manifest, diff the snapshot
├── package.json                 # pins axe-core exactly
└── requirements.txt             # pins playwright, pytest, jsonschema

2. Declare the route manifest #

The manifest is data, not code. Every route carries the parity context it must render in, so the scan is never ambiguous about viewport or auth state.

{
  "parity": { "viewport": [1280, 800], "locale": "en-US", "timezone": "UTC", "reducedMotion": true },
  "base_url": "https://staging.example.com",
  "routes": [
    { "id": "home",      "path": "/",           "auth": null },
    { "id": "dashboard", "path": "/app/home",    "auth": "member" },
    { "id": "billing",   "path": "/app/billing", "auth": "admin" }
  ]
}

3. Fingerprint each finding on stable dimensions #

A fingerprint must survive benign run-to-run variation. Hash only the identifying fields — never a screenshot, never a live element handle — so the same violation produces the same fingerprint on every runner.

import hashlib

def fingerprint(route_id: str, violation: dict, node: dict) -> str:
    """Stable identity for one violation on one node, independent of run noise.

    Uses rule id, impact, route, and the FIRST css target selector. Volatile
    fields (html snippet, xpath ordinals, screenshots) are deliberately excluded
    so a cosmetic DOM shuffle does not read as a brand-new finding.
    """
    target = node.get("target", ["<unknown>"])[0]
    basis = "|".join([route_id, violation["id"], violation.get("impact") or "minor", target])
    return hashlib.sha256(basis.encode("utf-8")).hexdigest()[:16]

def fingerprints_for_result(route_id: str, result: dict) -> set[str]:
    return {
        fingerprint(route_id, v, node)
        for v in result["violations"]
        for node in v["nodes"]
    }

4. Record the initial baseline #

Run every route in the manifest once, collect all fingerprints, and write them as the accepted set. This is the one moment you intentionally accept existing debt; every run after this measures against it.

import json
from pathlib import Path

def record_baseline(manifest: dict, scan_route) -> None:
    """Scan the whole manifest and freeze the current findings as accepted.

    `scan_route(route, parity)` returns an axe result dict for one route.
    Run this deliberately (e.g. `python -m a11y.scan --record`), never in CI.
    """
    accepted = {}
    for route in manifest["routes"]:
        result = scan_route(route, manifest["parity"])
        accepted[route["id"]] = sorted(fingerprints_for_result(route["id"], result))
    Path("a11y/baseline.snapshot.json").write_text(
        json.dumps({"version": 1, "accepted": accepted}, indent=2), encoding="utf-8"
    )

5. Diff every subsequent scan against the snapshot #

CI never records; it only compares. New fingerprints fail the gate, and — critically — the gate reports both regressions and resolved issues so the snapshot can be kept honest.

def diff_against_baseline(route_id: str, result: dict, snapshot: dict) -> dict:
    """Compare a fresh scan of one route to its accepted fingerprints."""
    accepted = set(snapshot["accepted"].get(route_id, []))
    current = fingerprints_for_result(route_id, result)
    return {
        "new": sorted(current - accepted),      # regressions -> fail the gate
        "fixed": sorted(accepted - current),    # progress -> prune from snapshot
    }

Configuration Reference #

The fields below drive the manifest and snapshot. The engine-side options (runOnly, rules, resultTypes) live in axe-config.json and are documented in the axe-core configuration guide.

A note on scope: an empty routes array or an all-null auth set is a silent coverage gap, not a passing baseline. Validate the manifest against a schema in CI — the same discipline described in JSON Schema validation for accessibility data — so an empty or malformed manifest fails loudly rather than gating nothing.

Verification & Testing #

Prove three things before you trust the baseline: that a re-scan of the recorded commit produces zero new fingerprints, that an injected violation is detected as new, and that a resolved issue is reported as fixed. The pytest below wires the diff into the runner.

import json
import pytest
from pathlib import Path
from a11y.scan import scan_route
from a11y.baseline import fingerprints_for_result, diff_against_baseline

MANIFEST = json.loads(Path("a11y/routes.manifest.json").read_text())
SNAPSHOT = json.loads(Path("a11y/baseline.snapshot.json").read_text())

@pytest.mark.parametrize("route", MANIFEST["routes"], ids=lambda r: r["id"])
def test_no_new_regressions(route):
    """The gate: a fresh scan may not introduce a fingerprint the snapshot lacks."""
    result = scan_route(route, MANIFEST["parity"])
    delta = diff_against_baseline(route["id"], result, SNAPSHOT)
    assert delta["new"] == [], (
        f"{route['id']}: {len(delta['new'])} new finding(s) not in baseline: {delta['new']}"
    )

def test_snapshot_is_not_empty():
    """Guard against a manifest or snapshot that silently gates nothing."""
    assert MANIFEST["routes"], "route manifest is empty — nothing is being scanned"
    assert SNAPSHOT["accepted"], "baseline snapshot has no routes — coverage gap"

Run it locally against the same staging build first, confirm it is green on an unchanged commit, then wire it into CI as the first single-journey accessibility gate before you expand to the full manifest. Role-gated routes need their session handled separately, which is covered in scanning authenticated application states.

Failure Modes & Troubleshooting #

Unpinned engine invalidates the snapshot overnight. Symptom: a green baseline starts reporting new fingerprints with no code change. Root cause: axe-core resolved from a floating range and a minor bump added a rule or relabeled an impact. Fix: pin the exact version in the lockfile, inject the engine from node_modules, and bump snapshot.version deliberately whenever you do choose to upgrade so the re-record is a reviewed event.

Missing or empty manifest gates nothing. Symptom: CI is green but no violation ever blocks a merge. Root cause: the routes array is empty, or every route silently 404s to a marketing shell with zero findings. Fix: validate the manifest against a schema, assert the route count is non-zero, and assert a route-specific landmark is present before trusting a clean result.

Empty scope from a mis-scoped include. Symptom: routes report zero violations even when visibly broken. Root cause: the axe include selector had not rendered when the scan fired, so the engine evaluated an empty subtree and recorded an empty fingerprint set into the baseline. Fix: stabilize the DOM in the traversal layer and guard against an include scope that matches no elements, as detailed in the Playwright workflows guide.

Fingerprint churn from volatile selectors. Symptom: the same violation flips between “known” and “new” across runs. Root cause: the fingerprint hashed a positional selector like :nth-child(4) that shifts when unrelated markup moves. Fix: fingerprint on the most stable available target (an id or a stable component selector) and exclude ordinal-based paths from the basis.

Parity drift between record and gate. Symptom: the baseline recorded on a laptop fails in CI. Root cause: viewport, locale, or reduced-motion differed between the record run and the gate run, flipping target-size and color-contrast findings. Fix: read the parity block from the manifest in both contexts so the recording environment and the gating environment are provably identical.

Frequently Asked Questions #

Related #

• Axe-Core Enterprise Configuration
• Playwright Headless Scanning Workflows
• Your First Single-Journey Accessibility Gate
• Scanning Authenticated Application States
• JSON Schema Validation for Accessibility Data