WCAG Audit Automation

Distinguishing Flaky Violations from Real Regressions

To keep an accessibility gate credible, a violation that appears on one run and vanishes on the next must be quarantined and confirmed across several runs before it becomes a ticket — while a finding that reproduces every single time is promoted straight to a regression. This page resolves the specific failure where timing races, hydration order, animation frames, and network variance make the same commit report different violations run to run, so a real defect drowns in intermittent noise and the team stops trusting the scan.

This is the intermittency-handling reference within the error categorization and triage pipelines guide, itself part of the broader automated scanning and dynamic content ingestion strategy for auditing client-rendered applications. It is deliberately narrower than the sibling page on categorizing false positives in automated scan results: a false positive is a finding that is wrong every time the engine sees that state, whereas a flaky violation is a finding that is correct sometimes and absent other times on the same page. Different problem, different fix — this page owns the second one.

When This Applies

Reach for flakiness handling when the violation set is unstable across identical runs, not when it is stably wrong. The tell is that rerunning the exact same commit changes the report:

  • A color-contrast or aria-required-attr finding shows up on run 1, disappears on run 2, and returns on run 4, with no code change between them.
  • A rule fires only when the scan snapshot lands mid-animation — a fade-in still at partial opacity, a spinner that the engine catches before content swaps in.
  • The violation count correlates with runner load: it climbs on a busy CI fleet and drops on a quiet one, because a slower machine snapshots the DOM before hydration completes.
  • A route behind a flaky upstream API sometimes renders an error shell (its own, different violations) instead of the real page.

If instead a finding reproduces on every run but is not perceivable by assistive technology, that is a false positive and belongs in the false-positive scoring stage, not here. Flakiness is specifically non-determinism: the same input state producing different outputs.

Minimal Reproducible Example

The naive triage step tickets whatever a single run reports. Feed it an intermittent finding and it opens — then, next cycle, auto-closes and reopens — a ticket that no one can reproduce on demand.

# triage_naive.py — DO NOT SHIP. One run decides everything.
import json
from pathlib import Path

report = json.loads(Path("results.json").read_text(encoding="utf-8"))

for violation in report["violations"]:
    for node in violation["nodes"]:
        # A single mid-hydration snapshot tickets a finding that vanishes next run.
        create_ticket(violation["id"], node["target"][0])

The problem is not the rule and not the node — it is the sample size. One snapshot of a page that hydrates asynchronously is a single draw from a distribution, and a violation present in 30% of draws will ticket on the runs it appears and silently drop on the runs it does not. Ticketing on N=1 turns every timing race into a phantom regression that churns the tracker.

Correct Implementation

Confirm a finding across several runs before it can become a ticket. The pattern has three parts: give every violation a stable fingerprint so the same finding is recognizable across runs, run each route N times and require the finding in a quorum of runs, and track a per-fingerprint flakiness rate so persistently unstable rules get quarantined instead of re-litigated every cycle.

Start with a fingerprint that survives across runs. It must exclude anything volatile — never the DOM snapshot HTML, never a timestamp — and key only on the rule id plus a stable selector.

# fingerprint.py — a run-stable identity for a violation instance.
import hashlib

def fingerprint(route: str, rule_id: str, target: list[str]) -> str:
    # target is axe-core's CSS selector path for the node. Use the FIRST
    # (most specific) selector only; deeper frame selectors can shift run to run.
    selector = target[0] if target else ""
    key = f"{route}|{rule_id}|{selector}"
    return hashlib.sha1(key.encode("utf-8")).hexdigest()[:16]

Run the route several times and count how many runs each fingerprint appears in. A finding present in every run is stable; one present in a minority is flaky and must not ticket until it crosses a confirmation quorum.

# confirm.py — N-run confirmation with a stabilization retry.
from collections import Counter
from fingerprint import fingerprint

N_RUNS = 5          # samples per route
QUORUM = 4          # a finding must appear in >= 4 of 5 runs to be "real"

def scan_with_stabilization(page, route):
    # Retry the SCAN, not the assertion: settle the DOM before each snapshot so
    # a race is given every chance to resolve before it is recorded as a finding.
    page.goto(route, wait_until="networkidle")
    page.wait_for_load_state("networkidle")
    page.evaluate("() => document.fonts.ready")          # webfonts affect contrast
    page.wait_for_timeout(400)                            # let entry animations finish
    return run_accessibility_scan(page, route)           # -> {"violations": [...]}

def confirm_route(page, route, n_runs=N_RUNS):
    seen = Counter()                 # fingerprint -> number of runs it appeared in
    detail = {}                      # fingerprint -> (rule_id, selector) for tickets
    for _ in range(n_runs):
        results = scan_with_stabilization(page, route)
        # Count each fingerprint AT MOST once per run, so a rule that flags the
        # same node twice in one snapshot cannot fake a quorum on its own.
        this_run = set()
        for v in results["violations"]:
            for node in v["nodes"]:
                fp = fingerprint(route, v["id"], node["target"])
                if fp not in this_run:
                    this_run.add(fp)
                    seen[fp] += 1
                    detail[fp] = (v["id"], node["target"][0])
    return seen, detail

def classify(seen, n_runs=N_RUNS, quorum=QUORUM):
    real, flaky = [], []
    for fp, hits in seen.items():
        rate = hits / n_runs
        if hits >= quorum:
            real.append((fp, rate))      # reproducible -> promote to regression
        else:
            flaky.append((fp, rate))     # intermittent -> quarantine, do not ticket
    return real, flaky

Only the real set is allowed to create tickets. The flaky set is appended to a persistent quarantine store with its observed rate, so its instability is tracked over time rather than rediscovered every run. A rule that is chronically flaky on a route is a signal in itself — usually an un-awaited animation or an upstream data race — and it should be surfaced to the route owner as a stabilization task, not filed as an accessibility defect.

# quarantine.py — persist flakiness rate per fingerprint across cycles.
import json, time
from pathlib import Path

STORE = Path("flakiness.json")

def update_quarantine(flaky, route):
    # Exponential moving average of the per-fingerprint flakiness rate, so a
    # finding that becomes stable over time eventually graduates out of quarantine.
    store = json.loads(STORE.read_text(encoding="utf-8")) if STORE.exists() else {}
    for fp, rate in flaky:
        prev = store.get(fp, {}).get("rate", rate)
        store[fp] = {
            "route": route,
            "rate": round(0.7 * prev + 0.3 * rate, 3),   # smooth, don't overreact
            "last_seen": int(time.time()),
        }
    STORE.write_text(json.dumps(store, indent=2), encoding="utf-8")
    return store

The confirmation quorum is what separates this from a retry loop that merely hides failures. A retry that stops at the first clean run would suppress genuine intermittent regressions; requiring a finding in a majority of runs, and recording the rest as tracked flakiness, keeps both the real defect and the noise visible without letting the noise become tickets.

Pipeline Integration Note

N-run confirmation sits between the raw scan and the ticketing stage of the error categorization and triage pipelines: findings that clear the quorum flow on to selector deduplication and component-owner routing, while quarantined fingerprints are diverted to a stabilization queue instead. Because the extra runs multiply scan cost, run confirmation only on routes whose single-pass result changed since the last cycle — and lean on the stabilization guards in the Playwright headless scanning workflows guide to drive the base flakiness rate down at the source, so most routes settle deterministically and never need the N-run tax. The distinction feeds the gate directly: only confirmed findings count toward the committed critical and serious threshold, so a flaky violation can no longer flip a deploy on runner load alone.

Gotchas

  • Retrying until green hides real intermittent regressions. A retry loop that accepts the first clean run treats a defect present in 40% of loads as passing. Always run a fixed N and require a quorum, so a finding that is genuinely intermittent is still promoted rather than retried away.
  • Authenticated and multi-tenant routes flake for non-timing reasons. A session that expires mid-run, or a tenant router that occasionally serves a different shell, produces run-to-run variance that is not a hydration race. Pin storage_state and assert a tenant-specific landmark before scanning, so genuine flakiness is not confused with an auth or routing wobble.
  • Viewport and reduced-motion settings change the flake surface. target-size and animation-dependent findings flip with viewport width and with prefers-reduced-motion. Fix both in the scan context and force reduced motion where you can, so entry animations do not manufacture intermittent contrast and target-size findings the user would never hit.

Frequently Asked Questions

How is a flaky violation different from a false positive?

A false positive reproduces every time the engine evaluates that state but is not actually perceivable by assistive technology, so it is deterministically wrong. A flaky violation is non-deterministic: the same page and commit report it on some runs and not others, usually because of a timing or hydration race. The first is fixed by suppression with an audit trail; the second by confirming across multiple runs before ticketing.

How many runs do I need before trusting a finding?

Enough to separate signal from noise given your base flakiness rate — five runs with a quorum of four is a workable default for a moderately racy single-page application. If a rule still swings above and below that quorum across cycles, that is itself the finding: the route has a stabilization problem the owner should fix rather than a threshold you should keep tuning.

Won't running every route five times make CI too slow?

Only if you confirm everything every cycle. Run the single pass first and trigger N-run confirmation solely on routes whose result changed since the last run, so the multiplier lands on a handful of routes rather than the whole manifest. Pushing base flakiness down with proper stabilization waits keeps that set small.

Should a stabilization retry go around the scan or around the assertion?

Around the scan. Retrying the assertion just re-reads the same stale snapshot, whereas re-settling the DOM — waiting for network idle, fonts, and entry animations before each snapshot — gives a race the chance to resolve before it is ever recorded. The N-run confirmation then measures what remains after honest stabilization.