Compliance Reporting Dashboards

A compliance reporting dashboard turns thousands of per-route scan payloads into a small set of trustworthy numbers a program owner can read in ten seconds: open violations by impact, WCAG conformance coverage, week-over-week compliance drift, and mean time to remediation. The engineering problem is not drawing charts — it is landing every scan run into a time-series store as normalized, deduplicated facts so that a panel measures the same thing today that it measured last quarter. This guide is the measurement layer of the Accessibility Compliance Baseline for Enterprise Web Ops: where the baseline decides pass or fail at merge time, the reporting layer records what happened across every run and renders the trend, so leadership sees direction rather than a single red build.

The distinction matters because a dashboard and a gate are different machines with different failure costs. A gate is allowed to be strict and blunt — it blocks a deploy on one critical finding. A dashboard must be complete and stable: it counts everything, attributes each finding to an owning app, team, and route, and never double-counts the same violation because it arrived on two shards. Get the ingestion contract right and the panels are trivial; get it wrong and every number is quietly negotiable.

Prerequisites & Environment Parity #

The reporting layer consumes finished scan output; it does not launch browsers. What it needs pinned is the shape and provenance of the data flowing into it:

• A validated finding contract. Every payload must have already passed the JSON Schema validation for accessibility data contract before it reaches the warehouse. The dashboard trusts its inputs precisely because a truncated or malformed result was rejected upstream as a structural error, not silently counted as zero violations. Ingestion revalidates the envelope but assumes the finding objects are already well-formed.
• A stable finding identity. Deduplication depends on a deterministic fingerprint per finding (rule id + normalized selector + route). If the scan emits volatile selectors — nth-child indices that shift between runs — the same violation looks new every night and drift becomes noise. Pin the selector strategy in your axe-core enterprise configuration so identity is reproducible.
• A time-series store with an append-only fact table. Postgres with a timestamptz column and a b-tree index on (scanned_at, app_id) is enough to start; TimescaleDB, ClickHouse, or BigQuery become worthwhile past roughly ten million findings. Whatever the engine, findings are inserted, never updated — remediation is a new observation, not a mutation of the old one.
• A retention and rollup policy. Raw per-node findings are large and short-lived; the aggregates that feed trend lines must outlive them. The split between hot raw evidence and long-lived rollups is governed by the audit data storage and retention policies, and the dashboard must read from the tier that is actually retained for the window it displays.
• A pinned metric-definition version. MTTR, coverage, and drift each have a formula. Version it (metrics_v3) and stamp every rollup row with the version that produced it, so a redefinition does not silently bend a historical curve.

Treat the warehouse schema and the metric definitions as versioned infrastructure, reviewed like any other contract. A dashboard whose denominator changed without a migration note is worse than no dashboard, because it looks authoritative while lying.

Conceptual model: the findings warehouse as a fact store #

The whole design is one idea borrowed from analytics engineering: model each accessibility finding as an immutable fact in a narrow table, and compute every metric as an aggregation over facts scoped by dimensions (app, team, route, impact, WCAG criterion, time). Nothing is ever recomputed by re-reading browsers; a metric is a GROUP BY over rows that already exist.

A finding fact is deliberately thin. It carries the finding identity, the dimensions needed to slice it, the impact and criterion for filtering, and two timestamps — when it was first observed and, once remediated, when it closed. Everything a panel needs is derivable from that shape:

• Open violations by impact is a count of facts with no close timestamp, grouped by impact.
• WCAG conformance coverage % is the share of applicable success criteria with zero open violations, computed against a criterion registry rather than against the findings alone — an empty result set is not the same as full coverage, a trap the WCAG conformance scorecard guide handles directly.
• Compliance drift is opened − resolved within a period: a net flow, positive when a surface is regressing.
• Mean time to remediation is the average of closed_at − opened_at over findings that closed in the window, developed in full in the MTTR and drift tracking guide.

Ingestion is where correctness is won or lost. A single scan run arrives as many findings, often split across parallel shards. The ingester must upsert on finding identity so that the same violation seen on shard 3 and shard 7 of the same run collapses to one open fact, and so that a finding present last night but absent tonight is closed rather than duplicated. The data flow below traces a run from raw scan output to the panels a program owner reads.

The arrows are one-directional on purpose. Nothing downstream ever writes back into the findings store; panels and rollups are pure reads. That property is what lets you rebuild every dashboard from the fact table alone after a metric redefinition, and what lets you unit-test aggregations against a fixture of facts without touching a browser or a live database.

Step-by-Step Implementation #

The sequence below stands up the warehouse table, the identity-aware ingester, the rollup query, and a panel API. Each step is independently testable against fixtures.

1. Define the findings fact table #

Model findings as immutable facts with the dimensions every panel slices on. The finding_key is the deterministic fingerprint; closed_at is null while a finding is open.

CREATE TABLE finding_fact (
    finding_key   TEXT        NOT NULL,   -- sha256(rule_id | norm_selector | route)
    run_id        TEXT        NOT NULL,   -- the scan run that first opened it
    app_id        TEXT        NOT NULL,
    team_id       TEXT        NOT NULL,
    route         TEXT        NOT NULL,
    rule_id       TEXT        NOT NULL,   -- e.g. color-contrast, image-alt
    wcag_sc       TEXT        NOT NULL,   -- e.g. 1.4.3, 1.1.1, 2.5.8
    impact        TEXT        NOT NULL,   -- critical | serious | moderate | minor
    opened_at     TIMESTAMPTZ NOT NULL,
    closed_at     TIMESTAMPTZ,            -- null == still open
    metrics_ver   TEXT        NOT NULL DEFAULT 'metrics_v3',
    PRIMARY KEY (finding_key)
);

CREATE INDEX ix_fact_open   ON finding_fact (app_id, impact) WHERE closed_at IS NULL;
CREATE INDEX ix_fact_closed ON finding_fact (closed_at) WHERE closed_at IS NOT NULL;

The partial indexes matter at scale: open-violation panels scan only unresolved rows, and MTTR queries scan only the closed set, so neither pays for the other’s volume.

2. Ingest a run with identity-aware upsert #

The ingester reconciles the current run against the open set. Findings present now that were already open stay open; findings newly present open; findings previously open but absent now close. This is what makes the same violation seen on two shards one fact, and what turns a remediation into a closed_at stamp rather than a deletion.

import hashlib
from datetime import datetime, timezone

def finding_key(rule_id: str, selector: str, route: str) -> str:
    """Deterministic identity so the same violation collapses across shards/runs."""
    norm = selector.strip().lower()
    return hashlib.sha256(f"{rule_id}|{norm}|{route}".encode()).hexdigest()

def ingest_run(conn, run_id, app_id, team_id, route, findings, seen_at=None):
    """Reconcile one route's findings for one run against the open fact set.

    `findings` is the validated `violations` list from a scan payload. The
    payload is assumed to have already passed JSON Schema validation upstream.
    """
    seen_at = seen_at or datetime.now(timezone.utc)
    current = {}
    for v in findings:
        for node in v["nodes"]:
            key = finding_key(v["id"], node["target"][0], route)
            current[key] = (v["id"], v["wcag_sc"], v.get("impact") or "minor")

    with conn.cursor() as cur:
        # Open new or still-present findings; ON CONFLICT keeps the original opened_at.
        for key, (rule_id, sc, impact) in current.items():
            cur.execute(
                """INSERT INTO finding_fact
                     (finding_key, run_id, app_id, team_id, route,
                      rule_id, wcag_sc, impact, opened_at)
                   VALUES (%s,%s,%s,%s,%s,%s,%s,%s,%s)
                   ON CONFLICT (finding_key) DO UPDATE
                     SET closed_at = NULL""",       # a reopened finding clears its close
                (key, run_id, app_id, team_id, route,
                 rule_id, sc, impact, seen_at),
            )
        # Close facts for THIS route that were open but are absent from this run.
        cur.execute(
            """UPDATE finding_fact
                 SET closed_at = %s
               WHERE route = %s AND closed_at IS NULL
                 AND NOT (finding_key = ANY(%s))""",
            (seen_at, route, list(current.keys())),
        )
    conn.commit()

Scoping the close step by route is essential: a run that scanned only the checkout flow must not close findings on the account page it never visited. Reconcile per route, never across the whole surface at once.

3. Roll up facts into panel metrics #

Every headline number is a GROUP BY over facts. Open violations by impact and per-app rollups come straight from the open set.

-- Open violations by impact, per app — the primary dashboard panel.
SELECT app_id,
       impact,
       COUNT(*) AS open_count
FROM   finding_fact
WHERE  closed_at IS NULL
GROUP  BY app_id, impact
ORDER  BY app_id,
          array_position(ARRAY['critical','serious','moderate','minor'], impact);

Compliance drift is the net flow of findings within a period — opened minus resolved — computed by bucketing the two timestamps into the same window:

-- Weekly compliance drift per team: positive means the surface is regressing.
WITH opened AS (
    SELECT team_id, date_trunc('week', opened_at) AS wk, COUNT(*) n
    FROM finding_fact GROUP BY 1, 2),
resolved AS (
    SELECT team_id, date_trunc('week', closed_at) AS wk, COUNT(*) n
    FROM finding_fact WHERE closed_at IS NOT NULL GROUP BY 1, 2)
SELECT COALESCE(o.team_id, r.team_id)  AS team_id,
       COALESCE(o.wk, r.wk)            AS week,
       COALESCE(o.n, 0)               AS opened,
       COALESCE(r.n, 0)               AS resolved,
       COALESCE(o.n, 0) - COALESCE(r.n, 0) AS drift
FROM opened o
FULL OUTER JOIN resolved r ON o.team_id = r.team_id AND o.wk = r.wk
ORDER BY team_id, week;

4. Materialize rollups so panels stay cheap #

Querying raw facts on every dashboard load does not scale, and raw findings age out under retention long before the trend line should. Persist daily rollups into a small aggregate table that outlives the raw evidence, and point panels at it.

CREATE TABLE daily_rollup (
    day         DATE  NOT NULL,
    app_id      TEXT  NOT NULL,
    team_id     TEXT  NOT NULL,
    impact      TEXT  NOT NULL,
    open_count  INT   NOT NULL,
    opened      INT   NOT NULL,
    resolved    INT   NOT NULL,
    metrics_ver TEXT  NOT NULL,
    PRIMARY KEY (day, app_id, team_id, impact, metrics_ver)
);

A nightly job writes one row per (day, app, team, impact). Because rollups are keyed by metrics_ver, a formula change writes a parallel series rather than overwriting history, and a panel can render old and new definitions side by side during a migration.

5. Serve panels from a read-only API #

Expose rollups through a thin read-only endpoint so the visualization layer never touches the fact table directly. Keeping reads on the rollup tier enforces the dashboard-versus-gate boundary in code: this API has no authority to fail a build, only to describe one.

def open_by_impact(conn, app_id: str) -> dict[str, int]:
    """Latest open-violation counts by impact for one app, read from rollups."""
    with conn.cursor() as cur:
        cur.execute(
            """SELECT impact, open_count FROM daily_rollup
               WHERE app_id = %s AND day = (SELECT max(day) FROM daily_rollup)
               ORDER BY array_position(
                 ARRAY['critical','serious','moderate','minor'], impact)""",
            (app_id,),
        )
        return {impact: n for impact, n in cur.fetchall()}

Configuration Reference #

The knobs below govern how findings are identified, rolled up, and retained. Getting identity and grain right is what keeps every downstream number stable.

Verification & Testing #

Reporting bugs are silent — a wrong denominator still renders a confident chart — so assert on aggregations against fixtures with known outcomes rather than eyeballing a dashboard.

def test_multishard_run_does_not_double_count(conn):
    """The same finding on two shards must collapse to one open fact."""
    v = {"id": "color-contrast", "wcag_sc": "1.4.3", "impact": "serious",
         "nodes": [{"target": ["main .price"]}]}
    ingest_run(conn, "run1", "app-a", "team-x", "/cart", [v])  # shard 3
    ingest_run(conn, "run1", "app-a", "team-x", "/cart", [v])  # shard 7, same finding
    assert open_by_impact(rollup(conn), "app-a")["serious"] == 1

def test_absent_finding_closes_not_deletes(conn):
    """A finding gone from the next run is closed, preserving MTTR history."""
    v = {"id": "image-alt", "wcag_sc": "1.1.1", "impact": "critical",
         "nodes": [{"target": ["img.hero"]}]}
    ingest_run(conn, "r1", "app-a", "team-x", "/home", [v])
    ingest_run(conn, "r2", "app-a", "team-x", "/home", [])   # remediated
    row = fetch_one(conn, "SELECT closed_at FROM finding_fact")
    assert row["closed_at"] is not None

In CI, run these against an ephemeral Postgres before any rollup ships. A regression in identity or close-scope logic surfaces here as a count that is off by exactly the number of shards, which is the fastest possible diagnosis. Findings themselves should arrive already conforming to the validation contract, so ingestion can assume structure and focus its assertions on reconciliation.

Failure Modes & Troubleshooting #

Double-counting across shards. Symptom: open-violation counts jump by a clean multiple when parallelism increases. Root cause: findings are inserted rather than upserted on identity, so shard 3 and shard 7 each create a row for the same violation. Fix: make finding_key deterministic and ON CONFLICT DO UPDATE, and add the multi-shard test above so the regression fails CI instead of the dashboard.

Volatile selectors inflate drift. Symptom: drift oscillates every night even on a frozen codebase. Root cause: the selector inside finding_key contains positional nth-child indices that shift between renders, so a stable violation looks resolved-then-reopened. Fix: normalize selectors to a stable strategy in the scanner configuration, or key on a hashed component path rather than a raw DOM path.

Coverage reads 100% on an unscanned surface. Symptom: conformance coverage shows full compliance for an app that was never scanned. Root cause: coverage was computed as “criteria with no open findings” over an empty result set instead of against the applicable-criteria registry. Fix: divide by applicable success criteria from the registry and treat “never observed” as unknown, not passing — the scorecard guide formalizes this.

Whole-surface close wipes real state. Symptom: MTTR collapses toward zero and drift swings wildly after a partial run. Root cause: the close step reconciled against the whole app when a run scanned only a subset of routes, closing findings on pages it never visited. Fix: scope reconciliation to the routes actually present in the run, keyed by route.

Trend gaps after retention expiry. Symptom: charts show data for recent weeks but flatline or vanish further back. Root cause: panels read raw facts that aged out under the raw-retention window while the trend needs the longer rollup horizon. Fix: back trend panels with the daily_rollup tier and align the rollup retention with the longest window any dashboard displays, per the retention policy.

Frequently Asked Questions #

Related #

• Tracking Accessibility MTTR and Compliance Drift
• Building a WCAG Conformance Scorecard
• JSON Schema Validation for Accessibility Data
• Audit Data Storage & Retention Policies
• Accessibility Compliance Baseline for Enterprise Web Ops