Sharding Audit Jobs Across Worker Pools

To scan a route manifest of thousands of URLs inside a CI time budget, split the manifest into balanced shards, hand each shard to a parallel worker that owns its own browser and writes its own artifacts, then converge every shard’s violation counts in a single final gate — assigning each route to a shard by a stable hash so the split is reproducible and no route is scanned twice. This page resolves the specific failure where one giant serial scan either times out the runner or, once naively parallelized, double-counts violations and produces a gate result nobody can reproduce.

This is the parallel-execution reference within the batch validation architecture guide, which itself sits inside the broader automated scanning and dynamic content ingestion strategy for auditing large client-rendered estates. The parent guide defines the manifest format and the aggregate schema; this page owns only the fan-out and fan-in — how to divide the work deterministically and merge the results without corrupting the counts.

When This Applies #

Sharding earns its complexity once a single-process crawl stops fitting the constraints of a CI job. The symptoms are concrete:

• A serial scan of the full manifest exceeds the runner’s wall-clock limit, or a memory leak accumulated across hundreds of page objects OOM-kills Chromium halfway through.
• You already parallelized with a naive round-robin split, and now reruns of the same commit assign different routes to different workers, so a flaky route lands in a different shard each time and the aggregate count drifts.
• Two workers each scanned an overlapping slice of the manifest, and the final report shows a route’s violations twice, inflating the gate total above the committed threshold.
• One shard finished in 40 seconds and another took 11 minutes because the split was by route count, not by how long each route actually takes to settle.

If instead a single route never finishes scanning, that is a traversal problem for the progressive loader in async crawling for infinite scroll pages, not a sharding problem. Sharding assumes each route terminates; it only decides which worker runs it and how the pieces reassemble.

The Shard Lifecycle #

The mechanism is a fan-out/fan-in: one manifest splits into N shards by a deterministic assignment, N workers scan in parallel and each emits a partial artifact, then one converge step unions the partials and applies the threshold once.

Minimal Reproducible Example #

The classic broken split slices the manifest by list position at runtime. It looks balanced and reproducible, but it is neither the moment the manifest changes or a worker retries.

# shard_naive.py — DO NOT SHIP. Positional slicing is not stable.
import math

def shard(routes, shard_index, shard_count):
    # Slice the list into contiguous blocks by position.
    per = math.ceil(len(routes) / shard_count)
    start = shard_index * per
    return routes[start:start + per]

Two failures hide in those three lines. First, the assignment depends on list order and length: add one route to the top of the manifest and every downstream route shifts to a different shard, so a flaky URL migrates between workers run to run and the aggregate becomes irreproducible. Second, if the manifest is regenerated from a crawl that yields routes in nondeterministic order, contiguous slicing gives each worker a different set on every run — and if two shard definitions ever overlap (a retried worker re-reads a mutated manifest), the same route is scanned by two workers and its violations are counted twice in the merge.

Correct Implementation #

Assign each route to a shard by hashing its normalized path, not its position. A hash is stable under insertion, deletion, and reordering: a given route always lands in the same shard for a given shard count, so reruns are byte-for-byte reproducible and the union at the end can dedupe on the route key with confidence.

# shard.py — deterministic, order-independent shard assignment.
import hashlib

def normalize(route: str) -> str:
    # Canonicalize so the same logical route always hashes identically:
    # drop the trailing slash, lowercase the host+path, strip the query.
    route = route.split("?", 1)[0].rstrip("/").lower()
    return route or "/"

def assign(route: str, shard_count: int) -> int:
    # SHA-1 of the canonical path, taken mod shard_count. Deterministic
    # across machines and Python runs (unlike the salted built-in hash()).
    digest = hashlib.sha1(normalize(route).encode("utf-8")).hexdigest()
    return int(digest, 16) % shard_count

def routes_for_shard(manifest, shard_index, shard_count):
    # Each worker filters the FULL manifest to the routes it owns. No
    # worker ever sees another's routes, so overlap — and double-counting — is
    # structurally impossible regardless of manifest order or retries.
    return [r for r in manifest if assign(r, shard_count) == shard_index]

Pure hash-modulo balances by route count, which is the wrong axis: a heavy dashboard that takes 12 seconds to settle and a static error page that scans in 300 ms count the same. Once you have per-route timing history, switch to a greedy longest-processing-time bin-pack so each shard carries roughly equal duration, not equal cardinality.

# balance.py — greedy LPT packing by historical scan duration.
import json
from pathlib import Path

def load_durations(path="scan_timings.json"):
    # {route: seconds} written by prior runs; default new routes to the
    # p75 so a first-seen route is treated as moderately heavy, not free.
    data = json.loads(Path(path).read_text(encoding="utf-8"))
    vals = sorted(data.values()) or [5.0]
    p75 = vals[int(len(vals) * 0.75)]
    return data, p75

def pack(manifest, shard_count, timings_path="scan_timings.json"):
    durations, default = load_durations(timings_path)
    # Heaviest routes first; assign each to the currently-lightest shard.
    ordered = sorted(manifest, key=lambda r: durations.get(normalize(r), default), reverse=True)
    shards = [[] for _ in range(shard_count)]
    load = [0.0] * shard_count
    for route in ordered:
        i = load.index(min(load))          # lightest shard right now
        shards[i].append(route)
        load[i] += durations.get(normalize(route), default)
    return shards, load

Each worker scans only its own routes and writes a partial artifact keyed by shard index — never a shared file. Isolating a browser per worker is what keeps the shards independent: a crash or memory leak in one worker’s Chromium cannot corrupt another shard’s results.

# worker.py — one worker, one browser, one partial artifact.
import json, sys
from pathlib import Path
from playwright.sync_api import sync_playwright
from scan import run_accessibility_scan   # returns {"violations": [...]}

def run_shard(manifest, shard_index, shard_count, out_dir="a11y-shards"):
    routes = routes_for_shard(manifest, shard_index, shard_count)
    Path(out_dir).mkdir(exist_ok=True)
    partial = {"shard": shard_index, "shard_count": shard_count, "routes": {}}
    with sync_playwright() as p:
        # One browser owned by THIS worker only; isolated from every other shard.
        browser = p.chromium.launch()
        for route in routes:
            ctx = browser.new_context()    # fresh context per route: no state bleed
            page = ctx.new_page()
            try:
                results = run_accessibility_scan(page, route)
                partial["routes"][normalize(route)] = results["violations"]
            finally:
                ctx.close()
            page.close()
        browser.close()
    # Partial artifact name carries the shard index — never a shared path.
    Path(out_dir, f"shard-{shard_index}.json").write_text(
        json.dumps(partial, indent=2), encoding="utf-8")

if __name__ == "__main__":
    manifest = json.loads(Path(sys.argv[1]).read_text(encoding="utf-8"))
    run_shard(manifest, int(sys.argv[2]), int(sys.argv[3]))

The converge step reads every partial, unions them keyed by normalized route, and applies the committed threshold exactly once over the whole surface. Keying on the route means a route that somehow appears in two partials contributes its violations only once — the merge is idempotent, so a re-uploaded shard artifact cannot inflate the total.

# converge.py — fan-in: union partials by route, gate once.
import glob, json, sys
from collections import Counter
from pathlib import Path

HARD = {"critical", "serious"}   # zero tolerance; matches the single-route gate

def converge(shard_glob="a11y-shards/shard-*.json"):
    by_route = {}                # route -> violations; last writer wins, but
    for f in sorted(glob.glob(shard_glob)):          # partials never overlap,
        partial = json.loads(Path(f).read_text(encoding="utf-8"))
        for route, violations in partial["routes"].items():
            by_route[route] = violations             # so this is a plain union
    counts = Counter()
    for violations in by_route.values():
        for v in violations:
            counts[v.get("impact") or "minor"] += 1
    hard_total = sum(counts[k] for k in HARD)
    print(f"routes merged: {len(by_route)}  counts: {dict(counts)}")
    return 1 if hard_total else 0    # threshold applied ONCE, over the union

if __name__ == "__main__":
    sys.exit(converge())

Deduping on the route key is the safety net that a positional split can never have: even if the CI matrix accidentally launches a shard twice, the union collapses the duplicate rather than double-counting it, and the gate total stays honest.

Pipeline Integration Note #

Sharding is the fan-out layer beneath the single-route gate described in the Playwright headless scanning workflows guide: each worker runs that same impact-aware scan, but the pass/fail threshold moves from per-route to the converge step so it applies to the whole estate at once. Downstream, the merged violations feed the error categorization and triage pipelines, which deduplicate selectors across shards and attach component ownership — and because the converge step already keys by route, those pipelines receive one clean union instead of N overlapping partials to reconcile. In GitHub Actions this maps directly onto a matrix job: strategy.matrix.shard: [0,1,2,3] launches four workers, each invoked as python worker.py manifest.json ${{ matrix.shard }} 4, and a dependent job runs converge.py after all four upload their shard-*.json artifacts.

Gotchas #

• Changing the shard count reshuffles every assignment. Because assignment is hash(route) % N, going from 4 to 6 shards moves most routes to a different worker and invalidates any per-shard duration cache. Treat shard count as a committed constant per manifest generation; if you must scale it, regenerate the timing history rather than trusting stale per-shard baselines. Consistent-hashing avoids the reshuffle but is rarely worth the complexity at CI scale.
• Authenticated shards each need their own storage state. When routes span multiple tenants or roles, a worker must load the storage_state for the routes it owns. Assign routes to shards by tenant and hash so a single worker never needs two conflicting auth contexts, and inject each tenant’s state as a separate encrypted secret rather than logging in inside the job.
• A silently-failed worker passes the gate by omission. If a shard crashes before writing its partial, converge simply unions fewer routes and may report zero breaches for routes never scanned. Assert in the converge step that the number of partials equals the shard count and that the union covers every route in the manifest, and fail closed when a shard’s artifact is missing.

Frequently Asked Questions #

Related #

• Batch Validation Architecture
• Playwright Headless Scanning Workflows
• Error Categorization & Triage Pipelines
• Async Crawling for Infinite Scroll Pages