Altair ToleshStart a project
01Work02About03ContactStart a project
All work

Release quality scanner · 2026 · Runs locally

TargetScanEvidenceGateVerdictBaseline

A deterministic release scanner that turns a website into evidence, then decides whether that release is allowed to ship.

  • Source on GitHub
  • Fixed rules, no model inference
  • Unit, scanner, reliability, and E2E suites
  • Overflow checked 320 to 1440
  • Optional Supabase mirror
View the source
Status
Runs locally
Year
2026
My role
Scanner architecture, Rule and scoring model, Release gate and verdict logic, Cross-scan comparison engine, SSRF and request safety, Test and reliability suites

Overview

You give it a URL. It crawls the site in a real Chromium browser inside a bounded envelope, records typed evidence from every page, turns that evidence into findings with fixed rules, compares them against a chosen baseline scan, and runs a release gate. The result is a verdict — ready to ship, ready with warnings, needs attention, or blocked — and every finding behind it can be opened.

Why this is hard

Release checks tend to be either a checklist somebody fills in by hand or a single number from a tool. A checklist records an opinion and cannot tell you what changed since the last deploy. A number tells you a site averages well, which is not the same as it being safe to ship. Neither one holds the evidence, so neither one can be argued with when it disagrees with the person who wants to deploy.

The problem

A score does not tell you whether a release is safe to ship. A site can average well and still have a page returning 500, a form control with no accessible name, or a regression that arrived since the last deploy. What a release decision needs is which findings block it, which are new since the baseline, and what evidence supports each one.

The system, step by step

  1. A target URL is crawled inside a bounded, same-origin envelope
  2. Each page produces typed evidence: HTTP, DOM, accessibility, console, and network
  3. Fixed rules turn that evidence into findings with stable fingerprints
  4. The baseline scan is compared to classify new, fixed, changed, and regressed findings
  5. A release gate checks score, critical findings, broken pages, and broken links
  6. The verdict is ready to ship, ready with warnings, needs attention, or blocked

Architecture

  1. A bounded same-origin crawl in real Chromium, with each page isolated so one failure cannot end the scan

    • Playwright
    • Chromium
    • axe-core
    • Lighthouse

  2. A fixed registry where every rule carries its own category, severity, and score impact

    • Rule registry
    • Weighted categories

  3. Typed findings with stable fingerprints, deduplicated so one problem is one finding

    • Typed evidence
    • Fingerprints
    • Dedupe

  4. A chosen project baseline and the previous scan, classified into new, fixed, unchanged, changed, and regression

    • Baseline
    • Delta engine
    • Pixel diff

  5. Four configurable checks — minimum score, critical findings, broken pages, broken links — where a blocker fails the release on its own

    • Gate config
    • Ship verdict

  6. A request policy that keeps the scanner from being pointed at anything it should not reach

    • Public-DNS only
    • Method allowlist
    • robots.txt
    • Bounds

  7. Local JSON and PNG evidence written atomically and treated as canonical, with an optional server-only mirror

    • Local records
    • Supabase mirror

  8. A container running a persistent worker, because the browser and its scan state have to outlive a request

    • Docker
    • Railway
Only the layers this project actually uses. Select one to trace it.

Key decisions

  1. Question

    Should a model decide whether a release is ready?

    No. Fixed rules, real browser measurements, and typed evidence only.

    A release decision has to be reproducible and arguable. The same site in the same state has to produce the same verdict every time, and every finding has to point at the evidence that produced it. A generated judgement can do neither.

  2. Question

    Is the score the release decision?

    No. The gate is separate, with its own four checks: minimum score, zero critical findings, zero broken pages, zero broken links.

    A weighted average can stay high while a page returns an error, because one broken page is diluted by everything that still works. The gate checks those conditions directly, so a blocker fails the release on its own rather than being averaged away.

  3. Question

    How does a finding stay the same finding across two scans?

    A stable fingerprint derived from the normalized finding, not from a database ID.

    Records are rewritten on every scan, so an identifier assigned at write time cannot survive. A fingerprint computed from the finding itself is what makes new, fixed, and regressed classifications possible at all.

  4. Question

    Can the scanner run on serverless?

    No. It runs as a persistent worker in a container.

    A scan holds a live browser, cancellation state in memory, and local evidence files. A serverless request can end, move instance, or be terminated mid-scan, which would leave that state stranded. The trade-off is real: there is no one-click serverless deploy, and a serverless architecture would need a durable queue, a dedicated browser worker, and shared object storage first.

  5. Question

    What stops a scanner from becoming an attack tool?

    A layered request policy: public DNS results only, same origin, a read-only method allowlist, and hard bounds on pages, redirects, size, and time.

    Anything that fetches a URL on request is a way to reach systems the caller cannot reach directly. The policy is enforced on the initial URL, the final URL after redirects, and every resource the browser itself requests, because checking only the first one leaves the other two open.

What I built

A scanner and a release decision on top of it. The scanner drives a real browser through a bounded crawl and records what it finds as typed evidence rather than prose. A fixed rule registry turns that evidence into findings, each with a stable fingerprint, so the same problem in two scans is recognisably the same problem. On top of that sit the parts that make it a release tool rather than a report: a comparison against a chosen baseline, a gate with explicit thresholds, a blocker-aware verdict, an ignore lifecycle for findings a team has accepted, and a receipt that records the decision.

What is not finished

Screenshots and visual comparison run on the primary page only, not on every crawled page. Two areas of the request policy are tested but not fully closed: DNS re-resolution between the check and the request, and byte enforcement on chunked responses, both of which need network isolation to test properly. The Supabase migration is additive and reviewed but has never been applied against a live database. And the deployment boundary is a real constraint, not a preference — the current worker cannot run on a serverless platform without a queue, a dedicated browser worker, and shared storage.

Outcome

Result

The scanner runs end to end: it crawls, records evidence, applies rules, compares against a baseline, evaluates the gate, and writes a verdict with a downloadable release receipt. A fixture test proves the comparison engine by scanning a site, mutating it, and scanning again — the run asserts findings classified as fixed, new, and regressed, a score movement, a measured visual change, and a failing gate.

What I learned

Deciding what counts as evidence was harder than collecting it. The first version could report a great deal and still not answer whether to ship, because nothing separated a finding that blocks a release from a finding that is merely worth knowing. The gate exists because a score alone kept saying yes when the answer was no.

Stack

  • Next.js
  • TypeScript
  • Playwright
  • Chromium
  • axe-core
  • Lighthouse
  • Zod
  • Supabase
  • Docker