Qualityforge

QualityForge

Evidence-first browser quality automation for modern web applications.

QualityForge is an open-source foundation on top of Playwright that turns a failing run into evidence a machine can read: normalized defect artifacts with trace, screenshot and video, a quality gate with explicit thresholds, capture of console, network and page-error signals — and a read-only MCP server that hands all of it to an AI assistant without granting it the ability to act.

Status — stable, published. qualityforge@0.2.0 is on npm since

2026-10-21 (0.1.0 on 2026-10-15 was the first stable release), and the

defect contract is defect.v1 at 1.6.0. The 0.x line still allows

breaking changes in a minor, so pin the version you install.

Contents

• Why this exists
• What you get
• Quick start
• The demo app
• The evidence pipeline
• Defect artifacts
• The MCP server
• Configuration
• Commands
• Rules enforced in CI
• Documentation
• Contributing

Why this exists

Most teams end up with a pile of one-off UI tests per project: duplicated selectors, no shared failure evidence, no cross-project view of quality. QualityForge is a reusable core instead — one configuration, one evidence format, one set of quality gates, pointed at any web application.

The differentiator is not “AI writes tests”. Playwright already ships agents for that. It is the evidence layer between a failing run and any agent: defect artifacts normalized into one schema, readable without granting the reader the ability to act.

What you get

• An evidence pipeline. Trace on first retry, screenshot and video on failure, in Chromium, Firefox and WebKit. A run that fails still publishes its artifacts.
• One normalized defect format. defect.v1, contract 1.4.0: a versioned, validated artifact per failure that extends Playwright's error-context.md rather than duplicating it.
• A quality gate. The collector splits by spec × project, refuses duplicate ids, treats an aborted suite as one outage instead of a defect per test, and exits non-zero when thresholds are violated.
• Signal capture. Console errors, uncaught page errors, failed requests and HTTP errors above 400 — redacted at the source, bounded per category, with a signals.dropped count so a truncated capture never looks complete.
• A read-only MCP server. Protocol 2026-07-28, thirteen tools, resources and a triage_failure prompt, verified against OpenCode v2.0.16 over real stdio.
• Run history. One compact file per run, so “is this a regression or has it always been this way” has an answer.
• CI that checks itself. Six legs — lint and typecheck, unit tests on Node 22 and 24, the suite on three browsers — plus a job that installs the real OpenCode client and asserts the frames that cross the wire.

Quick start

Development requires Node.js 22 or newer — Node 20 reached end-of-life in March 2026, and "engines": ">=22.0.0" says so instead of promising install politeness. CI runs Node 22 and 24, and the floor is the lower of the two. .nvmrc pins 22 for contributors: tsc dies with a segmentation fault under one Node 24.21.0 build on this machine, reproducibly. If you use a version manager, run nvm use before anything else.

Platforms. Linux is what CI runs and therefore what is verified. The core is written to be platform-neutral — path.sep and path.relative throughout, no POSIX-only calls — so Windows and macOS should work, and _have not been tested_. Treat them as unverified rather than supported.

git clone https://github.com/ninelegsdog/qualityforge.git

cd qualityforge npm ci npx playwright install --with-deps chromium npm test

npm test starts the bundled demo app automatically, so a fresh clone runs green with no configuration and no third-party network dependency. Open the HTML report with npm run report. For a step-by-step path — seeing a real failure with evidence, collecting defect artifacts, connecting an agent, pointing the suite at your own application — follow docs/quick-start.md, or docs/quick-start.ru.md.

The demo app

fixtures/ is a small multi-page app that exists so the suite has something real to test. It is not a mock: it renders, validates and fetches.

Every control is labelled, every message has a role, and every element the tests need has a stable handle — the suite asserts on what a user perceives, not on markup structure.

The evidence pipeline

The project's core promise is that a failure leaves enough evidence to diagnose it, and that claim is tested, not asserted:

QUALITYFORGE_EVIDENCE_CHECK=1 npx playwright test tests/smoke/evidence-pipeline.spec.ts

This runs two deliberately failing tests and exits non-zero. Afterwards test-results/ contains, per failure:

error-context.md is the most interesting one: it is written as an instruction to an assistant — “explain why, be concise, respect best practices” — followed by the test name, file location, error and the diff of expectations. It is the raw material an agent needs, and it costs nothing extra.

On CI, prefer the trace viewer over screenshots and videos when debugging: it gives the DOM snapshot at each action plus the full network log.

Defect artifacts

A failing test is only useful if the failure can be read later without re-running anything. QualityForge turns each failure into one normalized, versioned artifact:

npm test                    # writes artifacts/json/playwright-results.json

npm run defects:collect # writes artifacts/defects/ /

{

"id": "form-validation-smoke-shows-an-error-when-email-is-empty", "status": "failed", "test": { "file": "tests/smoke/form-validation.smoke.spec.ts", "line": 15 }, "failure": { "message": "Error: expect(locator).toHaveText(expected) failed\n\nExpected: \"Email is required\"", "errorContextRef": "test-results/.../error-context.md", }, "evidence": { "screenshot": "test-results/.../test-failed-1.png" }, "context": { "commit": "43c761f", "baseUrl": "http://127.0.0.1:4311" }, "flakiness": { "verdict": "unknown", "attempts": 1 }, }

The artifact deliberately extends Playwright's error-context.md rather than duplicating it: it records a path to that file and adds stable identity, run correlation, VCS context, evidence pointers and a flakiness verdict. The signals block — console errors, uncaught page errors, failed requests, HTTP errors above 400 — is collected live by the bundled fixture, because “element not found” is a symptom while “the page threw a TypeError” is the cause. See docs/defect-schema.md for the full contract.

Accessibility is a second, separate measurement rather than a field on that artifact: npm run a11y:audit runs axe-core against a page and writes a11y-audit.v1 — capped at ten nodes per rule with the discarded count stated, html reduced to structure because it carries page content, and target kept because without it the nodes are indistinguishable. See docs/a11y-schema.md.

npm run defects:collect doubles as a CI quality gate: it exits 0 when the gate passes, 1 when thresholds are violated, and 2 when collection could not run at all.

Installing the package

npm i qualityforge@0.2.0

latest points at this release. Pin the version you want rather than trusting a tag name anyway: npm forbids removing latest, so a tag can only ever move forward, and a bare npm i qualityforge takes whatever it points at rather than what you tested against.

A consuming project imports the fixture from the build — Playwright refuses to transpile TypeScript under node_modules, so the fixture is shipped as a subpath in exports:

import { expect, test } from "qualityforge/fixtures/quality-context.js";

That one import change is the whole integration: console, page-error and network capture become automatic. Everything captured is redacted at the source — URLs keep scheme, host and path with query strings and credentials removed, console text has tokens and JWTs replaced, sensitive header values are masked while their names are kept — and capture is bounded at 40 entries per category.

@playwright/test is a peer dependency: a consuming project brings its own, and npm 7+ installs it automatically when it is missing, because the fixture cannot run without it. The declared range is ^1.63.0; the number CI verifies is 1.63.0, pinned exactly in devDependencies, and npm run package:check fails if the package stops declaring the peer or npm stops installing it for the consumer.

The bin entries publish the two commands a consuming project runs — always through npx --no-install, which provably cannot reach the registry for somebody else's code:

npx --no-install qualityforge [flags]                          # the collector

npx --no-install qualityforge-mcp [--root ] [--history ]

The MCP server

npm run mcp starts a read-only MCP server over stdio, speaking protocol 2026-07-28. It hands an agent the evidence layer: what failed, where, and what the page was doing while it failed. Verified against OpenCode v2.0.16 on Linux by connecting with the block below; Kilo and MiMo are out of scope by decision — see the end of this section.

{

"$schema": "https://opencode.ai/config.json", "mcp": { "qualityforge": { "enabled": true, "type": "local", "command": ["node", "/absolute/path/to/qualityforge/dist/mcp/index.js"], }, }, }

A project that installed the package needs no path at all: ["npx", "--no-install", "qualityforge-mcp"] runs the same server out of `n