ego-webqa

ego-webqa

Deterministic coarse delivery QA for AI-generated web pages, H5 demos, and PWAs.

Point it at a URL. It opens the page in a real Chromium browser by default, checks a zero-config narrow-mobile matrix, safely probes the primary call to action, and leaves screenshots plus machine-readable evidence behind. Firefox and Playwright WebKit can be selected one engine at a time. The default eight variants combine:

• 360px and 390px widths;
• 1.0x and 1.5x text scaling;
• 0px and 34px simulated bottom safe area.

See the result in 30 seconds

Prefer to inspect a completed run first? Open the live before-and-after evidence board.

python -m pip install ego-webqa

ego-webqa setup ego-webqa https://your-preview.example --out .qa/first

setup first checks the selected browser. If it already launches, setup exits without downloading anything; otherwise it installs the pinned Playwright headless shell with the active ego-webqa Python interpreter and verifies a real headless launch. The first Chromium setup avoids the full headed browser build because audits only run headless. Ordinary doctor and audit commands never download a browser implicitly. The audit command exits 0 for pass/warnings, 1 for blocking delivery defects, and 2 for an audit or browser failure. Each run leaves screenshots and structured evidence:

.qa/first/

├── agent_summary.json # one aggregated, repair-ready item per root cause ├── feedback_bundle.json # privacy-safe summary for external issue reports ├── report.json ├── junit.xml ├── report.sarif └── screenshots/

A real dogfood finding looks like this:

{

"dimension_id": "text_scale_layout_break", "selector": "body > div:nth-of-type(2) > div:nth-of-type(2) > a", "text_snippet": "我也要创建 →", "problem": "page is 9px wider than the viewport at 1.5x text scale", "trigger": "mobile-360-150-safe0 (360x800); mobile-360-150-safe34 (360x800)", "severity": "block" }

The report explains what broke, where it happened, which viewport combinations were affected, and gives the Agent a command that reproduces the failure.

What real pages exposed

The v0.8.2 dogfood pass audited all 20 Ego-owned production or canonical generated pages with zero runtime errors. After separating network-sensitive performance observations from deterministic delivery findings, ten pages were clean and ten had actionable structural findings. Stable failures included six text-enlargement layout breaks, four bottom safe-area penetrations, five clipped-text root causes, four undersized-target groups, and one generated page loading a missing service worker. There are no known stable blocking false positives in this corpus. This is a small, disclosed corpus rather than a market-wide accuracy claim; the full evidence, raw mixed-profile counts, and limitations are in docs/real-world-dogfood-2026-10-31-v0.8.2.md.

Layout failures now identify the deepest unclipped elements that own the overflow, so an Agent receives a concrete repair selector instead of html.

See how one generated page went from three root causes to zero blocking defects in the vocabulary repair case study. If you want to test the same loop on one of your pages, follow the 15-minute first-trial protocol. The field feedback protocol defines how real cases are classified and converted into clean controls or benchmark fixtures.

Why

Generated pages often pass unit tests while still shipping obvious delivery defects: content wider than the phone, invisible buttons, broken images, console errors, a call-to-action that does nothing, or a page that shifts while the user is trying to tap it.

ego-webqa turns those into deterministic checks. It is not a visual-regression service and does not require writing tests or storing approved screenshots.

Install

Install the published package from PyPI:

python -m pip install ego-webqa

ego-webqa setup

To install the current checkout before a release:

git clone https://github.com/ROY32172310/qa.git

cd qa python -m pip install . ego-webqa setup

For development, use the committed lockfile:

uv lock --check

uv sync --locked --all-extras uv run --locked ego-webqa setup

Without uv, python -m pip install -e '.[dev]' remains supported.

If uv is available and you do not want a persistent Python package install, use the same commands through an isolated, version-pinned uvx environment:

uvx --from ego-webqa==0.8.2 ego-webqa setup

uvx --from ego-webqa==0.8.2 ego-webqa https://your-preview.example

Playwright is pinned by the package, and browser binaries live in Playwright's shared cache, so the temporary uvx environment and later MCP launches reuse the same browser revision.

If doctor reports missing Linux system libraries, ask setup to install them:

ego-webqa setup --with-deps

Setup is intentionally a separate, explicit command: wheel installs remain offline-safe, while browser downloads and operating system changes are visible, retryable user actions. setup is idempotent and reuses a browser that already passes a real launch check. Use setup --dry-run to inspect the exact interpreter-bound command, or --force to reinstall a damaged browser revision.

For a lower-level repair or explicit installation command, use:

ego-webqa install-browser

ego-webqa doctor

Chromium installation is headless-only by default. If another tool on the same machine requires the full headed Chromium build, opt in explicitly:

ego-webqa setup --full-chromium

Use a system Chrome instead of downloading Chromium:

ego-webqa https://example.com --browser-channel chrome

Firefox and WebKit

Install only the additional engine you need, then select it for the whole audit:

ego-webqa setup --browser-engine firefox

ego-webqa https://example.com --browser-engine firefox

ego-webqa setup --browser-engine webkit ego-webqa https://example.com --browser-engine webkit

Chromium remains the default. Each audit runs one engine; this does not silently multiply browser cost or claim cross-engine parity. The webkit option uses Playwright's WebKit build and supplies useful Safari-engine evidence, but it is not a real iPhone, iPad, or macOS Safari run. Use real-device testing when the delivery decision requires it.

Use

ego-webqa https://example.com

ego-webqa --dir ./dist --file index.html ego-webqa https://example.com --preset quick ego-webqa http://127.0.0.1:8000 --viewports 390x844:mobile@1.5~34,1280x800:desktop ego-webqa https://example.com --browser-engine firefox ego-webqa https://example.com --profile performance \ --ready-selector "#app-ready" --wait-for-network-idle ego-webqa https://example.com --smoke-success-selector "#thank-you" ego-webqa https://example.com --smoke-scope all --smoke-require-assertion \ --smoke-success-selector "#thank-you" ego-webqa --dir examples/demo --config examples/webqa-demo-flow.json ego-webqa https://example.com --profile accessibility ego-webqa https://example.com --profile performance ego-webqa https://example.com --profile full

Exit codes:

• 0: pass or warnings only
• 1: at least one blocking error
• 2: invalid invocation or browser/runtime failure

--viewports accepts WIDTHxHEIGHT:NAME[~SAFE_BOTTOM_PX]. Explicit viewports replace the default matrix. Text scale must be between 1.0 and 2.0; the simulated bottom safe area must be between 0 and 100px.

--preset quick selects four pairwise-balanced mobile variants for fast Agent loops: each width covers both text scales and both safe-area states, while the crossed scale/safe-area combinations alternate by width. --preset full explicitly selects the default eight-variant matrix. A preset cannot be combined with --viewports.

Every run writes self-contained evidence. Screenshot references in JSON are relative to the report directory, so agents and CI can move the complete artifact directory without rewriting paths:

webqa-report/

├── report.json ├── agent_summary.json ├── feedback_bundle.json ├── junit.xml ├── report.sarif └── screenshots/

Add --fail-on-warn for CI gates that require a completely clean report.

agent_summary.json is the compact Agent-facing view. Repeated findings are aggregated by registered dimension, so the same root cause across several viewports produces one repair item with affected triggers and a reproducible command. MCP callers receive the same summary as structured content and as the short text result.

feedback_bundle.json is the external-feedback view. By default it includes dimension ids, counts, sanitized viewport geometry, browser engine, and target type, but omits the target URL, local paths, custom viewport names, selectors, page text, messages, raw evidence, screenshots, and configuration contents. Review it before sharing. To deliberately add context for a private issue:

ego-webqa feedback --report webqa-report/report.json

ego-webqa feedback --report webqa-report/report.json \ --include-target --include-selectors --include-messages \ --out webqa-report/feedback-with-context.json

The first command regenerates the default bundle beside the report. The second is an explicit opt-in and may contain page-sensitive data.

Baseline and regression gate

Compare a run with a known-good report without maintaining screenshots. Stable fingerprints combine the registered dimension with its selector or root cause:

ego-webqa https://your-preview.example \

--out .qa/current \ --baseline-report .qa/known-good/report.json \ --gate new

ego-webqa delta \ --baseline .qa/known-good/report.json \ --current .qa/current/report.json

The run writes delta.json with new, persistent, changed, and resolved root causes. --gate new fails only when a blocking root cause is new, becomes blocking, increases its blocking occurrences, or affects an additional viewport. Existing unresolved blockers remain visible but do not make an unchanged build fail again. Add --fail-on-warn to gate new or changed warnings as well.

Both reports must use the same target source, profile, viewport matrix, browser engine, and browser channel. Intentional cross-target or cross-configuration comparisons require --allow-target-change and/or --allow-configuration-change. This is a deterministic finding delta, not a visual-regression baseline.

Each repair item keeps the first evidence object for compatibility and an evidence_by_viewport map for the complete cross-viewport evidence. Network failures additionally carry failure_details with resource type, frame URL, same-origin status, redirect source, HTTP status or browser error, and occurrence count. Query strings and credentials are removed from all rep