AWT (AI Watch Tester)

AWT — Eyes and Hands for Your AI Coding Tool

Your AI coding tool is smart. But it can't see or click a web page. AWT gives it a real browser — so it can test, find bugs, and fix them without you lifting a finger.

Works without AI API — write YAML scenarios manually and run aat run. No API key needed. AI coding tools make it easier, but are not required.

---

What is AWT?

AWT is the execution engine for AI-powered E2E testing. Your AI coding tool designs the tests; AWT runs them in a real browser with Playwright.

How it works in Skill Mode (no extra AI API key needed):

You: "Test the login flow on https://mysite.com"

Your AI coding tool: → Writes 5 YAML scenarios (30 steps) → Runs: aat run scenarios/ → Reads failure: "Step 4: Dashboard text not visible" → Reads screenshot + source code → Fixes src/pages/login.js:23 → Re-runs: aat run scenarios/ → All 5 scenarios pass ✓

Key capabilities:

• Execute tests in a real browser with human-like mouse/keyboard interaction
• See — take screenshots, detect elements via OCR and image matching
• Report — step-by-step pass/fail with error details and screenshots
• Self-heal (CLI mode) — aat loop auto-fixes failures with its own AI
• Learn — successful matches stored in SQLite, getting faster over time

Installation

Install the aat command first — every route below drives it:

pip install aat-devqa

python -m playwright install chromium

Claude Code plugin (recommended)

This repository is a plugin bundle that ships the awt skill.

claude plugin marketplace add ksgisang/awt-skill

claude plugin install ai-watch-tester@awt

Verify what got installed:

claude plugin details ai-watch-tester@awt

You should see Skills (1) awt.

Other AI coding tools

The skill is a directory of Markdown, so anything that reads skills can take it:

npx skills add ksgisang/awt-skill --skill awt -g

Or copy it by hand, globally or per project:

git clone https://github.com/ksgisang/awt-skill.git /tmp/awt-skill

cp -r /tmp/awt-skill/skills/awt ~/.claude/skills/awt # or .claude/skills/awt rm -rf /tmp/awt-skill

After installation, the skill auto-triggers on testing-related prompts.

Updating

claude plugin update ai-watch-tester@awt          # plugin

npx skills add ksgisang/awt-skill --skill awt -g -y # skill only

For hand-copied installations: cd ~/.claude/skills/awt && git pull

MCP tools (optional, separate)

The skill drives the aat command directly, which is all it needs. If you would rather call AWT as MCP tools (aat_scan, aat_run_skill_mode, aat_diff, and nine more), the server lives in the main repository and is installed separately:

pip install "mcp[cli]>=1.0.0"

git clone https://github.com/ksgisang/AI-Watch-Tester.git claude mcp add awt -- python3 "$PWD/AI-Watch-Tester/mcp/server.py"

Setup for other MCP clients is in mcp/README.md. This plugin does not bundle that server: the Claude Code plugin directory accepts MCP servers that run on your own machine only from verified organizations, so the bundle ships the skill alone.

Comparison with Other Testing Skills

When to use each:

• webapp-testing — Quick one-off Playwright scripts for simple page checks
• playwright-skill — When you need fine-grained Playwright API control
• AWT Skill Mode — Your AI coding tool designs tests, AWT executes them. No extra AI API key needed.
• AWT CLI Mode — Standalone automated testing with aat generate + aat loop for CI/CD

Supported AI Coding Tools

AWT follows the Agent Skills open standard and works with:

Quick Example

YAML Scenario

id: "SC-001"

name: "User Login" tags: ["auth", "login"] depends_on: ["SC-000"] steps:

• step: 1 action: navigate value: "{{url}}/login" description: "Go to login page"

• step: 2 action: find_and_type target: text: "Email" match_method: ocr value: "test@example.com" humanize: true description: "Enter email"

• step: 3 action: find_and_click target: text: "Login" description: "Click login button"

• step: 4 action: assert assert_type: text_visible expected:
• type: text_visible value: "Welcome back" description: "Verify login success"

Natural Language (in Claude Code)

"Login to mysite.com with test@example.com, then check if the dashboard loads"

AWT converts this to a YAML scenario and executes it automatically.

5-Step DevQA Workflow

Scan → Plan → Review → Execute → Heal

↑ | └─── Loop back on failure ────────┘

1. Scan — Crawl site, analyze DOM, detect features
2. Plan — AI generates YAML test scenarios
3. Review — User reviews/edits before execution
4. Execute — Playwright runs with live screenshots
5. Heal — AI analyzes failure → suggests fix → re-tests

AI Providers

Project Structure

awt-skill/                      # ← this repository is the plugin folder

├── .claude-plugin/ │ ├── plugin.json # Plugin manifest │ └── marketplace.json # Marketplace entry for this one plugin ├── assets/ │ └── icon.png # Plugin icon ├── skills/ │ └── awt/ # ← Skill content (installed by npx skills) │ ├── SKILL.md # Main skill definition │ ├── references/ │ │ ├── scenario-schema.md # Full YAML schema reference │ │ ├── cli-reference.md # CLI command reference │ │ └── config-reference.md # Configuration options │ └── templates/ │ ├── scenario-template.yaml │ └── config-template.yaml ├── README.md ├── LICENSE ├── CONTRIBUTING.md └── package.json

Key CLI Commands

System Dependencies

macOS:

brew install python@3.12 tesseract

pipx install aat-devqa playwright install chromium

Linux (Ubuntu/Debian):

sudo apt install python3.12 python3.12-venv tesseract-ocr

pipx install aat-devqa playwright install chromium

Windows:

winget install Python.Python.3.12

choco install tesseract pip install aat-devqa playwright install chromium

After install, run aat doctor to verify everything works.

OCR language packs. Tesseract installs English only. If you test an app in another language, install that language's pack too — otherwise text assertions that fall back to OCR report not visible on page for text that is plainly there. aat doctor names any missing pack.

brew install tesseract-lang                 # macOS, all languages

sudo apt install tesseract-ocr-kor # Linux, one language

What this plugin runs, sends, and fetches

Everything below is in this repository in readable source. Nothing is minified, compiled, or downloaded at install time.

Runs on your machine: nothing of its own. This bundle contains no executable code. It is Markdown and YAML templates that tell your AI coding tool which aat commands to run, and your tool shows you each command before running it. The work is done by the aat command you installed with pip install aat-devqa, which drives a local Chromium through Playwright and, for OCR, the local tesseract binary. The rest of this section describes what aat does when you let it run, because that is what the skill asks for.

Writes to disk. Under your project: .aat/ (config, run history, screenshots, the SQLite learning database, the append-only audit log) and reports/. Under your home directory: ~/.awt/templates/ / for the element images AWT saves so it can find a target again after a selector breaks.

Sends over the network. There is no telemetry and no analytics. Every destination below is either one you named yourself or the single call AWT makes on its own, which is listed first so it is not buried:

• `api.github.com`, once a day, by AWT itself. The CLI asks for the latest release so it can print a hint when a newer one exists. No request body, no account, no machine id, no version string, nothing about what you are testing; the request is identical for every user. It times out after 3 seconds and every failure is swallowed, so blocking the host breaks nothing. This is the only call AWT initiates.
• The site you are testing. Chromium loads the URL in your scenario. That is the whole point of the tool, and no other site is contacted by a test run.
• An AI provider, only for the commands that need one — aat generate, aat loop, aat devqa, and Vision matching. The request carries the page text or screenshot and your scenario. You choose the provider and supply the key; with Ollama the reque