Sitelooper

sitelooper

Write browser tests in plain English. Run them in CI as plain Playwright. Repair them automatically when the app changes.

sitelooper is a CLI with a small LLM agent inside. You give it one instruction at a time, such as "sign in as ops@example.com, create a ticket titled 'k7 Bench' and report its id". It works the live browser and returns one verified result. Every step it gets right is compiled into a procedure that replays with no model, no tokens and no cost, and can be exported as a standalone @playwright/test spec.

*Recorded once against the in-repo repair-desk app (24s, 28s and 42s with the agent driving). The app is reset, the same flow replays in 16s with zero model calls, and then the app's own state is queried.*

How it works

1. Record. A cheap model follows your instructions. As it works, sitelooper records durable locators, where each value came from, and what each step changed on the page.
2. Replay. The recording replays deterministically. If the app has changed under a step, only that step goes back to the model, and a recovery that validates is pinned into the flow, so the flow heals itself over runs.
3. Compile. sitelooper build emits a Playwright spec with effect checks on every step. It runs in CI with no sitelooper daemon and no model.

Results

Ten self-hosted apps, each job done once to record it, then twice more with no model, then as a compiled Playwright spec. The comparator is agent-browser driven by an LLM, which redoes the job on every run. Every run was scored against the app's own database or API, never against the tool's own report. September 2026:

• sitelooper verified every objective in all 30 runs, with GPT-6 Luna writing the instructions and DeepSeek V4.1 Flash driving the browser. Every replay ran with zero model calls. Nine of the ten compiled specs pass; EspoCRM's refuses to compile rather than guess.
• agent-browser on the same model was clean on 3 of 11 runs. It failed outright on Odoo, EspoCRM and Snipe-IT, posted duplicate comments on three apps, and twice reported work as done that the app had not saved.
• On GPT-6.1 Sol, about 20× the per-token price, agent-browser was clean on 9 of 10. It still pays that price on every run, and on Grafana it again reported unsaved settings as verified.

Report-only objectives ask a run to state a value in its final report, which a compiled spec does not write.

• bench/RESULTS.md: the protocol, the models, costs, and what the benchmark does not show
• bench/MATRIX-2026-09.md: every run, with its run id, commit and results branch
• bench/SWEEPS.md: one row per sweep, with the cause of every miss

Quick start

You need Node 20+ and an OpenRouter API key for authoring. The compiled tests need only @playwright/test.

npm install -g sitelooper

npm install -D @playwright/test && npx playwright install chromium export OPENROUTER_API_KEY=sk-or-... sitelooper doctor # shows the provider and models it will use, and why sitelooper init # writes sitelooper.config.json

Record one logical outcome per instruction. Use {{env:NAME}} for credentials, and declare values that should change between runs with var. Here, demo becomes a {{runid}} slot:

sitelooper --session ticket --learn open http://localhost:3000

sitelooper --session ticket var runid=demo sitelooper --session ticket do "Sign in as {{env:TEST_USER}} using {{env:TEST_PASSWORD}} and verify the dashboard opens" sitelooper --session ticket do "Create a ticket titled 'demo Test'; verify it appears and report its id" sitelooper --session ticket assert "the ticket list shows 'demo Test' with status Open" sitelooper --session ticket stop --save-flow ticket sitelooper flow export ticket --out .sitelooper/procedures.json

Build the test and run it:

sitelooper build ticket --var runid=test-{n} --reset-cmd "npm run reset:e2e"

or, if your Playwright fixtures already prepare fresh data:

sitelooper build ticket --var runid=test-{n} --fixture-isolation

with a reset command, build converges by default (2 rounds): it re-records the steps a refusal or

a failing spec names (the model works those steps only). Tune or opt out:

sitelooper build ticket --var runid=test-{n} --reset-cmd "npm run reset:e2e" --converge 3 [--rerecord-runs 2] [--json]

sitelooper build ticket --var runid=test-{n} --reset-cmd "npm run reset:e2e" --no-converge

npx playwright test tests/sitelooper/ticket.spec.ts

ticket.flow.ts is generated: it holds the procedures, a typed input/output API, named steps and effect checks. ticket.spec.ts belongs to you. Add business assertions and fixtures there, and recompiling will never overwrite it.

Why an agent's own script doesn't rerun

Asking the agent to write a script from what it did fails for structural reasons:

• The run's values are baked in. The script quotes today's record id, so tomorrow it opens yesterday's record, or works the wrong one to completion and reports success.
• The selectors name positions, not things. tr:nth-of-type(3), generated class names, a textbox named after the current minute.
• The waits are gone. Each of the agent's observation turns was a pause the app needed.
• Nobody checks the effect. A click can "succeed" on the wrong element, and a save can be refused by a dialog the script never saw.

sitelooper treats the recording as evidence to compile, not text to replay:

• Locator chains, including role and name, label, test id and a structural path. Candidates are retired by measuring which ones resolve on each replay.
• Parameters, not literals. Typed values become slots, and declared values become {{runid}}. A value that one step read and a later step used is threaded between them live. An id the app minted is re-read on each run, and anything with no source is left for recovery instead of being guessed.
• Effect gates. A step that ran but did not produce its recorded effect stops the replay before the next step acts on the wrong state.
• A recovery ladder. Each step first replays with zero model calls. If that fails, a cheap model recovers it; if that is blocked, a stronger model tries; if that fails too, the run halts and reports per-step state.
• Nothing app-specific in the tool. App knowledge goes in an optional per-session briefing, so a fix for one app never becomes a hack for it.

Using it in a project

Readiness means running the emitted code

build compiles the flow and runs the readiness gate. check --ready runs the same gate on an existing artifact. Plain compile works offline, and plain check runs the spec once.

build also converges by default when a reset command is available (--reset-cmd or resetCommand in the project config): it runs the compiled spec once and, if the compile is refused, the spec fails, or it passes only through a blocking locator fallback (below), re-records the step it names (2 rounds; --converge N sets the count), then runs the readiness gate on the spec that passed. Without a reset command it cannot check against a clean app, so it compiles and runs readiness exactly as before and prints one line (and, with --json, a convergence: { skipped: true, why } field) saying convergence was skipped. Re-recording calls the model: with no API key configured, build compiles and checks once and stops with status unavailable if a re-record is needed. --no-converge restores compile plus readiness only.

Convergence also redoes work the recording did that the flow left out. An instruction that changed the app but did not report success is not a flow step (omitted in the flow file, an omitted-work warning at compile); when nothing later in the flow entered its values again, build puts the instruction back as a step where it was recorded (04b-… after 04-…, so no existing step id moves), re-records it with the model, and drops the omission once the step is pinned. Without convergence build stops before readiness and says which instruction is missing: a spec that passes without that work does not show the task was done. Plain compile still only warns.

The gate requires three clean executions with retries disabled. Each run starts from state prepared by the reset command or by declared fixtures, and it uses at least two distinct datasets ({n} becomes 1, 2, 3). No step may be skipped or unresolved, and no locator fallback may decide what a run acts on or uses: a fallback on an action, or on a read whose value a later step or assertion uses, blocks. A fallback on a read that only feeds the final report is listed under warnings in the evidence and does not block. Convergence applies the same rule, so a spec it calls converged is not refused by readiness for the same fallback. The gate uses your Playwright config and project (--config, --project). Evidence goes to .readiness.json, or to --report . Three clean runs are an execution gate, not a statistical guarantee against flakiness.

--negative-spec adds an authored fault-injection test that must fail its outcome assertion, and it reports failureDetection: verified. See project fixtures and negative checks.

Assertions: conditions that must hold on every run

do records what the agent did. assert records something you want to be true, and it makes that check impossible to skip. It takes a sentence and never acts on the page:

sitelooper --session ticket assert "the order total is 370.00"

sitelooper --session ticket assert "no error banner is showing"

The sentence is checked against the page now. Exit 0 means it held, 1 means it did not, and 2 means the check could not run. In a --learn session a passing assert is also recorded as its own step of the saved flow. A model finds the element once, at record time. After that the step replays and compiles with no model.

An assert can record seven conditions: an element is visible, an element is hidden, its text equals a value, its text contains a value, a count of matching elements, a form field's value, and the page url contains a value.

The expected value must be written in the sentence. It can be a literal, the value of a declared var, or a value an earlier step reported. A value the agent only read off the page does not count: "the total is