Universal Test Framework

Universal Test Framework (UTF)

Contract-driven test enforcement and reporting for LLM-generated code — via MCP, VS Code, Copilot CLI, Claude, Cursor, or Python SDK.

UTF is an MCP server and contract enforcement engine. It does not replace your AI coding assistant — it gives every test your AI writes a mandatory quality gate, a structured audit trail, and a comprehensive compliance report.

---

Why UTF? (Not Another Test Generator)

AI tools — GitHub Copilot, Claude, Cursor, Gemini — generate tests fast. The problem: fast ≠ trustworthy.

UTF vs Markdown Instructions / Prompt Files

You may already use markdown files (AGENTS.md, copilot-instructions.md, .cursorrules) to guide your AI. UTF is complementary — not competing:

Use markdown instructions to *shape* how your AI thinks. Use UTF to *verify* and *report on* what it produced.

---

Key Features

• 13 MCP tools — generate, validate, register, analyze, trace, execute, mutate, report, and query tests
• 6 languages — Python, TypeScript, JavaScript, Java, Go, C++
• 7 test types — unit, integration, API, E2E, contract, performance, security
• 8-section contract — every test must pass all 8 sections or it is blocked (minimum score: 0.85)
• 3-segment test IDs — TC-{PRJ}-{MODULE}-{NNN} and classic TC-{MODULE}-{NNN} both accepted
• Five install modes — uvx (zero-clone), local clone, Docker, GitHub MCP Registry, pip SDK
• VS Code @utf — chat participant with slash commands
• CI/CD ready — GitHub Actions, GitLab CI, pre-commit hooks
• Per-project overrides — YAML rules in .utf/rules/ override global defaults

---

Quick Start

Setup Method Comparison

All methods (1, 2, 4) using stdio transport write the registry to `${workspaceFolder}/.utf/utf.db` — isolated per project and persistent across sessions.

---

Method 1 — uvx (Zero-Install, Recommended)

Requires uv. No cloning, no venv.

Add to %APPDATA%\Code\User\mcp.json (Windows) or ~/.config/Code/User/mcp.json (macOS/Linux):

{

"servers": { "utf": { "type": "stdio", "command": "uvx", "args": ["--from", "universal-test-framework", "utf-server", "--transport", "stdio"], "cwd": "${workspaceFolder}", "env": { "UTF_PROJECT_DIR": "${workspaceFolder}" } } } }

Registry: .utf/utf.db inside ${workspaceFolder} — isolated per project, persistent across sessions.

Reload VS Code after editing mcp.json.

---

Method 2 — Local Clone (VS Code + Copilot)

git clone https://github.com/phoenice-labs/Universal-Test-Framework

cd Universal-Test-Framework

Register MCP server, install @utf extension, copy global prompts

.\scripts\install-vscode-mcp.ps1

Optionally scaffold a specific project

.\scripts\install-vscode-mcp.ps1 -InitProject -ProjectDir C:\my-project

Reload VS Code → open Copilot Chat → ask naturally: generate e2e tests for my backend.

The install script writes this entry to mcp.json:

{

"utf": { "type": "stdio", "command": " ", "args": ["-m", "mcp_server.server", "--transport", "stdio"], "cwd": "${workspaceFolder}", "env": { "PYTHONPATH": " " } } }

Registry: ${workspaceFolder}/.utf/utf.db — isolated per project.

`@utf` slash commands are available after the VSIX extension is installed.

---

Method 3 — Docker (Team / Remote)

# Start the UTF server

cd Universal-Test-Framework docker compose -f docker/docker-compose.yml up -d

MCP server available at http://localhost:8765/sse

Connect from VS Code by adding to mcp.json:

{

"servers": { "utf-remote": { "type": "sse", "url": "http://localhost:8765/sse" } } }

Registry: persisted in a named Docker volume (utf_registry → /app/.utf/utf.db).

For per-project isolation with Docker, use a bind-mount in docker-compose.yml:

`yaml

volumes:

- /path/to/your/project/.utf:/app/.utf

`

Because Docker uses HTTP/SSE transport (no ${workspaceFolder} templating), pass project_dir

explicitly in tool calls, or set UTF_PROJECT_DIR in the container environment.

Per-project Docker workflow:

# docker-compose.override.yml

services: utf-server: volumes:

• ./my-project/.utf:/app/.utf environment: UTF_PROJECT_DIR: /app

---

Method 4 — GitHub MCP Registry

Once published, UTF is discoverable via the MCP marketplace. Clients that support server.json install it automatically. The generated mcp.json entry is equivalent to Method 1 (uvx).

Registry isolation: The MCP registry schema does not support a cwd field at the registry level.

UTF resolves project isolation via (in priority order):

1. project_dir argument passed to each tool call

2. UTF_PROJECT_DIR environment variable

3. Path.cwd() fallback (server's working directory)

For correct isolation, ensure the MCP client writes "cwd": "${workspaceFolder}" and

"UTF_PROJECT_DIR": "${workspaceFolder}" in the generated entry (UTF's mcp-gallery.json does this).

---

Method 5 — pip + Python SDK

pip install universal-test-framework

from mcp_server.tools.generate_tests import generate_tests

from pathlib import Path

result = generate_tests( test_type="unit", source_code=open("src/auth.py").read(), project_dir=str(Path.cwd()), # ← pass explicitly for correct registry isolation ) print(result["suite_code"])

Registry: /.utf/utf.db when project_dir is passed; falls back to Path.cwd().

---

MCP Tools Reference

---

The 8-Section Test Contract

Every test generated by UTF must satisfy all 8 sections. Tests that fail any hard section are blocked — returned in blocked_tests, never silently included.

Minimum passing score: 0.85. Tests below this threshold are blocked regardless of individual section presence. "none" in gaps or vague rationale like "to test the function" are rejected.

---

Supported Matrix

---

The UTF 3-Phase E2E Workflow

This is the canonical flow for using UTF with any AI CLI (Copilot, Claude, Cursor) or VS Code. Follow the phases in order — skipping Phase 1 registration means the report has no Per-Test Contract Detail cards.

┌─────────────────────────────────────────────────────────────────────────┐

│ PHASE 1 — CONTRACT GENERATION (LLM writes, UTF validates + registers) │ │ │ │ ① generate_tests (scaffold) │ │ ② LLM writes real test methods — each with its own TC-ID and │ │ 8-section comment block (WHY / REQ / HOW / COV / OUT / GAP / MEAN)│ │ ③ validate_test_contract — score must be ≥ 0.85 per test │ │ ④ register_contracts — parse test file, upsert status=generated rows │ │ ⑤ generate_report — verify Per-Test Contract Detail cards appear │ │ │ │ PHASE 2 — EXECUTION (pytest/jest/maven runs, results captured) │ │ │ │ ⑥ pytest --junit-xml=utf-tests/reports/results.xml │ │ ⑦ import_test_results — upsert executed/failed rows