CompletionKit

Your prompts need tests too.

Run every prompt against real data. Score each output with an LLM judge against criteria you define. Change anything: the prompt, the model, the temperature, the dataset. Re-run and see exactly what got better and what broke. When the scores tell you something's off, CompletionKit suggests an improved prompt based on the judge's actual feedback on your runs. You inspect the diff, apply it as a new version, and verify the improvement.

It's the difference between "this prompt seems to work" and "this prompt scores 4.3 out of 5 across 200 inputs, up from 3.8 last version."

[Start on completionkit.com →](https://completionkit.com) | [RubyGems](https://rubygems.org/gems/completion-kit)

Just want to use it? CompletionKit Cloud is the same engine, fully hosted — zero install, no Rails ops, plans at completionkit.com/pricing.

Three ways to run it

Same engine, same UI, same REST API and MCP server — pick the deployment that fits. The first two are stack-agnostic: you run CompletionKit as a product and talk to it over HTTP and MCP, whatever language your own app is written in. The third is for teams already building on Rails.

1. Hosted — completionkit.com (recommended)

The fastest path. Sign up and you're running on the same engine you'd self-host, without touching a Rails app. No db:migrate, no Puma, no Solid Queue, no provider key management — multi-tenant workspaces, your team logs in, you go. Plans at completionkit.com/pricing.

2. Self-hosted — the bundled standalone app

Run it on your own infra as a self-contained product. There's nothing to integrate and no Ruby to write — once it's up, you drive everything through the web UI, the REST API, and the MCP server, from whatever stack your own app is built in. It needs Postgres and a host that can run the app (Fly, Render, Heroku, Docker, …).

git clone https://github.com/homemade-software-inc/completion-kit.git

cd completion-kit/standalone bundle install bin/rails completion_kit:install:migrations bin/rails db:migrate

Run both a web server and a Solid Queue worker. In two terminals:

bin/rails server

bin/jobs

Or with foreman in one terminal: foreman start -f Procfile.dev.

Visit http://localhost:3000. Add a provider credential (Settings), create a prompt, upload a CSV dataset, and run it. See Deploying self-hosted for the production-env setup.

3. Rails engine — mount into your existing Rails app

gem "completion-kit"

bin/rails generate completion_kit:install

bin/rails db:migrate

The engine mounts at /completion_kit. Generate / judge flows enqueue Active Job jobs (CompletionKit::GenerateRowJob, CompletionKit::JudgeReviewJob, CompletionKit::RunCompletionCheckJob), so your host app needs an Active Job adapter that actually processes them — Solid Queue, Sidekiq, GoodJob, etc. The :async adapter is not suitable for production: it runs jobs in the web Puma's thread pool with no durability and no retry, and a long LLM call will block request handling.

Host-app layout integration. If your host app overrides the engine layout (e.g. layout "application" on engine controllers, or rendering engine views inside your own shell), include both the engine's stylesheet and JavaScript in that layout:

<%= stylesheet_link_tag "completion_kit/application", media: "all" %>

<%= javascript_include_tag "completion_kit/application", defer: true, nonce: true %>

Without the JavaScript include, in-page behaviours silently fail: clickable table rows, dialogs, copy buttons, the API token reveal, the run form, live run status, live tag-breadcrumb updates, relative-time ticking, CSV row hover-expand, model-refresh progress, focus-first-error, and local-time formatting.

Content Security Policy. The engine's pages have no inline event handlers or inline scripts, and the engine layout puts the request nonce on its script tags, so the pages work under a nonce-based script-src. The views still use inline style attributes, so leave style-src out of the nonce directives and allow 'unsafe-inline' there for now.

Use a nonce that lasts the whole session, not a fresh random one per request. Turbo Drive re-creates the scripts in each new page body with the nonce from the csp-nonce meta tag, but the browser keeps enforcing the policy header of the first page it loaded, so a per-request nonce blocks nonce scripts after the first Drive visit.

Keep a random nonce in the session, and skip it for API controllers so cookie-less API and MCP calls get no session cookie, as the example below does. The Rails default, ->(request) { request.session.id.to_s }, also lasts the session, but it is empty on a brand-new visitor's first response, so a policy that admits scripts only by nonce blocks that first page's scripts until the session exists.

Rails.application.configure do

config.content_security_policy do |policy| policy.default_src :self policy.script_src :self policy.style_src :self, :unsafe_inline end

config.content_security_policy_nonce_generator = lambda do |request| next if request.controller_instance.is_a?(ActionController::API)

request.session[:csp_nonce] ||= SecureRandom.base64(16) end config.content_security_policy_nonce_directives = %w(script-src) end

Providers

CompletionKit discovers available models from each provider's API automatically.

Set these as environment variables or configure them in the generated initializer. You can also add provider credentials through the web UI under Settings.

The Ollama key from OLLAMA_API_KEY or config.ollama_api_key goes only to the Ollama endpoint from OLLAMA_API_ENDPOINT or config.ollama_api_endpoint, or to http://localhost:11434/v1 when neither is set and config.allow_loopback_endpoints is on, which is the default. A provider credential with no key of its own gets that key only while it points at that same endpoint. Point it anywhere else and CompletionKit sends no key, so store the key on the credential itself. A key stored on a credential always goes to that credential's endpoint. Two endpoints match when their scheme, host, port and path match. The case of the scheme and host, a default port written out, a trailing slash and a trailing /v1 make no difference. A blank stored key counts as no key.

Encryption

Provider API keys are encrypted at rest using Active Record encryption. You need three encryption keys configured before the app will boot in production.

Generate them:

bin/rails db:encryption:init

Then set them as environment variables:

COMPLETION_KIT_ENCRYPTION_PRIMARY_KEY=<generated value>

COMPLETION_KIT_ENCRYPTION_DETERMINISTIC_KEY= COMPLETION_KIT_ENCRYPTION_KEY_DERIVATION_SALT=

Or add them to config/credentials.yml.enc under active_record_encryption. In development, the standalone app uses built-in fallback values so you can skip this step locally.

Authentication

CompletionKit requires authentication in any deployed environment. In development and test, routes are open by default (with a log warning); every other environment returns 403 until auth is configured.

Basic Auth (recommended for simple setups)

CompletionKit.configure do |c|

c.username = "admin" c.password = ENV["COMPLETION_KIT_PASSWORD"] end

Custom Auth (Devise, etc.)

CompletionKit.configure do |c|

c.auth_strategy = ->(controller) { controller.authenticate_user! } end

Only one mode can be active.

Reviewer name

When someone records a verdict on a judge's score in the web UI, CompletionKit saves it under a reviewer name. Each reviewer keeps one verdict per response and metric, so give every person their own name. Otherwise their verdicts overwrite each other. Set current_reviewer to a callable that receives the controller and returns the signed-in person's name:

CompletionKit.configure do |c|

c.auth_strategy = ->(controller) { controller.authenticate_user! } c.current_reviewer = ->(controller) { controller.current_user&.email } end

The callable must return a String or nil. Any other value, such as a user record, raises CompletionKit::ConfigurationError. If current_reviewer is unset or returns a blank value, verdicts are saved under username, or "operator" when no username is set. The web UI does not read the X-Remote-User header on its own. If a proxy in front of your app sets that header, pass it through:

CompletionKit.configure do |c|

c.current_reviewer = ->(controller) { controller.request.headers["X-Remote-User"] } end

Only use this header when a trusted proxy always overwrites it. Otherwise any client can send it and claim any reviewer name.

The REST API and the MCP server take the reviewer name from their own created_by parameter.

Rate limiting

The REST API, the MCP endpoint, and the web UI are rate limited per IP, per minute. The defaults are generous; tune them in the initializer:

CompletionKit.configure do |c|

c.api_rate_limit = 120 # REST API + MCP, requests per minute (default 120) c.web_rate_limit = 300 # web UI, requests per minute (default 300) end

Limiting uses Rails.cache. A shared cache store (Solid Cache, Redis) throttles accurately across multiple app instances; a per-process store still throttles each instance independently.

Input limits

Uploads, prompt templates, and regex check patterns have limits, so one large or hostile input cannot tie up a worker. Tune them in the initializer:

CompletionKit.configure do |c|

c.max_upload_bytes = 25 * 1024 * 1024 # largest dataset CSV or import file (default 25 MB) c.max_template_bytes = 1024 * 1024 # largest prompt template (default 1 MB) c.regex_timeout = 1.0 # seconds a regex check may spend matching one row (default 1.0) end

A prompt whose template is over max_template_bytes fails validation when its template is saved. Prompts saved before a