vecscope

English · 한국어

vecscope

Embedding space evidence for agents.

Before an agent trusts a vector index, let it see the space it is searching.

Quick start · What it measures · Queries and clusters · Index sweep · Compare two embeddings · Retrieval quality and the fix order · Vector stores · Use with an agent · Use in CI · Input formats · Real embeddings · FAQ

pip install vecscope

---

Agents build RAG pipelines, semantic search and recommendation retrieval every day, but they cannot see the embedding space they are searching.

When retrieval goes wrong it rarely looks like a bug. It looks like:

• "the same three chunks come back for everything"
• "every score is 0.8-something"
• "top-k is full of copies"
• "brute force finds it and HNSW does not"

Each of those failures has a known, deterministic diagnostic in the literature. vecscope turns them into one command and one JSON:

• measurements
• ok / warn / fail verdicts
• an explanation an agent can act on

It runs locally on NumPy, needs no model or network, and reports the same numbers every time.

vecscope diagnoses the vector search implementation: the geometry the model produced, the hygiene of the corpus, the distance metric, the index and the query pipeline.

It does not measure relevance. Whether the right documents come back for a question needs labeled queries, and a clean audit is evidence that search is not sabotaging retrieval, not proof that retrieval is good.

Quick start

1. Install

pip install vecscope

2. Get a file of embeddings

vecscope reads a file with one embedding vector per row: the (n, dim) array a model's encode returns, saved as .npy, .parquet, .jsonl, .json or .csv (all formats).

import numpy as np

embeddings = model.encode(texts) # shape (n, dim), e.g. (3000, 128) np.save("embeddings.npy", embeddings)

No embeddings at hand? Generate three synthetic matrices (healthy, anisotropic, collapsed):

python -c "import urllib.request as u; u.urlretrieve('https://raw.githubusercontent.com/076923/vecscope/main/examples/make_examples.py', 'make_examples.py')"

python make_examples.py

3. Run the audit

vecscope audit embeddings.npy

vecscope x.y.z  n=3000 dim=128 k=10  verdict=FAIL

[ok ] hubness skewness=1.1661 (ok < 2, warn >= 2, fail >= 4) k-occurrence skewness 1.17; top 1% of items take 3.9% of all neighbor slots (max N_k=51, expected 10). [WARN] antihubs antihub_excess=0.0523 (ok < 0.05, warn >= 0.05, fail >= 0.2) 5.2% of items are never anyone's neighbor (random data at k=10 would give 0.0%); the excess is unreachable through k-NN retrieval. try: inspect a sample of never-retrieved items; drop empty inputs, then apply the hubness fix [FAIL] anisotropy mean_random_cosine=0.8732 (ok < 0.5, warn >= 0.5, fail >= 0.85) random pairs have mean cosine 0.873 (isotropic ~0); mean unit vector has norm 0.93; centering would bring the cosine to -0.000. try: subtract the corpus mean vector and re-normalize, applying the same transform to queries; details.mean_random_cosine_after_centering shows the expected result [ok ] effective_rank effective_rank_ratio=0.5407 (ok >= 0.1, warn < 0.1, fail < 0.05) effective rank 69.2 of 128 attainable (128 dims; 54.1%); low values mean most dimensions carry nothing. [ok ] collapse near_duplicate_ratio=0.0000 (ok < 0.02, warn >= 0.02, fail >= 0.1) 0.0% of items have a neighbor with cosine >= 0.99 or an exact copy (exact duplicates 0.0%); no row has a near-duplicate. [ok ] rogue_dimensions top1_dim_variance_ratio=1.3070 (ok < 10, warn >= 10, fail >= 50) dimension 94 alone holds 1.0% of variance, 1.3x the uniform share of 0.78%; top 3 dims hold 3.0%. Rogue dimensions dominate cosine. [ok ] norms norm_cv=0.0000 (ok < 0.1, warn >= 0.1, fail >= 0.3) vector norms vary with CV 0.000 (median 1.000, range 1.000-1.000); 0.00% are zero vectors. Unequal norms bias inner-product search toward long vectors. [ok ] metric_mismatch topk_overlap=1.0000 (ok >= 0.9, warn < 0.9, fail < 0.7) cosine and inner-product top-10 share 100.0% of neighbors; 100.0% of items get identical lists. Low overlap means the index metric changes the results. [ok ] distance_concentration contrast_over_noise=2.2784 (ok >= 1.4, warn < 1.4, fail < 1.1) on 3000 rows a random item is 3.27x farther than the nearest neighbor (random distance 0.127); isotropic noise at 128-d and the same rows would give 1.44x, so the contrast is 2.28x that of noise. On all rows the contrast is 3.27x (median NN distance 0.039). Near 1.0 means distances carry no signal. [ok ] outliers outlier_ratio=0.0030 (ok < 0.01, warn >= 0.01, fail >= 0.05) 0.30% of items have no neighbor within cosine distance 0.050 (median 0.039); see details.sample_ids. Typically junk, empty or foreign-language content. [ok ] connectivity island_ratio=0.0000 (ok < 0.01, warn >= 0.01, fail >= 0.1) 0.00% of items sit in 0 small island(s) of the k-NN graph (fewer than 15 items; 60 components in total). Graph-based ANN (HNSW) attaches islands to the wrong region or never reaches them.

4. Read the result

• The first line gives the overall verdict: the worst verdict among the findings.
• Each finding shows the measured value, the range that would be ok, and a sentence saying what the number means for retrieval. Findings that are not ok add a try: line with the smallest change to test first.
• The exit code is 1 on FAIL, so the same command works as a CI gate.
• Add --json to get the same evidence as JSON for an agent or a script.

What vecscope measures

Every finding in the JSON carries its healthy_range, warn_at and fail_at, plus a details block with the raw numbers. For anisotropy that block includes the cosine the same vectors would have after centering, so the agent can see whether centering alone fixes the problem.

Some findings appear only when you pass their inputs:

• ann_recall: an ANN neighbor file with --ann
• query_gap, query_hubness and ood_queries: query embeddings with --queries
• query_shift: query embeddings, and --baseline also pointing at an earlier audit run with queries

Each finding also carries likely_causes, suggested_action and verify_by. They are hypotheses, not prescriptions:

• apply the smallest change
• re-audit
• compare retrieval on real queries before adopting it

The agent skill spells out that lo