llmshift

LLMShift

Switch AI models without breaking your product.

GPT-4o, GPT-4o mini and GPT-4.1 retire on 14 January 2027, GPT-5 and o3 on 11 December, Claude Sonnet 4.5 on 30 November. Does your code call one of them? Find out in seconds, with nothing to install:

uvx llmshift scan        # or: pipx run llmshift scan

A real run: scan on a sample repository, then compare replaying llmshift's demo traffic on Claude Haiku 5.5.

Leaving the OpenAI Evals platform? It goes read-only on 31 October and its API shuts down on 30 November 2026. llmshift import openai-evals --api --all pulls every eval, its latest run and the graded outputs into tests you keep, and runs them on any model (how, or the step-by-step guide).

Then test the replacement on your real traffic before the shutdown date: llmshift snapshot turns your logs into tests and llmshift compare --to shows what changes. llmshift demo walks through it with simulated models, no API key needed.

Providers now retire models roughly every six months. When a model's behaviour changes, outputs can look fine and still be wrong, and nothing raises an error. LLMShift is built around the moment of switching models, from finding the calls to proving the new model works:

AI coding assistants can use all of this as tools: llmshift mcp is an MCP server for Claude Code, Cursor and VS Code.

LLMShift is a single pure-Python package with no compiled code: its only dependency is tomli on Python 3.9 and 3.10 (to read pyproject.toml), and a copy of PyYAML's pure-Python YAML parser is included. It talks to OpenAI, Azure OpenAI, Anthropic, Gemini and any OpenAI-compatible endpoint over plain HTTPS, so there are no SDK version conflicts. You can also run it as a GitHub Action.

pip install llmshift

llmshift demo # 1-minute offline walkthrough with simulated models (no API key)

It runs on Python 3.9 to 3.15 on Linux, macOS and Windows, and installs without a compiler on any of them.

Quick start with your own app

# 1. Which models does the code use, and when do they retire?

llmshift scan

2. Turn real traffic into tests (OpenAI/Anthropic/Gemini logs, LangSmith/Langfuse exports, CSV, ...)

llmshift snapshot logs/ --model "gpt-4o*" --since 2026-09-01

3. What changes on the new model? Or: which cheaper model would do?

export OPENAI_API_KEY=... llmshift compare --to openai:gpt-5.6-sol llmshift pick --models openai:gpt-6-luna,anthropic:claude-haiku-5-5 --volume 100k

4. Get prompt changes that close the gaps, then open the upgrade pull request

llmshift fix --to openai:gpt-5.6-sol llmshift upgrade --from gpt-4o-2024-05-13 --apply --compare --pr

5. After switching, watch the live model every night

llmshift watch --model openai:gpt-5.6-sol --alert slack:$SLACK_WEBHOOK_URL

llmshift init --github writes a llmshift.yaml and two workflows: PR checks and a nightly watch. More workflows are in examples/workflows/.

scan: find every model call and its retirement date

$ llmshift scan

LLMShift scan 5 files, 6 model references, 4 distinct models (lifecycle data: 2026-10-21)

✗ RETIRED claude-3-5-haiku-20241022 anthropic retired on 2026-02-19; replacement: claude-haiku-4-5-20251001 app/summaries.py:10 model="claude-3-5-haiku-20241022", ! 2 DAYS LEFT gpt-4o-2024-05-13 openai retires 2026-10-23 (2 days left); replacement: gpt-5.6-sol app/triage.py:7 TRIAGE_MODEL = "gpt-4o-2024-05-13" app/triage.py:17 model=TRIAGE_MODEL, (via TRIAGE_MODEL, line 7) app/replies.py:15 model=os.getenv("REPLY_MODEL", "gpt-4o-2024-05-13"),

• Python is analysed with the AST. It finds model=-style arguments and models passed to known constructors (GenerativeModel, init_chat_model, pydantic-ai Agent, LiteLLM, ...). It also handles {"model": ...} dicts, constants used elsewhere (MODEL = "..." then model=MODEL) and os.getenv("X", "default") fallbacks. Docstrings and comments are ignored.
• Other code (JS/TS, Go, Java, Kotlin, C#, Ruby, PHP, Rust, Swift, ...): string literals that are model IDs, and values of model-like keys.
• Config: YAML, JSON, TOML, .env, INI, Terraform and Dockerfiles. Notebooks: code cells, reported with cell numbers.
• ID formats are normalised. Bedrock (us.anthropic.claude-...-v1:0, full ARNs), Vertex (claude-...@20241022, projects/.../models/...), OpenRouter/LiteLLM prefixes (openai/gpt-4o), dotted spellings (claude-3.5-sonnet) and fine-tunes (ft:gpt-4o-mini-..., which follow their base model's schedule) all map to their base model.
• In a git repository, files git ignores are skipped unless you name them, and so is LLMShift's own .llmshift/ folder. --exclude takes globs: a bare name (legacy) matches at any depth, and paths may be relative to the scanned directory or the current one.
• Lifecycle data covers 617 models from OpenAI, Anthropic, Google (Gemini API and Vertex AI), Mistral, Cohere, xAI and DeepSeek. It was built from the providers' official deprecation pages on 2026-10-21. Status is computed from dates at run time, so a model turns "retired" on its shutdown day. llmshift update downloads newer data (below); add or override entries with --registry extra.json; llmshift models gpt-4o searches the data.
• In CI, use --fail-on retired|retiring|deprecated|unknown|never (default retired). Output formats: --format table|json|markdown|sarif. Inside GitHub Actions, findings appear as annotations on the diff and in the job summary.

snapshot: real usage logs become a test set

llmshift snapshot logs/ -o .llmshift/tests.jsonl

snapshot reads JSONL, JSON and CSV files (optionally .gz) and detects the format of each record:

• OpenAI Chat Completions, Responses API and Batch files (each batch's input and output files are joined on custom_id)
• Anthropic Messages and Gemini generateContent request/response pairs, including logged streams (server-sent events are reassembled)
• Helicone-style {"request": {"body": ...}} records and flat logs
• chat datasets ({"messages": [...]}, ShareGPT) and Langfuse / LangSmith-style exports
• any rows with prompt / input / question and output / response / answer columns

Embedding, moderation, image and audio calls are skipped, and every skipped record is counted with its reason. --append adds only cases that are not already in the test file. Logs of any size are read as a stream: beyond --max-records calls (default 100,000) a fair sample is kept for every prompt template.

Then it:

1. Drops failed calls and truncated outputs, and applies --model, --since and --until filters. Each case keeps the request's settings, extended thinking included: OpenAI's reasoning effort, Claude's thinking and effort, and Gemini's thinkingConfig.
2. Redacts secrets. With --redact-pii it also replaces e-mails, phone numbers and card numbers, using consistent placeholders so echoed values still match.
3. Removes duplicates and groups requests by prompt template.
4. Picks a diverse sample per template (--max-cases, --per-group).
5. Infers checks from what the old model actually did:

Logs without outputs? Pass --baseline-model openai:gpt-4o-2024-08-06 to generate them.

No logs yet? Record them from your app with one line:

from llmshift.record import wrap

client = wrap(OpenAI(), path="logs/llm.jsonl") # works with OpenAI, AsyncOpenAI, Anthropic, AsyncAnthropic, streaming

Streamed responses, including the SDKs' stream helpers (with client.messages.stream(...), client.chat.completions.stream(...)), are logged whole once the stream ends, with tool calls and token usage. Calls through `w