AISIX AI Gateway

AISIX OSS

The open-source, Rust-native AI gateway for LLMs and AI agents

One OpenAI-compatible API in front of every model. Route, govern, secure, cache, and observe all your LLM and AI-agent traffic from a single control point — shipped as one static binary with low per-request overhead. Run it in your infrastructure for free, forever.

*Built by the original creators of Apache APISIX.*

Start free · Documentation · Quickstart · AISIX Enterprise and AISIX Cloud · Roadmap

---

AISIX OSS is the open-source product built around the Rust-native AISIX gateway. The gateway puts a single, OpenAI-compatible API in front of every LLM provider — OpenAI, Anthropic, Google Gemini, AWS Bedrock, Azure OpenAI, DeepSeek, and any OpenAI-compatible endpoint. It gives platform teams one place to route, govern, secure, and observe LLM traffic, with first-class SSE streaming and low gateway overhead.

It runs as a single static binary — low cold-start, lock-free config reads, and hot configuration reloads with no restarts: declare resources in one resources.yaml and reload on SIGHUP, or point the gateway at etcd for a multi-replica cluster. Run the gateway in your infrastructure, or connect it to AISIX Enterprise or [AISIX Cloud](https://api7.ai/ai-gateway?utm_source=github&utm_medium=readme&utm_campaign=cloud) for centralized management with team governance, budgets, audit, and a dashboard.

AISIX OSS (this repo) is the open-source product. It runs without a control plane

using declarative configuration or etcd. The same gateway binary serves as the data

plane in the commercial products: customers host the control plane with **AISIX

Enterprise, while API7 hosts it with AISIX Cloud**. In all three products, the

gateway runs in your environment and calls providers directly; live AI traffic does not

pass through either control plane. The proxy API is identical throughout.

[Talk to us about AISIX Enterprise or AISIX Cloud →](https://api7.ai/contact?utm_source=github&utm_medium=readme&utm_campaign=cloud)

⚡ Quickstart

One container. No control plane, no database, no configuration store — the gateway reads every dynamic resource from one declarative resources.yaml.

# config.yaml

resources_file: /etc/aisix/resources.yaml proxy: addr: "0.0.0.0:3000" admin: enabled: false # a declarative gateway needs no admin listener observability: metrics: prometheus: enabled: true addr: "0.0.0.0:9090"

# resources.yaml

_format_version: "1"

provider_keys:

• display_name: openai-main provider: openai api_key: ${OPENAI_API_KEY} # interpolated from the environment

models:

• display_name: my-model provider: openai model_name: gpt-4o-mini provider_key: openai-main

api_keys:

• display_name: local-dev key_env: CALLER_API_KEY # hashed at load; the plaintext is never stored allowed_models: ["my-model"]

export OPENAI_API_KEY="YOUR_PROVIDER_KEY"

export CALLER_API_KEY="YOUR_CALLER_KEY"

docker run -d --name aisix \ -v "$(pwd)/config.yaml:/etc/aisix/config.yaml:ro" \ -v "$(pwd)/resources.yaml:/etc/aisix/resources.yaml:ro" \ -e OPENAI_API_KEY -e CALLER_API_KEY \ -p 3000:3000 -p 127.0.0.1:9090:9090 \ ghcr.io/api7/aisix:latest # proxy → :3000, metrics + status → :9090

^ the metrics/status listener is unauthenticated;

keep it on loopback or a private network

Then call the gateway exactly like OpenAI:

curl http://localhost:3000/v1/chat/completions \

-H "Authorization: Bearer $CALLER_API_KEY" \ -H "Content-Type: application/json" \ -d '{"model":"my-model","messages":[{"role":"user","content":"hello"}]}'

Edit resources.yaml and send SIGHUP (docker kill -s HUP aisix) to apply changes with no restart — an invalid file is rejected whole and the last good configuration keeps serving. Check a file before booting with aisix validate --resources resources.yaml.

Full walkthrough: the Gateway Quickstart · every field: the resources file reference. For a multi-replica cluster, point the gateway at etcd instead — resources_file and etcd are mutually exclusive.

✨ Why AISIX

• One API, every model. Speak the OpenAI *or* Anthropic wire format in; the gateway translates to whichever provider each model points at. Point an OpenAI or Claude SDK at one base_url and switch models without changing code.
• A real gateway, in Rust. Single static binary, low cold-start, lock-free config reads on the hot path, native streaming.
• Open source, free forever. Apache-2.0 licensed and built to run in your infrastructure. Choose AISIX Enterprise or AISIX Cloud when you want centralized management through a control plane and dashboard.
• Production controls built in. Routing & failover, rate limits, guardrails, caching, and observability ship in the box. (Budgets and spend caps are AISIX Enterprise and AISIX Cloud features — the gateway enforces the control plane's decisions.)

🧩 Features — available today

Covered by 183 end-to-end scenario files (496 cases) that run against real gateway processes.

• OpenAI-compatible proxy (:3000) — chat/completions, completions, responses, embeddings, rerank, images/{generations,edits}, audio/{speech,transcriptions,translations}, videos (submit → poll → fetch), files, batches, fine_tuning/jobs, realtime, GET /v1/models, plus a root-level /passthrough/:provider/* escape hatch. Native SSE streaming, tool/function calling, JSON mode, vision/multimodal input, and reasoning-content support.
• Anthropic Messages API — POST /v1/messages as a first-class route, working against any configured upstream: requests and responses (including streaming) are translated both ways when a model points at a non-Anthropic provider.
• Routing & failover — virtual/routing models with six strategies: round_robin (smooth weighted round-robin), consistent_hash (session affinity keyed by header / cookie / API key / client IP), failover, plus metric-based least_cost, least_latency, and least_busy. Per-target priority tiers (active/backup pools), retry budgets, cooldowns, tag-conditional targets, and per-attempt timeouts.
• Ensemble models — fan one request out to a panel of models concurrently, then have a judge model synthesize a single answer, with a minimum-successful-responses threshold.
• Semantic routing — one virtual model that dispatches by the *meaning* of each request: it embeds the prompt, scores it against per-route example utterances, and routes to the best match (or a default). See the semantic routing docs.
• Rate limiting & concurrency — RPS/RPM/RPH/RPD + TPM/TPD + concurrency caps, AND-combined across caller keys, models, and policy scopes (api_key / model / team / member / team_member). Counters are per-process by default, or shared across replicas with the Redis backend.
• Guardrails — content-policy enforcement on input and output, in-process or through a provider: keyword/regex, built-in PII detection and redaction, Presidio, Lakera, OpenAI Moderation, AWS Bedrock Guardrails, Azure AI Content Safety (Prompt Shield + text moderation), and two Alibaba Cloud services. A block returns 422 content_filter; monitor mode records what would have happened without blocking.
• Caching — exact-match response cache with per-policy TTL and model/key scope matchers; memory and Redis backends; cost-saved telemetry on every hit. Separately, automatic prompt caching can be enabled per direct Anthropic model to inject cache breakpoints, so callers get provider-side prompt discounts without changing their requests.
• MCP gateway — front registered upstream MCP servers at /mcp with gateway-held credentials, per-server tool namespaces, and per-caller access. It serves every Streamable HTTP revision from 2025-03-26 through stateless 2026-07-28 without downstream sessions. Upstreams use initialize by default or server/discover with protocol_version: "2026-07-28". CI runs the official MCP suite's applicable tools-only protocol scenarios. Also exposes a REST API as MCP tools from its OpenAPI description.
• A2A agent gateway — front A2A (Agent-to-Agent) agents at /a2a/:agent, serving each agent's card with URLs rewritten to the gateway, over JSON-RPC 2.0.
• Inbound authentication — caller API keys (SHA-256 hashed, model allowlists, expiry, rotation), or OIDC/JWT bearer tokens validated against registered providers (Entra ID, Okta, Google Workspace, or any OIDC issuer) with JWKS caching.
• Observability — Prometheus /metrics, structured per-request access logs, usage events, OTLP/GenAI span export (Langfuse, Honeycomb, Grafana Cloud, or any OTLP receiver), plus dedicated Datadog and Aliyun SLS log exporters and object-storage (S3/GCS/Azure Blob) telemetry.
• Declarative configuration — one resources.yaml carries all ten resource collections (provider keys, models, caller keys, guardrails, MCP servers, A2A agents, cache policies, observability exporters, rate-limit policies, OIDC providers), validated against the same JSON Schemas the gateway uses at runtime. aisix validate checks a file offline; SIGHUP reloads it atomically.
• Operational endpoints — /livez and /readyz on the proxy listener; /status/config, /status/ready, /status/models, and Prometheus /metrics on a dedicated metrics listener (:9090). The admin listener (:3001) additionally serves a read-only resource surface, OpenAPI 3 with a Scalar UI, and a playground. Resources are managed declaratively — through the resources_file (reloaded on SIGHUP) or direct etcd writes — not through the admin listener; its former write endpoints were removed.

🔌 Supported providers

AISIX dispatches through five native adapter families — distinct wire-protocol bridges, not one generic relabel. Whatever the upstream protocol, the client-facing API stays OpenAI-shaped.

| bedrock | AWS Bedrock — Anthropic, Meta Llama, Mist