A Blazing Fast AI Gateway. Route to 200+ LLMs with 1 fast & friendly API.
The AI Gateway streamlines requests to 250+ language, vision, audio and image models with a unified API. It is production-ready with support for caching, fallbacks, retries, timeouts, loadbalancing, and can be edge-deployed for minimum latency.
✅ Blazing fast (9.9x faster) with a tiny footprint (~100kb build) ✅ Load balance across multiple models, providers, and keys ✅ Fallbacks make sure your app stays resilient ✅ Automatic Retries with exponential fallbacks come by default ✅ Configurable Request Timeouts to easily handle unresponsive LLM requests ✅ Multimodal to support routing between Vision, TTS, STT, Image Gen, and more models ✅ Plug-in middleware as needed ✅ Battle tested over 480B tokens ✅ Enterprise-ready for enhanced security, scale, and custom deployments
[!TIP] ⭐️ Star this repo to get Github release notifications for new provider integrations and features.
Use the AI gateway through the hosted API or self-host the open-source or enterprise versions on your environment.
The hosted API is the fastest way to setup an AI Gateway for your Gen AI application. We process billions of tokens daily and is in production with companies like Postman, Haptik, Turing, MultiOn, SiteGPT, and more.
To run the AI gateway locally, execute the following command in your terminal. (Needs npx installed) Or, explore deployment guides for Cloudflare, Docker, Node.js and more here.
npx @portkey-ai/gateway
Your AI Gateway is now running on http://localhost:8787 🚀
The AI Gateway's enterprise version offers enterprise-ready capabilities for org management, governance, security and more out of the box. Compare the open source, hosted and enterprise versions here.
The enterprise deployment architecture, supported platforms is available here - Enterprise Private Cloud Deployments
The AI Gateway is compatible with the OpenAI API & SDKs, and extends them to call 200+ LLMs reliably. To use the Gateway through OpenAI, update the client to include the gateway's URL and headers and make requests as usual. The AI gateway can translate requests written in the OpenAI format to the signature expected by the specified provider. View examples
Portkey Python SDK is a wrapper over the OpenAI Python SDK with added support for additional parameters across all other providers. If you're building with Python, this is the recommended library to connect to the Gateway.
pip install -qU portkey-ai
Portkey JS/TS SDK is a wrapper over the OpenAI JS SDK with added support for additional parameters across all other providers. If you're building with JS or TS, this is the recommended library to connect to the Gateway.
npm install --save portkey-ai
The AI gateway supports OpenAI compatible endpoints with added parameter support for all other providers and models. View API Reference.
Language | Supported SDKs |
---|---|
JS / TS | LangchainJS LlamaIndex.TS |
Python | Langchain LlamaIndex |
Go | go-openai |
Java | openai-java |
Rust | async-openai |
Ruby | ruby-openai |
Explore Gateway integrations with 25+ providers and 6+ frameworks.
Provider | Support | Stream | |
---|---|---|---|
OpenAI | ✅ | ✅ | |
Azure OpenAI | ✅ | ✅ | |
Anyscale | ✅ | ✅ | |
Google Gemini & Palm | ✅ | ✅ | |
Anthropic | ✅ | ✅ | |
Cohere | ✅ | ✅ | |
Together AI | ✅ | ✅ | |
Perplexity | ✅ | ✅ | |
Mistral | ✅ | ✅ | |
Nomic | ✅ | ✅ | |
AI21 | ✅ | ✅ | |
Stability AI | ✅ | ✅ | |
DeepInfra | ✅ | ✅ | |
Ollama | ✅ | ✅ | |
Novita AI | ✅ | ✅ |
Gateway seamlessly integrates with popular agent frameworks. Read the documentation here.
Framework | Call 200+ LLMs | Advanced Routing | Caching | Logging & Tracing* | Observability* | Prompt Management* |
---|---|---|---|---|---|---|
Autogen | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
CrewAI | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
LangChain | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
Phidata | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
Llama Index | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
Control Flow | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
Build Your Own Agents | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
*Only available on the hosted app. For detailed documentation click here.
These features are configured through the Gateway Config added to the x-portkey-config
header or the config
parameter in the SDKs.
Here's a sample config JSON showcasing the above features. All the features are optional
{
"retry": { "attempts": 5 },
"request_timeout": 10000,
"strategy": { "mode": "fallback" }, // or 'loadbalance', etc
"targets": [{
"provider": "openai",
"api_key": "sk-***"
},{
"strategy": {"mode": "loadbalance"}, // Optional nesting
"targets": {...}
}]
}
Then use the config in your API requests to the gateway.
Here's a guide to use the config object in your request.
Make your AI app more reliable and forward compatible, while ensuring complete data security and privacy.
✅ Secure Key Management - for role-based access control and tracking ✅ Simple & Semantic Caching - to serve repeat queries faster & save costs ✅ Access Control & Inbound Rules - to control which IPs and Geos can connect to your deployments ✅ PII Redaction - to automatically remove sensitive data from your requests to prevent indavertent exposure ✅ SOC2, ISO, HIPAA, GDPR Compliances - for best security practices ✅ Professional Support - along with feature prioritization
Schedule a call to discuss enterprise deployments
The easiest way to contribute is to pick an issue with the good first issue
tag 💪. Read the contribution guidelines here.
Bug Report? File here | Feature Request? File here
Join our growing community around the world, for help, ideas, and discussions on AI.