AI Gateway
Here you'll learn about Javelin's AI Gateway, the intelligent proxy that routes your AI traffic, enforces security policies, and sets the foundation for observability and governance.
The AI Gateway is built around three key concepts: Providers, Routes, and Endpoints.
Providers
A Provider is the bridge between your application logic and the underlying AI models. They centralize the necessary setup to authenticate, connect, and communicate with different model sources, so it's easy to manage and navigate among multiple services.
When you set up a Provider in Javelin, you're basically teaching the Gateway how to communicate with that service. This typically involves naming the Provider and securely storing the credentials (like an API key) in the Secrets Vault.
Routes
A Route is a named, reusable set of rules that dictates how to handle a specific type of request. Routes allow different parts of your application to use different models, providers, and configurations, all while maintaining clear visibility and control. When your application sends a request to Javelin it specifies a route name, and Javelin uses its configuration to process the request.
Routes consist of these key details:
Path: A unique name that becomes part of the API endpoint
Models and Providers: The models and providers handling the request
Policies: Granular controls that you apply to a Route
Behavior: Set up actions for the Route, like rate limiting or caching
Types of Routes
Custom Routes: Tailor the behavior for a route by setting up a unique payload or connecting with custom or internal LLM endpoints. You can control the exact request format, headers, and response parsing logic. These are useful for integrating proprietary models, APIs that don’t follow standard formats, or specialized ML workflows.
Unified Routes: Handle different model types under one logical route, which dynamically changes how requests are handled based on request metadata, model preferences, or business logic. These are useful for centralizing multiple capabilities (like chat + completion) into one route, A/B testing models, or conditional routing to fallback providers.
Auto-Provisioning: When you create a provider, Javelin automatically creates a unified route for that provider, labeled as Reserved Routes. You can use them as default access points for your providers, edit or extend them as needed, and speed up development and testing. These are designed to work out of the box, so you don't have to spend time on extra manual setup
Endpoints
Your AI applications use Endpoints to send requests. When you integrate your applications with Javelin, you change their API endpoints to route through Javelin rather than directly to providers.
What's Next?
Read about Guardrails & Policies to apply security rules to your routes.
Read about Observability to see what's happening inside the Gateway.
Get started with a walkthrough via the Quick Start Guide for Administrators.
Last updated